View Issue Details

IDProjectCategoryView StatusLast Update
0008281CentOS-7qemu-kvmpublic2017-11-16 08:11
Reporterjirand 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.0-1406 
Target VersionFixed in Version 
Summary0008281: Live migration of KVM guests with non-shared storage broken
Descriptionvirsh migrate with flags --live and --copy-storage-(all|inc) fails on CentOS 7 with error
"error: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported"

Background:

Based on https://bugzilla.redhat.com/show_bug.cgi?id=1022392 qemu native storage migration has been disabled in favor to nbd storage migration. This seems sane. However, current spec file and source for qemu-kvm disables drive-mirror feature of qemu-monitor unles the build is marked for rhev. (See qapi-schema.json from line 1845 (#_rhev-only CONFIG_LIVE_BLOCK_OPS) to line 1956.
Combination of these two changes efectively disables any possibilito to live-migrate KVM Guests with non-shared storage.

I can understand the decission to limit live migration of guests only to RHEV (some sales reasons I suppose). In CentOS however I'd expect this to work since there is no RHEV equivalent build which would have this feature enabled.

What happens in libvirt:
1. Libvirt performs cookie exchange
2. Finds that drive-mirror feature is not available on either side
3. Decides to fall-back to old-style storage migration
4. Upon this request qemu returns "this feature or command is not currently supported" due to previously mentined patch.

Proposed solution:
IMHO there are two possible solutions (or combination of both)
1. (My preference) Update qemu-kvm.spec file for CentOS builds to include CONFIG_LIVE_BLOCK_OPS features.
2. Update qemu-kvm.spec to include "--enable-live-block-migration" in config parameters. (This makes the migration process backward compatible with CentOS 6, provided that machine type and other params of the guest are supported on CentOS 6 too. However, this falls back to the supposedly buggy old method)
Steps To ReproduceCreate qemu-kvm guest with non-shared storage. Method of creation or actual parameters are insignifficant. My test guest xml is as follows:

<domain type='kvm'>
  <name>test</name>
  <uuid>d12f4b5a-f7b8-4d69-ba2e-73a062912f91</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5c:ec:ea'/>
      <source bridge='vbr104'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='8192' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Get another Libvirt - KVM host ready
Start migration using

virsh migrate --live --copy-storage-all test qemu-ssh://<your other host>/system
TagsNo tags attached.
abrt_hash
URL

Activities

beinbliss

beinbliss

2015-04-17 02:05

reporter   ~0022794

IMO this should get higher severity and higher priority since it 'blocks' smooth migrations between kvm servers.

I am using lvm as a backing devices:

<domain type='kvm'>
  <name>test-lvm</name>
  <uuid>39c42304-8173-4ac7-a26f-f477d977b996</uuid>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>12</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/test/test-root'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/test/test-lvm1'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/test/test-lvm2'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <source bridge='br00'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br01'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us' passwd='test'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

# virsh migrate --live test-lvm qemu+ssh://192.168.0.2/system --copy-storage-all --verbose --persistent
error: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported
roidelapluie

roidelapluie

2015-12-09 14:47

reporter   ~0025020

+1. This should get higher priority.
msedlmair

msedlmair

2016-03-23 13:48

reporter   ~0026104

Any updates here? I would love to see this feature active again on CentOS 7.

+ 1 for higher priority
grumpycat

grumpycat

2016-05-27 09:26

reporter   ~0026706

I've got the same behavior with:

* CentOS Linux release 7.2.1511 (Core)
* qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64
* libvirt-1.2.17-13.el7_2.2.x86_64

The VMs are created by OpenStack (Kilo) on local storage, so we've got the same problem here.

+ 1 for higher priority
mysteq

mysteq

2016-06-06 07:56

reporter   ~0026794

We've got same problem between centos 7 hosts.

internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported

* CentOS Linux release 7.2.1511 (Core)
* libvirt-1.2.17-13.el7_2.4.x86_64
* qemu-kvm-1.5.3-105.el7_2.4.x86_64

We create VMs by hand from virt-manager on local storage.

+1 for higher priority
apevec

apevec

2016-07-26 11:17

reporter   ~0027124

centos7 base will keep qemu as is in RHEL7 but you can use qemu-kvm-ev from Virt SIG, it is also updated version:

# yum install centos-release-qemu-ev # from CenOS7 Extras
# yum install qemu-kvm-ev
# rpm -q qemu-kvm-ev
qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64
grumpycat

grumpycat

2016-07-27 15:17

reporter   ~0027141

qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64 comes actually from virt CentOS repository.

Does the new minor version patch this problem ?
JensKuehnel

JensKuehnel

2017-09-26 13:57

reporter   ~0030212

The current version qemu-kvm-ev-2.9.0-16.el7_4.5.1.x86_64 is still broken.
From Centos 7.4.1708

Error message:
"error: internal error: unable to execute QEMU command 'drive-mirror': failed to connect socket: No route to host.

I tried to rebuild this package with --enable-live-block-migration in SOURCES/build_configure.sh, but with the same result.
TrevorH

TrevorH

2017-09-26 14:48

developer   ~0030213

No route to host is what you get when iptables rules on the destination server do not allow connections on that port. No idea if that's what's happening here but live migration uses a random port from 49152:49261 so you need to allow those ports on the server to-be-migrated-to.
JensKuehnel

JensKuehnel

2017-10-10 16:16

reporter   ~0030347

I was able to successfully migrate my systems between two Centos7 machines.
I used qemu-kvm-ev-2.9.0-16.el7_4.5.1.x86_64 from centos-qemu-ev/7/x86_64 on Centos 7.4.1709.

I think this case can be closed now.
nukleonus

nukleonus

2017-11-15 20:12

reporter   ~0030586

Any update? We see, that on https://bugzilla.redhat.com/show_bug.cgi?id=1219541 problem was resolved, but we don't see, in changes, that this bug was fixed..
apevec

apevec

2017-11-16 08:11

reporter   ~0030590

rhbz 1219541 is unrelated to the originally reported issue.
qemu-kmv-ev in CBS is maintained by the VirtSIG, I'd suggest to join their meetings and ask to update the build if newer SRPM is available on ftp.r.c.

Issue History

Date Modified Username Field Change
2015-03-06 09:10 jirand New Issue
2015-04-17 02:05 beinbliss Note Added: 0022794
2015-12-09 14:47 roidelapluie Note Added: 0025020
2016-03-23 13:48 msedlmair Note Added: 0026104
2016-05-27 09:26 grumpycat Note Added: 0026706
2016-06-06 07:56 mysteq Note Added: 0026794
2016-07-26 11:17 apevec Note Added: 0027124
2016-07-27 15:17 grumpycat Note Added: 0027141
2017-09-26 13:57 JensKuehnel Note Added: 0030212
2017-09-26 14:48 TrevorH Note Added: 0030213
2017-10-10 16:16 JensKuehnel Note Added: 0030347
2017-11-15 20:12 nukleonus Note Added: 0030586
2017-11-16 08:11 apevec Note Added: 0030590