2017-12-15 21:50 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0008281CentOS-7qemu-kvmpublic2017-11-16 08:11
Reporterjirand 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
Product Version7.0-1406 
Target VersionFixed in Version 
Summary0008281: Live migration of KVM guests with non-shared storage broken
Descriptionvirsh migrate with flags --live and --copy-storage-(all|inc) fails on CentOS 7 with error
"error: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported"

Background:

Based on https://bugzilla.redhat.com/show_bug.cgi?id=1022392 qemu native storage migration has been disabled in favor to nbd storage migration. This seems sane. However, current spec file and source for qemu-kvm disables drive-mirror feature of qemu-monitor unles the build is marked for rhev. (See qapi-schema.json from line 1845 (#_rhev-only CONFIG_LIVE_BLOCK_OPS) to line 1956.
Combination of these two changes efectively disables any possibilito to live-migrate KVM Guests with non-shared storage.

I can understand the decission to limit live migration of guests only to RHEV (some sales reasons I suppose). In CentOS however I'd expect this to work since there is no RHEV equivalent build which would have this feature enabled.

What happens in libvirt:
1. Libvirt performs cookie exchange
2. Finds that drive-mirror feature is not available on either side
3. Decides to fall-back to old-style storage migration
4. Upon this request qemu returns "this feature or command is not currently supported" due to previously mentined patch.

Proposed solution:
IMHO there are two possible solutions (or combination of both)
1. (My preference) Update qemu-kvm.spec file for CentOS builds to include CONFIG_LIVE_BLOCK_OPS features.
2. Update qemu-kvm.spec to include "--enable-live-block-migration" in config parameters. (This makes the migration process backward compatible with CentOS 6, provided that machine type and other params of the guest are supported on CentOS 6 too. However, this falls back to the supposedly buggy old method)
Steps To ReproduceCreate qemu-kvm guest with non-shared storage. Method of creation or actual parameters are insignifficant. My test guest xml is as follows:

<domain type='kvm'>
  <name>test</name>
  <uuid>d12f4b5a-f7b8-4d69-ba2e-73a062912f91</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5c:ec:ea'/>
      <source bridge='vbr104'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='8192' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Get another Libvirt - KVM host ready
Start migration using

virsh migrate --live --copy-storage-all test qemu-ssh://<your other host>/system
TagsNo tags attached.
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0022794

beinbliss (reporter)

IMO this should get higher severity and higher priority since it 'blocks' smooth migrations between kvm servers.

I am using lvm as a backing devices:

<domain type='kvm'>
  <name>test-lvm</name>
  <uuid>39c42304-8173-4ac7-a26f-f477d977b996</uuid>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>12</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.6.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/test/test-root'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/test/test-lvm1'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/test/test-lvm2'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <source bridge='br00'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br01'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us' passwd='test'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

# virsh migrate --live test-lvm qemu+ssh://192.168.0.2/system --copy-storage-all --verbose --persistent
error: internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported

~0025020

roidelapluie (reporter)

+1. This should get higher priority.

~0026104

msedlmair (reporter)

Any updates here? I would love to see this feature active again on CentOS 7.

+ 1 for higher priority

~0026706

grumpycat (reporter)

I've got the same behavior with:

* CentOS Linux release 7.2.1511 (Core)
* qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64
* libvirt-1.2.17-13.el7_2.2.x86_64

The VMs are created by OpenStack (Kilo) on local storage, so we've got the same problem here.

+ 1 for higher priority

~0026794

mysteq (reporter)

We've got same problem between centos 7 hosts.

internal error: unable to execute QEMU command 'migrate': this feature or command is not currently supported

* CentOS Linux release 7.2.1511 (Core)
* libvirt-1.2.17-13.el7_2.4.x86_64
* qemu-kvm-1.5.3-105.el7_2.4.x86_64

We create VMs by hand from virt-manager on local storage.

+1 for higher priority

~0027124

apevec (reporter)

centos7 base will keep qemu as is in RHEL7 but you can use qemu-kvm-ev from Virt SIG, it is also updated version:

# yum install centos-release-qemu-ev # from CenOS7 Extras
# yum install qemu-kvm-ev
# rpm -q qemu-kvm-ev
qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64

~0027141

grumpycat (reporter)

qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64 comes actually from virt CentOS repository.

Does the new minor version patch this problem ?

~0030212

JensKuehnel (reporter)

The current version qemu-kvm-ev-2.9.0-16.el7_4.5.1.x86_64 is still broken.
From Centos 7.4.1708

Error message:
"error: internal error: unable to execute QEMU command 'drive-mirror': failed to connect socket: No route to host.

I tried to rebuild this package with --enable-live-block-migration in SOURCES/build_configure.sh, but with the same result.

~0030213

TrevorH (developer)

No route to host is what you get when iptables rules on the destination server do not allow connections on that port. No idea if that's what's happening here but live migration uses a random port from 49152:49261 so you need to allow those ports on the server to-be-migrated-to.

~0030347

JensKuehnel (reporter)

I was able to successfully migrate my systems between two Centos7 machines.
I used qemu-kvm-ev-2.9.0-16.el7_4.5.1.x86_64 from centos-qemu-ev/7/x86_64 on Centos 7.4.1709.

I think this case can be closed now.

~0030586

nukleonus (reporter)

Any update? We see, that on https://bugzilla.redhat.com/show_bug.cgi?id=1219541 problem was resolved, but we don't see, in changes, that this bug was fixed..

~0030590

apevec (reporter)

rhbz 1219541 is unrelated to the originally reported issue.
qemu-kmv-ev in CBS is maintained by the VirtSIG, I'd suggest to join their meetings and ask to update the build if newer SRPM is available on ftp.r.c.
+Notes

-Issue History
Date Modified Username Field Change
2015-03-06 09:10 jirand New Issue
2015-04-17 02:05 beinbliss Note Added: 0022794
2015-12-09 14:47 roidelapluie Note Added: 0025020
2016-03-23 13:48 msedlmair Note Added: 0026104
2016-05-27 09:26 grumpycat Note Added: 0026706
2016-06-06 07:56 mysteq Note Added: 0026794
2016-07-26 11:17 apevec Note Added: 0027124
2016-07-27 15:17 grumpycat Note Added: 0027141
2017-09-26 13:57 JensKuehnel Note Added: 0030212
2017-09-26 14:48 TrevorH Note Added: 0030213
2017-10-10 16:16 JensKuehnel Note Added: 0030347
2017-11-15 20:12 nukleonus Note Added: 0030586
2017-11-16 08:11 apevec Note Added: 0030590
+Issue History