View Issue Details

IDProjectCategoryView StatusLast Update
0016773CentOS-7libvirtpublic2019-11-27 14:43
Reportermartin85 
PrioritynormalSeveritymajorReproducibilitysometimes
Status newResolutionopen 
Product Version7.7-1908 
Target VersionFixed in Version 
Summary0016773: qemu-kvm: kvm_init_vcpu failed: Cannot allocate memory
DescriptionI am facing qemu-kvm allocation memory issue during online guest migration between two hosts. I have activated guest placement to NUMA nodes. Destination host should have enough of free memory (using hugepages) in NUMA node 1 for migrated guest (consuming 16GB of RAM).

guest memory configuration:
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>


source host:
2019-11-27 11:06:28.523+0000: 2562914: info : libvirt version: 4.5.0, package: 23.el7_7.1 (CentOS BuildSystem <http://bugs.centos.org>, 2019-09-13-18:01:52, x86-02.bsys.centos.org)
2019-11-27 11:06:28.523+0000: 2562914: info : hostname: ***
2019-11-27 11:06:28.523+0000: 2562914: error : virNetClientProgramDispatchError:174 : internal error: qemu unexpectedly closed the monitor: 2019-11-27T11:06:28.307440Z qemu-kvm: kvm_init_vcpu failed: Cannot allocate memory

cat /proc/meminfo | grep Huge
AnonHugePages: 256000 kB
HugePages_Total: 20480
HugePages_Free: 12288
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

numastat -czs libvirt kvm qemu

Per-node process memory usage (in MBs)
PID Node 0 Node 1 Total
--------------- ------ ------ -----
338873 (qemu-kvm 14 16444 16458 <------------------ Migration of this instance fail on memory allocation
2562889 (libvirt 12 11 23
--------------- ------ ------ -----
Total 26 16455 16481

destination host:

2019-11-27 11:06:28.319+0000: 2612064: info : hostname: ***
2019-11-27 11:06:28.319+0000: 2612064: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer
2019-11-27 11:06:28.319+0000: 2612064: error : qemuProcessReportLogError:1923 : internal error: qemu unexpectedly closed the monitor: 2019-11-27T11:06:28.307440Z qemu-kvm: kvm_init_vcpu failed: Cannot allocate memory

cat /proc/meminfo | grep Huge
AnonHugePages: 327680 kB
HugePages_Total: 20480
HugePages_Free: 15872
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

numastat -czs libvirt kvm qemu
Per-node process memory usage (in MBs)
PID Node 0 Node 1 Total
--------------- ------ ------ -----
2616135 (qemu-kv 4157 10 4166
2618045 (qemu-kv 4144 10 4153
2617183 (qemu-kv 1075 10 1085
2612064 (libvirt 9 16 25
--------------- ------ ------ -----
Total 9385 45 9430

Same issue has been reported here https://bugzilla.redhat.com/show_bug.cgi?id=1010885 by last two comments.
Additional Informationsource and destination hosts have the same package versions:

libvirt-daemon-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-secret-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-interface-4.5.0-23.el7_7.1.x86_64
libvirt-bash-completion-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-scsi-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-config-nwfilter-4.5.0-23.el7_7.1.x86_64
libvirt-libs-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-network-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-nodedev-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-mpath-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-qemu-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-kvm-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-lxc-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-core-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-23.el7_7.1.x86_64
libvirt-client-4.5.0-23.el7_7.1.x86_64
centos-release-qemu-ev-1.0-4.el7.centos.noarch
qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64
ipxe-roms-qemu-20180825-2.git133f4c.el7.noarch
libvirt-daemon-driver-qemu-4.5.0-23.el7_7.1.x86_64
qemu-img-ev-2.12.0-33.1.el7.x86_64
qemu-kvm-ev-2.12.0-33.1.el7.x86_64
kernel-3.10.0-1062.4.1.el7.x86_64
numad-0.5-18.20150602git.el7.x86_64
Tagslibvirt, numa
abrt_hash
URLhttps://bugzilla.redhat.com/show_bug.cgi?id=1010885

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2019-11-27 14:43 martin85 New Issue
2019-11-27 14:43 martin85 Tag Attached: libvirt
2019-11-27 14:43 martin85 Tag Attached: numa