2017-08-23 00:41 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0012008CentOS-7libvirtpublic2017-07-06 04:51
Reportersomesh_pant 
PrioritynormalSeveritycrashReproducibilityalways
StatusnewResolutionopen 
PlatformOSCentOSOS Version
Product Version7.2.1511 
Target VersionFixed in Version 
Summary0012008: VM on host machine shutsdown automatically [qemu: terminating on signal 15 from pid 5931]
DescriptionHost server (runs Oracle):

CentOS Linux release 7.2.1511 (Core), Linux ora18-dl.intermesh.net 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

VM on hostserver:

CentOS release 6.8 (Final), Linux ora17-dl.intermesh.net 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


Frequency of error: every month approximately at similar day

 
Error reported on guest xml's log [/var/log/libvirt/qemu/ora-server.log]:

  
qemu: terminating on signal 15 from pid 5931
2016-08-14 02:10:56.876+0000: shutting down
2016-08-14 02:11:06.025+0000: starting up libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org), qemu version: 1.5.3 (qemu-kvm-1.5.3-105.el7_2.3)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name ora17 -S -machine rhel6.3.0,accel=kvm,usb=off -m 40960 -realtime mlock=off -smp 20,maxcpus=24,sockets=24,cores=1,threads=1 -uuid 4f7f086c-31e0-3f79-8f67-b045fbf78bba -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-ora17/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/KVM/ora17,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/KVM/data_ora17,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x16,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/KVM/swap_ora17,if=none,id=drive-virtio-disk2,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x15,drive=drive-virtio-disk2,id=virtio-disk2 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1b:76:e2,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:60:2f:3c,bus=pci.0,addr=0x7 -netdev tap,fd=27,id=hostnet2,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:c6:27:a4,bus=pci.0,addr=0x8 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
qemu: terminating on signal 15 from pid 5931
2016-09-11 14:59:35.458+0000: shutting down
2016-09-11 14:59:41.705+0000: starting up libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org), qemu version: 1.5.3 (qemu-kvm-1.5.3-105.el7_2.3)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name ora17 -S -machine rhel6.3.0,accel=kvm,usb=off -m 40960 -realtime mlock=off -smp 20,maxcpus=24,sockets=24,cores=1,threads=1 -uuid 4f7f086c-31e0-3f79-8f67-b045fbf78bba -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-ora17/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/KVM/ora17,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/KVM/data_ora17,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x16,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/KVM/swap_ora17,if=none,id=drive-virtio-disk2,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x15,drive=drive-virtio-disk2,id=virtio-disk2 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1b:76:e2,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:60:2f:3c,bus=pci.0,addr=0x7 -netdev tap,fd=27,id=hostnet2,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:c6:27:a4,bus=pci.0,addr=0x8 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
qemu: terminating on signal 15 from pid 5931
2016-10-09 08:23:06.093+0000: shutting down
2016-10-09 08:23:11.236+0000: starting up libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org), qemu version: 1.5.3 (qemu-kvm-1.5.3-105.el7_2.3)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name ora17 -S -machine rhel6.3.0,accel=kvm,usb=off -m 40960 -realtime mlock=off -smp 20,maxcpus=24,sockets=24,cores=1,threads=1 -uuid 4f7f086c-31e0-3f79-8f67-b045fbf78bba -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-ora17/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/KVM/ora17,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/KVM/data_ora17,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x16,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/KVM/swap_ora17,if=none,id=drive-virtio-disk2,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x15,drive=drive-virtio-disk2,id=virtio-disk2 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:1b:76:e2,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:60:2f:3c,bus=pci.0,addr=0x7 -netdev tap,fd=27,id=hostnet2,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:c6:27:a4,bus=pci.0,addr=0x8 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)

***************************************************


Pid 5931 belongs to libvirtd daemon


On HOST server:

$ ps ax|grep 5931
 5931 ? Ssl 0:23 /usr/sbin/libvirtd


This HOST server has only one VM.

No memory exhaustion, space issues, Utilization normal

/var/log/messages do not show any error
journalct does not have any messages

The host server crashes with this error.
Tagsqemu-kvm
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0027695

somesh_pant (reporter)

Error reported on guest xml's log [/var/log/libvirt/qemu/ora-server.log]:

chhanged to

Error reported on Host xml's (guest) log [/var/log/libvirt/qemu/ora-server.log]:

~0027706

somesh_pant (reporter)

Saw same error in other two VMs we have.

~0027858

somesh_pant (reporter)

It happened again today with same errors. Please help. This is frustrating.

qemu: terminating on signal 15 from pid 6411
2016-11-07 03:47:29.558+0000: shutting down
2016-11-07 03:47:36.106+0000: starting up libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org), qemu version: 1.5.3 (qemu-kvm-1.5.3-105.el7_2.4)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name ora10-dl -S -machine rhel6.3.0,accel=kvm,usb=off -m 45056 -realtime mlock=off -smp 22,sockets=22,cores=1,threads=1 -uuid be0778f3-24ba-9a44-7a6d-cc9be0d27b66 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-ora10-dl/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/KVM/ora10-dl,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sdb,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/KVM/ora10-dl_disk,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x11,drive=drive-virtio-disk2,id=virtio-disk2 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:ca:3a:bc,bus=pci.0,addr=0x3 -netdev tap,fd=25,id=hostnet1,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:3a:c4:eb,bus=pci.0,addr=0x7 -netdev tap,fd=27,id=hostnet2,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=52:54:00:8c:f6:97,bus=pci.0,addr=0x8 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)

~0028012

somesh_pant (reporter)

Happened again with no clue on what's wrong.

~0028065

somesh_pant (reporter)

Happened again with one server with no clue.

Is this forum active?

~0029498

somesh_pant (reporter)

The problem is still the same with no luck

~0029598

cloudabove (reporter)

We are also seeing similar symptoms on two hypervisors (running 2.6.32-642.1.1.el6.x86_64). At the time of guest poweroff, the following entry is within /var/log/libvirt/libvirtd.log:

error : qemuMonitorIO:615 : internal error End of file from monitor

However, we only started seeing this issue following updates to the hypervisors applied in April 2017 (current theory).

Guests which have not been recently rebooted are *not* affected but all new guests or those that have been rebooted will power off at random intervals but typically once every 7-21 days.

We also saw similar reports in the Arch community but this was following an update to the 'jemalloc' package; https://bugs.archlinux.org/task/54483.

There's no evidence of resource contention or OOM'ing. We have tried rolling back updates to the KVM and QEMU packages (no impact), disabling cgroups (no impact) and changing between virtio, IDE and Intel virtual hardware (no impact). A full reboot of the hypervisor hasn't yet been attempted.

Affected guests are running CentOS 6/7 and Windows Server.

@somesh_pant I don't believe we're facing identical issues but I thought I'd post an update to see if there were any other similarities in our findings.

~0029611

somesh_pant (reporter)

@cloudabove we are on 2.6.32-642.4.2.el6.x86_64. we upgraded it after this issue but no relief.

The error that you get looks a bit different. For us, the daemon, libvirtd is getting a shutdown signal and server becomes unresponsive.

Yes the VMs run oracle. What application do you run on?
OS: CentOS release 6.8 (Final)
No recource exhaustion in server

It happened again today 6hrs back. The only option was to reboot and bring up the database.
+Notes

-Issue History
Date Modified Username Field Change
2016-10-11 07:04 somesh_pant New Issue
2016-10-11 07:04 somesh_pant Tag Attached: qemu-kvm
2016-10-12 03:29 somesh_pant Note Added: 0027695
2016-10-13 11:35 somesh_pant Note Added: 0027706
2016-11-07 09:24 somesh_pant Note Added: 0027858
2016-11-28 08:00 somesh_pant Note Added: 0028012
2016-12-04 06:26 somesh_pant Note Added: 0028065
2017-06-16 07:36 somesh_pant Note Added: 0029498
2017-07-05 09:04 cloudabove Note Added: 0029598
2017-07-06 04:51 somesh_pant Note Added: 0029611
+Issue History