View Issue Details

IDProjectCategoryView StatusLast Update
0008340CentOS-6qemu-kvmpublic2015-03-25 23:48
ReporterJunaid Shahid 
PrioritynormalSeveritycrashReproducibilitysometimes
Status newResolutionopen 
PlatformSupermicro AS -2122TG-HTRF, AMDOS6.4OS Version2.6.32-358.6.2
Product Version6.4 
Target VersionFixed in Version 
Summary0008340: There is a kernel panic after very few months or weeks, with qemu-kvm tainted, VMs can't run. The machine needs to be rebooted.
DescriptionThis is a heavily loaded KVM hypervisor node with ~90 windows 2K12 instances running on it. There are multiple machines with the same configuration and VMs are configured as highly available (HA) services through Red Hat Clustering Suite.

These hypervisor nodes crash after qemu-kvm module is "tainted" and a kernel panic follows.

Here's the crash trace:

2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: BUG: unable to handle kernel paging request at 00000000a04743a5
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: IP: [<ffffffffa0471d05>] ftrace_profile_kvm_cr+0xf5/0x100 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: PGD 0
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Oops: 0000 [#1] SMP
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: last sysfs file: /sys/devices/system/node/node3/meminfo
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: CPU 26
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Modules linked in: dlm configfs ip6table_filter ip6_tables iptable_filter ip_tables ebtable_nat ebtables softdog nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 openvswitch(U) libcrc32c vhost_net macvtap macvlan tun kvm_amd kvm igb ixgbe dca ptp pps_core mdio serio_raw fam15h_power k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core sg shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic pata_atiixp rcraid(P)(U) dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel:
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Pid: 11586, comm: qemu-kvm Tainted: P --------------- 2.6.32-358.6.2.el6.x86_64 #1 Supermicro AS -2122TG-HTRF/H8DGT
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RIP: 0010:[<ffffffffa0471d05>] [<ffffffffa0471d05>] ftrace_profile_kvm_cr+0xf5/0x100 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RSP: 0018:ffff883b46313c10 EFLAGS: 00010246
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RAX: ffffffffa03a1960 RBX: ffff883b46314c78 RCX: 000000000000ec42
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RDX: ffff883b46314cf8 RSI: 0000000000000246 RDI: ffff883b46314c78
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RBP: 00000000a04743b5 R08: ffff883b46314d00 R09: 00000000ffffffff
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff883b46313c38
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: R13: ffff883b46314cf8 R14: ffff883e1a030ae0 R15: ffff883e1a030ae0
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: FS: 00007f43dc877700(0000) GS:ffff886090c80000(0000) knlGS:0000000000000000
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: CR2: 00000000a04743a5 CR3: 0000003170ca1000 CR4: 00000000000407e0
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Process qemu-kvm (pid: 11586, threadinfo ffff883b46312000, task ffff883e1a030ae0)
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Stack:
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: ffffffffa04743b5 ffff883b46313c38 ffff883b46314c78 ffff883b46313c88
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: <d> ffffffffa046c095 0000000000000000 ffff883e1a030ae0 ffffffff81096ca0
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: <d> ffff883b46314d00 ffff883b46314d00 ffff883b46314c78 ffff883b46314c78
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Call Trace:
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffffa04743b5>] ? kvm_arch_vcpu_runnable+0x45/0x60 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffffa046c095>] ? kvm_vcpu_block+0x95/0xc0 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffffa047f0af>] ? kvm_arch_vcpu_ioctl_run+0x5df/0x10f0 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff810aa3ee>] ? futex_wake+0x10e/0x120
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffffa0467ff4>] ? kvm_vcpu_ioctl+0x434/0x580 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff81194fb2>] ? vfs_ioctl+0x22/0xa0
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff811950dc>] ? do_vfs_ioctl+0xc/0x580
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff8119547a>] ? do_vfs_ioctl+0x3aa/0x580
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff810ace2b>] ? sys_futex+0x7b/0x170
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff811956d1>] ? sys_ioctl+0x81/0xa0
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff810dc645>] ? __audit_syscall_exit+0x265/0x290
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Code: 7d 0c 8b 3d 86 f6 02 00 e8 e9 57 ca e0 44 89 e7 e8 51 d0 c9 e0 48 89 df 57 9d 66 66 90 66 90 48 8b 5d d8 4c 8b 65 e0 4c 8b 6d e8 <4c> 8b 75 f0 4c 8b 7d f8 c9 c3 90 55 48 89 e5 48 83 ec 40 4c 89
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RIP [<ffffffffa0471d05>] ftrace_profile_kvm_cr+0xf5/0x100 [kvm]
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: RSP <ffff883b46313c10>
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: CR2: 00000000a04743a5
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: ---[ end trace cf8987a57032fbac ]---
2015-03-25T03:56:19-07:00 CH3KVMC16-0709-B kernel: Kernel panic - not syncing: Fatal exception
Steps To ReproduceAfter an uptime of a couple of months, we hit this bug/ issue on one of our many AMD machines.
TagsNo tags attached.

Activities

Junaid Shahid

Junaid Shahid

2015-03-25 23:48

reporter  

info.zip (176,955 bytes)

Issue History

Date Modified Username Field Change
2015-03-25 23:23 Junaid Shahid New Issue
2015-03-25 23:48 Junaid Shahid File Added: info.zip