View Issue Details

IDProjectCategoryView StatusLast Update
0007381Xen4[CentOS-6] kernelpublic2015-03-30 05:35
Reporteradip 
PriorityhighSeveritymajorReproducibilityhave not tried
Status newResolutionopen 
Product Version[CentOS-6] 6.5 
Target VersionFixed in Version 
Summary0007381: [kernel-3.10.43-11.el6.centos.alt.x86_64] Network lost during domU shutdown
DescriptionRunning the latest CentOS 6.5 64bit Xen kernel, I've got this stack trace. After this all the running domUs on this node lost their network and the node had to be rebooted.
[...]
Jul 16 14:50:48 xenlocal1 kernel: br0: port 15(vif31.1) entered disabled state
Jul 16 14:50:48 xenlocal1 kernel: BUG: unable to handle kernel paging request at ffffc90010abe0d0
Jul 16 14:50:48 xenlocal1 kernel: IP: [<ffffffffa02bb899>] netbk_gop_skb+0xb9/0x290 [xen_netback]
Jul 16 14:50:48 xenlocal1 kernel: PGD 388c8067 PUD 388c9067 PMD 34b2c067 PTE 0
Jul 16 14:50:48 xenlocal1 kernel: Oops: 0000 [#1] SMP
Jul 16 14:50:48 xenlocal1 kernel: Modules linked in: ext3 jbd xen_pciback xen_gntalloc ipmi_devintf ipmi_si ipmi_msghandler bridge stp llc bonding ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_connt
rack ip6table_filter ip6_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_tr
ansport_iscsi blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd gpio_ich iTCO_wdt iTCO_vendor_support dcdbas coretemp freq_table mperf crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_i
ntel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 microcode pcspkr ses enclosure sg joydev lpc_ich shpchp igb acpi_power_meter ixgbe hwmon dca ptp pps_core mdio ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom megaraid_sas ahci libahci wmi ttm drm_kms_helper dm_mirror dm_region_hash dm_log dm_mod
Jul 16 14:50:48 xenlocal1 kernel: CPU: 0 PID: 823 Comm: netback/0 Not tainted 3.10.43-11.el6.centos.alt.x86_64 #1
Jul 16 14:50:48 xenlocal1 kernel: Hardware name: Dell Inc. PowerEdge R620/01W23F, BIOS 2.2.2 01/16/2014
Jul 16 14:50:48 xenlocal1 kernel: task: ffff880005691540 ti: ffff880035b76000 task.ti: ffff880035b76000
Jul 16 14:50:48 xenlocal1 kernel: RIP: e030:[<ffffffffa02bb899>] [<ffffffffa02bb899>] netbk_gop_skb+0xb9/0x290 [xen_netback]
Jul 16 14:50:48 xenlocal1 kernel: RSP: e02b:ffff880035b77cd8 EFLAGS: 00010202
Jul 16 14:50:48 xenlocal1 kernel: RAX: ffffc9001041aac0 RBX: ffff880011ac20c0 RCX: ffffc90010abe000
Jul 16 14:50:48 xenlocal1 kernel: RDX: 000000000000001a RSI: 0000000000000000 RDI: ffff880023886b80
Jul 16 14:50:48 xenlocal1 kernel: RBP: ffff880035b77d48 R08: 0000000000000000 R09: 0000000000000000
Jul 16 14:50:48 xenlocal1 kernel: R10: 0000000000007ff0 R11: 0000000000000002 R12: 0000000000000000
Jul 16 14:50:48 xenlocal1 kernel: R13: 0000000000000000 R14: ffff880035b77d98 R15: ffff880011bc7800
Jul 16 14:50:48 xenlocal1 kernel: FS: 00007f62b3a1f700(0000) GS:ffff88003f400000(0000) knlGS:0000000000000000
Jul 16 14:50:48 xenlocal1 kernel: CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 16 14:50:48 xenlocal1 kernel: CR2: ffffc90010abe0d0 CR3: 0000000037046000 CR4: 0000000000042660
Jul 16 14:50:48 xenlocal1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 16 14:50:48 xenlocal1 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 16 14:50:48 xenlocal1 kernel: Stack:
Jul 16 14:50:48 xenlocal1 kernel: ffff880038b00040 ffff880005691bd0 0000000000000000 ffff88003f411880
Jul 16 14:50:48 xenlocal1 kernel: ffff880035b77d18 000000008100392e 0000000000000000 00000001815f7ec7
Jul 16 14:50:48 xenlocal1 kernel: ffff880035b77d48 ffff880011ac20c0 0000000000000000 0000000000000000
Jul 16 14:50:48 xenlocal1 kernel: Call Trace:
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffffa02bbb55>] xen_netbk_rx_action+0xe5/0x600 [xen_netback]
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff813572a0>] ? process_msg+0x290/0x290
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff815f7ec7>] ? _raw_spin_unlock_irqrestore+0x17/0x20
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffffa02bddf0>] xen_netbk_kthread+0x80/0x1b0 [xen_netback]
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff81082800>] ? wake_up_bit+0x40/0x40
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffffa02bdd70>] ? xen_netbk_tx_build_gops+0x8b0/0x8b0 [xen_netback]
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff81081fee>] kthread+0xce/0xe0
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff8100392e>] ? xen_end_context_switch+0x1e/0x30
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff81081f20>] ? kthread_freezable_should_stop+0x70/0x70
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff81600e2c>] ret_from_fork+0x7c/0xb0
Jul 16 14:50:48 xenlocal1 kernel: [<ffffffff81081f20>] ? kthread_freezable_should_stop+0x70/0x70
Jul 16 14:50:48 xenlocal1 kernel: Code: 47 60 04 0f 85 89 01 00 00 8b b3 d0 00 00 00 48 8b bb d8 00 00 00 0f b7 74 37 02 89 70 08 89 d2 c7 40 04 00 00 00 00 48 83 c2 08 <0f> b7 34 d1 89 30 41 c7 46 20 00 00 00 00 8b 44 d1 04 41 89 46
Jul 16 14:50:48 xenlocal1 kernel: RIP [<ffffffffa02bb899>] netbk_gop_skb+0xb9/0x290 [xen_netback]
Jul 16 14:50:48 xenlocal1 kernel: RSP <ffff880035b77cd8>
Jul 16 14:50:48 xenlocal1 kernel: CR2: ffffc90010abe0d0
Jul 16 14:50:48 xenlocal1 kernel: ---[ end trace b7a22eb1f1f1e4df ]---
[...]
TagsNo tags attached.

Activities

MihkelParna

MihkelParna

2015-03-29 15:59

reporter   ~0022613

CentOS 6.5
Kernel: 3.10.68-11.el6.centos.alt.x86_64
xen-4.2.5-38.el6.centos.alt.x86_64


Same issue, during domU reboot/shutdown occasionally get the same call trace that ends with the hypervisor crashing and hard resetting.

grep 'Call Trace' /var/log/messages -A10
Mar 29 09:06:35 host00 kernel: Call Trace:
Mar 29 09:06:35 host00 kernel: [<ffffffff8100392e>] ? xen_end_context_switch+0x1e/0x30
Mar 29 09:06:35 host00 kernel: [<ffffffffa0385b55>] xen_netbk_rx_action+0xe5/0x600 [xen_netback]
Mar 29 09:06:35 host00 kernel: [<ffffffff815f9bd7>] ? _raw_spin_unlock_irqrestore+0x17/0x20
Mar 29 09:06:35 host00 kernel: [<ffffffffa0387df0>] xen_netbk_kthread+0x80/0x1b0 [xen_netback]
Mar 29 09:06:35 host00 kernel: [<ffffffff810828d0>] ? wake_up_bit+0x40/0x40
Mar 29 09:06:35 host00 kernel: [<ffffffffa0387d70>] ? xen_netbk_tx_build_gops+0x8b0/0x8b0 [xen_netback]
Mar 29 09:06:35 host00 kernel: [<ffffffff810820be>] kthread+0xce/0xe0
Mar 29 09:06:35 host00 kernel: [<ffffffff81081ff0>] ? kthread_freezable_should_stop+0x70/0x70
Mar 29 09:06:35 host00 kernel: [<ffffffff81602cec>] ret_from_fork+0x7c/0xb0
Mar 29 09:06:35 host00 kernel: [<ffffffff81081ff0>] ? kthread_freezable_should_stop+0x70/0x70
tigalch

tigalch

2015-03-29 18:07

manager   ~0022615

You should consider updateing to CentOS-6.6 and the current xen and kernel packages. Might be enough to solve this.
MihkelParna

MihkelParna

2015-03-30 05:35

reporter   ~0022621

Updated the distro to 6.6 but we are unable to upgrade the kernel/Xen packages due to the software layer on top of the hypervisor, will keep this updated as soon as we manage to upgrade the xen/kernel and see if we can reproduce the issue

Issue History

Date Modified Username Field Change
2014-07-16 05:58 adip New Issue
2015-03-29 15:59 MihkelParna Note Added: 0022613
2015-03-29 18:07 tigalch Note Added: 0022615
2015-03-30 05:35 MihkelParna Note Added: 0022621