View Issue Details

IDProjectCategoryView StatusLast Update
0013713CentOS-6kernelpublic2017-08-24 22:44
Reportersh4d0w Assigned To 
PrioritynormalSeveritycrashReproducibilityalways
Status newResolutionopen 
Summary0013713: events_unbound flush_to_ldisc kernel oops
DescriptionUpgraded multiple xen hypervisors from 3.18.x kernel to 4.9.x and started to see frequent crashes.

Researching the oops I found this kernel patch which seems to have been missed by upstream: https://lkml.org/lkml/2016/5/17/440

Applying the patch to the 4.9.44 kernel appears to have resolved it on my systems.
Steps To ReproduceSSH in frequently while system is under load, send commands before the prompt has returned.
TagsNo tags attached.

Activities

sh4d0w

sh4d0w

2017-08-24 22:44

reporter  

kernel_oops.txt (13,931 bytes)   
Aug 23 10:19:31 xen-028 kernel: [590071.735515] BUG: unable to handle kernel paging request at 0000000000002260
Aug 23 10:19:31 xen-028 kernel: [590071.735795] IP: [<ffffffff8152e6a4>] n_tty_receive_buf_common+0xa4/0x1f0
Aug 23 10:19:31 xen-028 kernel: [590071.736031] PGD 0 
Aug 23 10:19:31 xen-028 kernel: [590071.736083] 
Aug 23 10:19:31 xen-028 kernel: [590071.736300] Oops: 0000 [#1] SMP
Aug 23 10:19:31 xen-028 kernel: [590071.736470] Modules linked in: ebt_ip6 ebt_ip ebtable_filter ebtables arptable_filter arp_tables bridge xen_pciback xen_gntalloc nfsd auth_rpcgss nfsv3 nfs_acl nfs fscache lockd sunrpc grace 8021q mrp garp stp llc bonding blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd ipmi_devintf ipmi_si ipmi_msghandler gpio_ich iTCO_wdt iTCO_vendor_support fjes acpi_power_meter dcdbas pcspkr serio_raw joydev lpc_ich igb ixgbe dca ptp pps_core mdio i7core_edac edac_core bnx2 raid1 megaraid_sas ttm
Aug 23 10:19:31 xen-028 kernel: [590071.740051] CPU: 14 PID: 21615 Comm: kworker/u48:1 Not tainted 4.9.39-29.el6.x86_64 #1
Aug 23 10:19:31 xen-028 kernel: [590071.740330] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.0.7 08/18/2011
Aug 23 10:19:31 xen-028 kernel: [590071.740607] Workqueue: events_unbound flush_to_ldisc
Aug 23 10:19:31 xen-028 kernel: [590071.740806] task: ffff88008a6011c0 task.stack: ffffc9004cfec000
Aug 23 10:19:31 xen-028 kernel: [590071.740966] RIP: e030:[<ffffffff8152e6a4>]  [<ffffffff8152e6a4>] n_tty_receive_buf_common+0xa4/0x1f0
Aug 23 10:19:31 xen-028 kernel: [590071.741282] RSP: e02b:ffffc9004cfefb08  EFLAGS: 00010296
Aug 23 10:19:31 xen-028 kernel: [590071.741442] RAX: 0000000000002260 RBX: 0000000000000000 RCX: 000000000000000a
Aug 23 10:19:31 xen-028 kernel: [590071.741714] RDX: 0000000000000000 RSI: ffff88015ecd6420 RDI: ffff8800afd654d8
Aug 23 10:19:31 xen-028 kernel: [590071.741994] RBP: ffffc9004cfefb78 R08: 0000000000000001 R09: ffffffff81f0af00
Aug 23 10:19:31 xen-028 kernel: [590071.742274] R10: 0000000000007ff0 R11: 0000000000000078 R12: 000000000000000a
Aug 23 10:19:31 xen-028 kernel: [590071.742549] R13: ffff8800afd65400 R14: 0000000000000000 R15: ffff88015ecd6420
Aug 23 10:19:31 xen-028 kernel: [590071.742830] FS:  00007f81da7317c0(0000) GS:ffff8801c0980000(0000) knlGS:0000000000000000
Aug 23 10:19:31 xen-028 kernel: [590071.743112] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 23 10:19:31 xen-028 kernel: [590071.743283] CR2: 0000000000002260 CR3: 000000008f61f000 CR4: 0000000000002660
Aug 23 10:19:31 xen-028 kernel: [590071.743564] Stack:
Aug 23 10:19:31 xen-028 kernel: [590071.743719]  ffffc9001160000c 0000000000000000 ffff8800afd654d8 00000001c0999970
Aug 23 10:19:31 xen-028 kernel: [590071.744149]  0000000000002260 000000008a603340 ffff8801c0997000 0000000000000000
Aug 23 10:19:31 xen-028 kernel: [590071.744577]  ffff8801c098b890 ffff88015ecd6400 ffff8800b19e9c00 ffffc9004cfefbf8
Aug 23 10:19:31 xen-028 kernel: [590071.745008] Call Trace:
Aug 23 10:19:31 xen-028 kernel: [590071.745169]  [<ffffffff8152e804>] n_tty_receive_buf2+0x14/0x20
Aug 23 10:19:31 xen-028 kernel: [590071.745335]  [<ffffffff81531533>] tty_ldisc_receive_buf+0x23/0x50
Aug 23 10:19:31 xen-028 kernel: [590071.745501]  [<ffffffff81531958>] flush_to_ldisc+0xc8/0x100
Aug 23 10:19:31 xen-028 kernel: [590071.745669]  [<ffffffff8102eb3c>] ? __switch_to+0x1dc/0x680
Aug 23 10:19:31 xen-028 kernel: [590071.745836]  [<ffffffff810c0490>] process_one_work+0x170/0x500
Aug 23 10:19:31 xen-028 kernel: [590071.746005]  [<ffffffff818d4658>] ? __schedule+0x238/0x530
Aug 23 10:19:31 xen-028 kernel: [590071.746169]  [<ffffffff810c1234>] ? maybe_create_worker+0x94/0x120
Aug 23 10:19:31 xen-028 kernel: [590071.746342]  [<ffffffff818d4a3a>] ? schedule+0x3a/0xa0
Aug 23 10:19:31 xen-028 kernel: [590071.746506]  [<ffffffff810c1426>] worker_thread+0x166/0x580
Aug 23 10:19:31 xen-028 kernel: [590071.746671]  [<ffffffff818d4658>] ? __schedule+0x238/0x530
Aug 23 10:19:31 xen-028 kernel: [590071.749537]  [<ffffffff810d3882>] ? default_wake_function+0x12/0x20
Aug 23 10:19:31 xen-028 kernel: [590071.749706]  [<ffffffff810c12c0>] ? maybe_create_worker+0x120/0x120
Aug 23 10:19:31 xen-028 kernel: [590071.749872]  [<ffffffff818d4a3a>] ? schedule+0x3a/0xa0
Aug 23 10:19:31 xen-028 kernel: [590071.750040]  [<ffffffff818d8826>] ? _raw_spin_unlock_irqrestore+0x16/0x20
Aug 23 10:19:31 xen-028 kernel: [590071.750204]  [<ffffffff810c12c0>] ? maybe_create_worker+0x120/0x120
Aug 23 10:19:31 xen-028 kernel: [590071.750369]  [<ffffffff810c62c5>] kthread+0xe5/0x100
Aug 23 10:19:31 xen-028 kernel: [590071.750532]  [<ffffffff810c61e0>] ? __kthread_init_worker+0x40/0x40
Aug 23 10:19:31 xen-028 kernel: [590071.750698]  [<ffffffff818d8f55>] ret_from_fork+0x25/0x30
Aug 23 10:19:31 xen-028 kernel: [590071.750860] Code: 89 fe 4c 89 ef 89 45 98 e8 aa fb ff ff 8b 45 98 48 63 d0 48 85 db 48 8d 0c 13 48 0f 45 d9 01 45 bc 49 01 d7 41 29 c4 48 8b 45 b0 <48> 8b 30 48 89 75 c0 49 8b 0e 8d 96 00 10 00 00 29 ca 41 f6 85 
Aug 23 10:19:31 xen-028 kernel: [590071.753725] RIP  [<ffffffff8152e6a4>] n_tty_receive_buf_common+0xa4/0x1f0
Aug 23 10:19:31 xen-028 kernel: [590071.753928]  RSP <ffffc9004cfefb08>
Aug 23 10:19:31 xen-028 kernel: [590071.754087] CR2: 0000000000002260
Aug 23 10:19:31 xen-028 kernel: [590071.754247] ---[ end trace 3533c918d837d330 ]---
Aug 23 10:19:31 xen-028 kernel: [590071.760173] BUG: unable to handle kernel paging request at ffffffffffffffd8
Aug 23 10:19:31 xen-028 kernel: [590071.760422] IP: [<ffffffff810c5aa0>] kthread_data+0x10/0x20
Aug 23 10:19:31 xen-028 kernel: [590071.760632] PGD 1e0a067 
Aug 23 10:19:31 xen-028 kernel: [590071.760676] PUD 1e0c067 
Aug 23 10:19:31 xen-028 kernel: [590071.760871] PMD 0 
Aug 23 10:19:31 xen-028 kernel: [590071.760910] 
Aug 23 10:19:31 xen-028 kernel: [590071.761103] Oops: 0000 [#2] SMP
Aug 23 10:19:31 xen-028 kernel: [590071.761262] Modules linked in: ebt_ip6 ebt_ip ebtable_filter ebtables arptable_filter arp_tables bridge xen_pciback xen_gntalloc nfsd auth_rpcgss nfsv3 nfs_acl nfs fscache lockd sunrpc grace 8021q mrp garp stp llc bonding blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd ipmi_devintf ipmi_si ipmi_msghandler gpio_ich iTCO_wdt iTCO_vendor_support fjes acpi_power_meter dcdbas pcspkr serio_raw joydev lpc_ich igb ixgbe dca ptp pps_core mdio i7core_edac edac_core bnx2 raid1 megaraid_sas ttm
Aug 23 10:19:31 xen-028 kernel: [590071.764349] CPU: 14 PID: 21615 Comm: kworker/u48:1 Tainted: G      D         4.9.39-29.el6.x86_64 #1
Aug 23 10:19:31 xen-028 kernel: [590071.764633] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.0.7 08/18/2011
Aug 23 10:19:31 xen-028 kernel: [590071.764928] task: ffff88008a6011c0 task.stack: ffffc9004cfec000
Aug 23 10:19:31 xen-028 kernel: [590071.765092] RIP: e030:[<ffffffff810c5aa0>]  [<ffffffff810c5aa0>] kthread_data+0x10/0x20
Aug 23 10:19:31 xen-028 kernel: [590071.765412] RSP: e02b:ffffc9004cfefdd8  EFLAGS: 00010086
Aug 23 10:19:31 xen-028 kernel: [590071.765579] RAX: 0000000000000000 RBX: ffff8801c0999900 RCX: 000000000000000e
Aug 23 10:19:31 xen-028 kernel: [590071.765858] RDX: ffff8801bc009400 RSI: ffff88008a6011c0 RDI: ffff88008a6011c0
Aug 23 10:19:31 xen-028 kernel: [590071.766135] RBP: ffffc9004cfefdd8 R08: ffff8801c0980000 R09: 0000000000000001
Aug 23 10:19:31 xen-028 kernel: [590071.766415] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000019900
Aug 23 10:19:31 xen-028 kernel: [590071.766693] R13: ffff88008a6011c0 R14: 0000000000000000 R15: ffff88008a601b80
Aug 23 10:19:31 xen-028 kernel: [590071.766982] FS:  00007f81da7317c0(0000) GS:ffff8801c0980000(0000) knlGS:0000000000000000
Aug 23 10:19:31 xen-028 kernel: [590071.767276] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 23 10:19:31 xen-028 kernel: [590071.767441] CR2: 0000000000000028 CR3: 000000008f61f000 CR4: 0000000000002660
Aug 23 10:19:31 xen-028 kernel: [590071.767727] Stack:
Aug 23 10:19:31 xen-028 kernel: [590071.767886]  ffffc9004cfefe08 ffffffff810bd282 ffffc9004cfefdf8 ffff8801c0999900
Aug 23 10:19:31 xen-028 kernel: [590071.768332]  0000000000019900 ffff88008a6011c0 ffffc9004cfefe78 ffffffff818d4834
Aug 23 10:19:31 xen-028 kernel: [590071.768762]  0000000000000001 ffff8801aad54000 ffffc9004cfefe48 ffff8801ab7b5c08
Aug 23 10:19:31 xen-028 kernel: [590071.769185] Call Trace:
Aug 23 10:19:31 xen-028 kernel: [590071.769343]  [<ffffffff810bd282>] wq_worker_sleeping+0x12/0xa0
Aug 23 10:19:31 xen-028 kernel: [590071.769506]  [<ffffffff818d4834>] __schedule+0x414/0x530


Aug 23 14:29:55 xen-001 kernel: [15199.824132] BUG: unable to handle kernel paging request at 0000000000002260
Aug 23 14:29:55 xen-001 kernel: [15199.824541] IP: [<ffffffff8152e6a4>] n_tty_receive_buf_common+0xa4/0x1f0
Aug 23 14:29:55 xen-001 kernel: [15199.824885] PGD 0
Aug 23 14:29:55 xen-001 kernel: [15199.824950]
Aug 23 14:29:55 xen-001 kernel: [15199.825274] Oops: 0000 [#1] SMP
Aug 23 14:29:55 xen-001 kernel: [15199.825541] Modules linked in: mpt3sas scsi_transport_sas raid_class mptctl mptbase dell_rbu ebt_ip6 ebt_ip ebtable_filter ebtables arptable_filter arp_tables bridge xen_pciback xen_gntalloc nfsd $
Aug 23 14:29:55 xen-001 kernel: [15199.830906] CPU: 2 PID: 11441 Comm: kworker/u48:2 Not tainted 4.9.39-29.el6.x86_64 #1
Aug 23 14:29:55 xen-001 kernel: [15199.831383] Hardware name: Dell Inc. PowerEdge C6220/03C9JJ, BIOS 1.1.19 02/25/2013
Aug 23 14:29:55 xen-001 kernel: [15199.831867] Workqueue: events_unbound flush_to_ldisc
Aug 23 14:29:55 xen-001 kernel: [15199.832197] task: ffff88004f5cd240 task.stack: ffffc90060ebc000
Aug 23 14:29:55 xen-001 kernel: [15199.832470] RIP: e030:[<ffffffff8152e6a4>]  [<ffffffff8152e6a4>] n_tty_receive_buf_common+0xa4/0x1f0
Aug 23 14:29:55 xen-001 kernel: [15199.833011] RSP: e02b:ffffc90060ebfb08  EFLAGS: 00010296
Aug 23 14:29:55 xen-001 kernel: [15199.833281] RAX: 0000000000002260 RBX: 0000000000000000 RCX: 0000000000000004
Aug 23 14:29:55 xen-001 kernel: [15199.833556] RDX: 0000000000000000 RSI: ffff88006f9ef020 RDI: ffff8801c8c5f0d8
Aug 23 14:29:55 xen-001 kernel: [15199.833830] RBP: ffffc90060ebfb78 R08: 0000000000000001 R09: ffffffff81f0af00
Aug 23 14:29:55 xen-001 kernel: [15199.834105] R10: 0000000000007ff0 R11: 0000000000000078 R12: 0000000000000004
Aug 23 14:29:55 xen-001 kernel: [15199.834378] R13: ffff8801c8c5f000 R14: 0000000000000000 R15: ffff88006f9ef020
Aug 23 14:29:55 xen-001 kernel: [15199.834657] FS:  00007f65711087c0(0000) GS:ffff880201a80000(0000) knlGS:0000000000000000
Aug 23 14:29:55 xen-001 kernel: [15199.835131] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 23 14:29:55 xen-001 kernel: [15199.835402] CR2: 0000000000002260 CR3: 00000001f6dbe000 CR4: 0000000000042660
Aug 23 14:29:55 xen-001 kernel: [15199.835676] Stack:
Aug 23 14:29:55 xen-001 kernel: [15199.835938]  ffffc90060ebfb38 0000000000000000 ffff8801c8c5f0d8 0000000101a99970
Aug 23 14:29:55 xen-001 kernel: [15199.836659]  0000000000002260 000000004f5cf3c0 ffff880201a97000 0000000000000000
Aug 23 14:29:55 xen-001 kernel: [15199.837369]  ffff880201a8b890 ffff88006f9ef000 ffff880008932200 ffffc90060ebfbf8
Aug 23 14:29:55 xen-001 kernel: [15199.838081] Call Trace:
Aug 23 14:29:55 xen-001 kernel: [15199.838355]  [<ffffffff8152e804>] n_tty_receive_buf2+0x14/0x20
Aug 23 14:29:55 xen-001 kernel: [15199.838627]  [<ffffffff81531533>] tty_ldisc_receive_buf+0x23/0x50
Aug 23 14:29:55 xen-001 kernel: [15199.838900]  [<ffffffff81531958>] flush_to_ldisc+0xc8/0x100
Aug 23 14:29:55 xen-001 kernel: [15199.839177]  [<ffffffff8102eb3c>] ? __switch_to+0x1dc/0x680
Aug 23 14:29:55 xen-001 kernel: [15199.839454]  [<ffffffff810c0490>] process_one_work+0x170/0x500
Aug 23 14:29:55 xen-001 kernel: [15199.839730]  [<ffffffff818d4658>] ? __schedule+0x238/0x530
Aug 23 14:29:55 xen-001 kernel: [15199.840008]  [<ffffffff818d4a3a>] ? schedule+0x3a/0xa0
Aug 23 14:29:55 xen-001 kernel: [15199.840280]  [<ffffffff810c1426>] worker_thread+0x166/0x580
Aug 23 14:29:55 xen-001 kernel: [15199.840554]  [<ffffffff810e6209>] ? put_prev_entity+0x29/0x140
Aug 23 14:29:55 xen-001 kernel: [15199.840826]  [<ffffffff818d4658>] ? __schedule+0x238/0x530
Aug 23 14:29:55 xen-001 kernel: [15199.841099]  [<ffffffff810d3882>] ? default_wake_function+0x12/0x20
Aug 23 14:29:55 xen-001 kernel: [15199.841373]  [<ffffffff810c12c0>] ? maybe_create_worker+0x120/0x120
Aug 23 14:29:55 xen-001 kernel: [15199.841646]  [<ffffffff818d4a3a>] ? schedule+0x3a/0xa0
Aug 23 14:29:55 xen-001 kernel: [15199.841919]  [<ffffffff818d8826>] ? _raw_spin_unlock_irqrestore+0x16/0x20
Aug 23 14:29:55 xen-001 kernel: [15199.842193]  [<ffffffff810c12c0>] ? maybe_create_worker+0x120/0x120
Aug 23 14:29:55 xen-001 kernel: [15199.842468]  [<ffffffff810c62c5>] kthread+0xe5/0x100
Aug 23 14:29:55 xen-001 kernel: [15199.842741]  [<ffffffff810c61e0>] ? __kthread_init_worker+0x40/0x40
Aug 23 14:29:55 xen-001 kernel: [15199.843017]  [<ffffffff818d8f55>] ret_from_fork+0x25/0x30
Aug 23 14:29:55 xen-001 kernel: [15199.843288] Code: 89 fe 4c 89 ef 89 45 98 e8 aa fb ff ff 8b 45 98 48 63 d0 48 85 db 48 8d 0c 13 48 0f 45 d9 01 45 bc 49 01 d7 41 29 c4 48 8b 45 b0 <48> 8b 30 48 89 75 c0 49 8b 0e 8d 96 00 10 00 00$
Aug 23 14:29:55 xen-001 kernel: [15199.847901] RIP  [<ffffffff8152e6a4>] n_tty_receive_buf_common+0xa4/0x1f0
Aug 23 14:29:55 xen-001 kernel: [15199.848244]  RSP <ffffc90060ebfb08>
Aug 23 14:29:55 xen-001 kernel: [15199.848511] CR2: 0000000000002260
Aug 23 14:29:55 xen-001 kernel: [15199.848781] ---[ end trace f98e9cf48e3a6111 ]---
Aug 23 14:29:55 xen-001 kernel: [15199.849242] BUG: unable to handle kernel paging request at ffffffffffffffd8
Aug 23 14:29:55 xen-001 kernel: [15199.849638] IP: [<ffffffff810c5aa0>] kthread_data+0x10/0x20
Aug 23 14:29:55 xen-001 kernel: [15199.849977] PGD 1e0a067
Aug 23 14:29:55 xen-001 kernel: [15199.850044] PUD 1e0c067




kernel_oops.txt (13,931 bytes)   

Issue History

Date Modified Username Field Change
2017-08-24 22:44 sh4d0w New Issue
2017-08-24 22:44 sh4d0w File Added: kernel_oops.txt