View Issue Details

IDProjectCategoryView StatusLast Update
0013731CentOS-7kernelpublic2020-03-07 04:07
Reporterfelix 
PriorityurgentSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version7.1-1503 
Target VersionFixed in Version 
Summary0013731: kernel panic when cpu running with high load for serveral days in cgroup
Descriptionkernel version: 3.10.0-327.28.3
[2342438.135131] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[2342438.135200] IP: [<ffffffff812f9d41>] rb_next+0x1/0x50
[2342438.135247] PGD 3f1d705067 PUD 36b9189067 PMD 0
[2342438.135292] Oops: 0000 [#1] SMP
[2342438.135324] Modules linked in: iptable_mangle ip6table_mangle veth ip6table_nat ip6t_rpfilter iptable_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6table_raw xt_multiport ip_set_hash_net ip_set nfnetlink ip6_tables xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop dm_mirror dm_region_hash dm_log intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas ipmi_devintf sb_edac sg edac_core lpc_ich pcspkr mei_me shpchp mei mfd_core ipmi_si ipmi_msghandler acpi_power_meter
[2342438.135982] wmi dm_mod ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper crct10dif_pclmul ttm crct10dif_common crc32c_intel drm ixgbe ahci mdio libahci ptp i2c_core libata pps_core megaraid_sas dca
[2342438.136213] CPU: 4 PID: 151342 Comm: exe Not tainted 3.10.0-327.28.3.el7.x86_64 #1
[2342438.136270] Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.4.3 01/17/2017
[2342438.136328] task: ffff883e8382b980 ti: ffff883c24120000 task.ti: ffff883c24120000
[2342438.136383] RIP: 0010:[<ffffffff812f9d41>] [<ffffffff812f9d41>] rb_next+0x1/0x50
[2342438.136444] RSP: 0018:ffff883c24123c28 EFLAGS: 00010046
[2342438.136485] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[2342438.136538] RDX: 0000000000000001 RSI: ffff883f15be6828 RDI: 0000000000000010
[2342438.136592] RBP: ffff883c24123c70 R08: 0000000000000000 R09: 0000000000000001
[2342438.136645] R10: 0000000000000001 R11: 00000000ffffffff R12: ffff883efdabe000
[2342438.136697] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[2342438.136751] FS: 00007f7bef7fe700(0000) GS:ffff883f7c700000(0000) knlGS:0000000000000000
[2342438.136810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2342438.136854] CR2: 0000000000000010 CR3: 0000003efd595000 CR4: 00000000003407e0
[2342438.136907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2342438.136960] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2342438.137013] Stack:
[2342438.137032] ffff883c24123c70 ffffffff810be589 ffff883c24123c60 ffff883f7c7167c0
[2342438.137102] ffff883e8382bf60 ffff883f7c7167c0 0000000000000004 ffff883c24123de0
[2342438.137171] ffffc90028c16f40 ffff883c24123cd0 ffffffff8163b35a ffff883e8382b980
[2342438.137239] Call Trace:
[2342438.137269] [<ffffffff810be589>] ? pick_next_task_fair+0x129/0x1d0
[2342438.137324] [<ffffffff8163b35a>] __schedule+0x12a/0x900
[2342438.137368] [<ffffffff8163bb59>] schedule+0x29/0x70
[2342438.137413] [<ffffffff810e26e4>] futex_wait_queue_me+0xc4/0x120
[2342438.137463] [<ffffffff810e3259>] futex_wait+0x179/0x280
[2342438.137511] [<ffffffff810b5df5>] ? check_preempt_curr+0x75/0xa0
[2342438.137561] [<ffffffff810b5fcd>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[2342438.137617] [<ffffffff810b8766>] ? try_to_wake_up+0x1b6/0x300
[2342438.137666] [<ffffffff811694ab>] ? unlock_page+0x2b/0x30
[2342438.139625] [<ffffffff810e52ee>] do_futex+0xfe/0x5b0
[2342438.141578] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
[2342438.143526] [<ffffffff810e5820>] SyS_futex+0x80/0x180
[2342438.145467] [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[2342438.147360] Code: e5 48 85 c0 75 07 eb 19 66 90 48 89 d0 48 8b 50 10 48 85 d2 75 f4 48 8b 50 08 48 85 d2 75 eb 5d c3 31 c0 5d c3 0f 1f 44 00 00 55 <48> 8b 17 48 89 e5 48 39 d7 74 3b 48 8b 47 08 48 85 c0 75 0e eb
[2342438.151278] RIP [<ffffffff812f9d41>] rb_next+0x1/0x50
[2342438.153154] RSP <ffff883c24123c28>
[2342438.155014] CR2: 0000000000000010
Steps To Reproducerunning nginx in cgroup with high cpu load,it crashed occasionally
Tagskernel panic
abrt_hash
URL

Activities

felix

felix

2017-08-29 03:18

reporter  

vmcore-dmesg.txt (4,528 bytes)
[2341958.156530] XFS (dm-17): Mounting V4 Filesystem
[2341958.167097] XFS (dm-17): Ending clean mount
[2341958.169344] XFS (dm-17): Unmounting Filesystem
[2341958.235172] XFS (dm-17): Mounting V4 Filesystem
[2341958.242029] XFS (dm-17): Ending clean mount
[2342438.088738] XFS (dm-18): Mounting V4 Filesystem
[2342438.098236] XFS (dm-18): Ending clean mount
[2342438.102966] XFS (dm-18): Unmounting Filesystem
[2342438.135131] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[2342438.135200] IP: [<ffffffff812f9d41>] rb_next+0x1/0x50
[2342438.135247] PGD 3f1d705067 PUD 36b9189067 PMD 0 
[2342438.135292] Oops: 0000 [#1] SMP 
[2342438.135324] Modules linked in: iptable_mangle ip6table_mangle veth ip6table_nat ip6t_rpfilter iptable_raw nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6table_raw xt_multiport ip_set_hash_net ip_set nfnetlink ip6_tables xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter nf_nat nf_conntrack bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop dm_mirror dm_region_hash dm_log intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas ipmi_devintf sb_edac sg edac_core lpc_ich pcspkr mei_me shpchp mei mfd_core ipmi_si ipmi_msghandler acpi_power_meter
[2342438.135982]  wmi dm_mod ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper crct10dif_pclmul ttm crct10dif_common crc32c_intel drm ixgbe ahci mdio libahci ptp i2c_core libata pps_core megaraid_sas dca
[2342438.136213] CPU: 4 PID: 151342 Comm: exe Not tainted 3.10.0-327.28.3.el7.x86_64 #1
[2342438.136270] Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.4.3 01/17/2017
[2342438.136328] task: ffff883e8382b980 ti: ffff883c24120000 task.ti: ffff883c24120000
[2342438.136383] RIP: 0010:[<ffffffff812f9d41>]  [<ffffffff812f9d41>] rb_next+0x1/0x50
[2342438.136444] RSP: 0018:ffff883c24123c28  EFLAGS: 00010046
[2342438.136485] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[2342438.136538] RDX: 0000000000000001 RSI: ffff883f15be6828 RDI: 0000000000000010
[2342438.136592] RBP: ffff883c24123c70 R08: 0000000000000000 R09: 0000000000000001
[2342438.136645] R10: 0000000000000001 R11: 00000000ffffffff R12: ffff883efdabe000
[2342438.136697] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[2342438.136751] FS:  00007f7bef7fe700(0000) GS:ffff883f7c700000(0000) knlGS:0000000000000000
[2342438.136810] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2342438.136854] CR2: 0000000000000010 CR3: 0000003efd595000 CR4: 00000000003407e0
[2342438.136907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2342438.136960] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2342438.137013] Stack:
[2342438.137032]  ffff883c24123c70 ffffffff810be589 ffff883c24123c60 ffff883f7c7167c0
[2342438.137102]  ffff883e8382bf60 ffff883f7c7167c0 0000000000000004 ffff883c24123de0
[2342438.137171]  ffffc90028c16f40 ffff883c24123cd0 ffffffff8163b35a ffff883e8382b980
[2342438.137239] Call Trace:
[2342438.137269]  [<ffffffff810be589>] ? pick_next_task_fair+0x129/0x1d0
[2342438.137324]  [<ffffffff8163b35a>] __schedule+0x12a/0x900
[2342438.137368]  [<ffffffff8163bb59>] schedule+0x29/0x70
[2342438.137413]  [<ffffffff810e26e4>] futex_wait_queue_me+0xc4/0x120
[2342438.137463]  [<ffffffff810e3259>] futex_wait+0x179/0x280
[2342438.137511]  [<ffffffff810b5df5>] ? check_preempt_curr+0x75/0xa0
[2342438.137561]  [<ffffffff810b5fcd>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[2342438.137617]  [<ffffffff810b8766>] ? try_to_wake_up+0x1b6/0x300
[2342438.137666]  [<ffffffff811694ab>] ? unlock_page+0x2b/0x30
[2342438.139625]  [<ffffffff810e52ee>] do_futex+0xfe/0x5b0
[2342438.141578]  [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
[2342438.143526]  [<ffffffff810e5820>] SyS_futex+0x80/0x180
[2342438.145467]  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
[2342438.147360] Code: e5 48 85 c0 75 07 eb 19 66 90 48 89 d0 48 8b 50 10 48 85 d2 75 f4 48 8b 50 08 48 85 d2 75 eb 5d c3 31 c0 5d c3 0f 1f 44 00 00 55 <48> 8b 17 48 89 e5 48 39 d7 74 3b 48 8b 47 08 48 85 c0 75 0e eb 
[2342438.151278] RIP  [<ffffffff812f9d41>] rb_next+0x1/0x50
[2342438.153154]  RSP <ffff883c24123c28>
[2342438.155014] CR2: 0000000000000010
vmcore-dmesg.txt (4,528 bytes)
TrevorH

TrevorH

2017-08-29 11:04

manager   ~0029957

That's a 7.2 kernel and not even the latest of those. 7.4 is already in the CR repo so your first step is to yum update from there and retest to see if the problem still exists.
nobody_caq8

nobody_caq8

2020-03-07 02:43

reporter   ~0036478

3.10.0-957.27.2.el7.x86_64 also have the same problem´╝Ü
[61767.021070] CPU: 33 PID: 437013 Comm: java Kdump: loaded Not tainted 3.10.0-957.27.2.el7.x86_64 #1
[61767.021119] Hardware name: Dell Inc. PowerEdge R740xd/00WGD1, BIOS 1.4.5 03/30/2018
[61767.021161] task: ffff8fe556e6d140 ti: ffff8ff96dc3c000 task.ti: ffff8ff96dc3c000
[61767.021201] RIP: 0010:[<ffffffff9737ed31>] [<ffffffff9737ed31>] rb_next+0x1/0x50
[61767.021247] RSP: 0018:ffff8ff96dc3fbe0 EFLAGS: 00010046
[61767.021277] RAX: 0000000000000000 RBX: ffff90003fa21400 RCX: 0000000000000000
[61767.021316] RDX: 0000000000000001 RSI: ffff9014a1f60428 RDI: 0000000000000010
[61767.021354] RBP: ffff8ff96dc3fc28 R08: 0000000000000000 R09: 0000000000000001
[61767.021392] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000
[61767.021431] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[61767.021470] FS: 00007f6194070700(0000) GS:ffff90157c200000(0000) knlGS:0000000000000000
[61767.021512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[61767.021543] CR2: 0000000000000010 CR3: 0000003c715b4000 CR4: 00000000007607e0
[61767.021582] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[61767.021620] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[61767.021660] PKRU: 55555554
[61767.021676] Call Trace:
[61767.021699] [<ffffffff970dfb82>] ? pick_next_task_fair+0x102/0x1b0
[61767.021736] [<ffffffff977697ca>] __schedule+0x13a/0x860
[61767.021768] [<ffffffff97769f19>] schedule+0x29/0x70
[61767.021799] [<ffffffff9710e286>] futex_wait_queue_me+0xc6/0x130
[61767.021833] [<ffffffff9710ef6b>] futex_wait+0x17b/0x280
[61767.021864] [<ffffffff9710deb8>] ? get_futex_key+0x1c8/0x2c0
[61767.021898] [<ffffffff97110cb6>] do_futex+0x106/0x5a0
[61767.021927] [<ffffffff971111d0>] SyS_futex+0x80/0x190
[61767.021958] [<ffffffff97776ddb>] system_call_fastpath+0x22/0x27
[61767.021991] Code: e5 48 85 c0 75 07 eb 19 66 90 48 89 d0 48 8b 50 10 48 85 d2 75 f4 48 8b 50 08 48 85 d2 75 eb 5d c3 31 c0 5d c3 0f 1f 44 00 00 55 <48> 8b 17 48 89 e5 48 39 d7 74 3b 48 8b 47 08 48 85 c0 75 0e eb
[61767.022206] RIP [<ffffffff9737ed31>] rb_next+0x1/0x50
[61767.022241] RSP <ffff8ff96dc3fbe0>
[61767.022261] CR2: 0000000000000010
TrevorH

TrevorH

2020-03-07 04:07

manager   ~0036479

Good. Still out of date. Run yum update to get onto 3.10.0-1062.12.1

Issue History

Date Modified Username Field Change
2017-08-29 03:18 felix New Issue
2017-08-29 03:18 felix File Added: vmcore-dmesg.txt
2017-08-29 03:18 felix Tag Attached: kernel panic
2017-08-29 11:04 TrevorH Note Added: 0029957
2020-03-07 02:43 nobody_caq8 Note Added: 0036478
2020-03-07 04:07 TrevorH Note Added: 0036479