2017-09-22 08:04 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013682CentOS-7kernelpublic2017-09-13 12:54
Reportermassimo.marin 
PriorityhighSeveritycrashReproducibilityrandom
StatusnewResolutionopen 
Platformx86_64OSCentOS Linux release 7.3.1611 (COS Version7.3.1611
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0013682: panic with kernel-3.10.0-514.21.2.el7.x86_64 and kernel-3.10.0-514.10.2.el7.x86_64 OOPS audit_copy_inode+0x29/0xb0
DescriptionWe are experiencing frequent crashes, at least 8 on different servers in the last 15 days, with the following event

[1230784.665862] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040
[1230784.665928] IP: [<ffffffff81119d29>] audit_copy_inode+0x29/0xb0
[1230784.665979] PGD 178c625067 PUD 13e4adc067 PMD 0
[1230784.666018] Oops: 0000 [#1] SMP
[1230784.666043] Modules linked in: nfsv3 nfs fscache 8021q garp mrp stp llc bonding intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclm
ul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt ipmi_ssif ses ipmi_devintf enclosure iTCO_vendor_support
scsi_transport_sas pcspkr sg acpi_power_meter i7core_edac ipmi_si gpio_ich dcdbas ipmi_msghandler edac_core shpchp lpc_ich nfsd auth_rpcgss nfs_a
cl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ixgbe igb mdio ptp pps_core i2c_algo_bit megaraid_sas dca
i2c_core bnx2 fjes
[1230784.666602] CPU: 8 PID: 73502 Comm: lmtp Tainted: G I ------------ 3.10.0-514.21.2.el7.x86_64 #1
[1230784.666648] Hardware name: Dell Inc. PowerEdge R710/0MD99X, BIOS 3.0.0 01/31/2011
[1230784.666684] task: ffff880cc3f56dd0 ti: ffff8800256b8000 task.ti: ffff8800256b8000
[1230784.666716] RIP: 0010:[<ffffffff81119d29>] [<ffffffff81119d29>] audit_copy_inode+0x29/0xb0
[1230784.666755] RSP: 0018:ffff8800256bbc78 EFLAGS: 00010246
[1230784.666779] RAX: 0000000000000000 RBX: ffff880bdd7c6060 RCX: 0000000000000000
[1230784.666817] RDX: 0000000000000000 RSI: ffff880bdd7c609c RDI: ffff880bdd7c6060
[1230784.666848] RBP: ffff8800256bbca8 R08: 0000000000000000 R09: 0000000000000000
[1230784.666877] R10: 0000000000000400 R11: 0000000000001400 R12: ffff8811d04fe240
[1230784.666915] R13: ffff880bdd7c6060 R14: 0000000012120c42 R15: 0000000000000000
[1230784.666959] FS: 00007fbe4327b740(0000) GS:ffff88180f300000(0000) knlGS:0000000000000000
[1230784.667005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1230784.667038] CR2: 0000000000000040 CR3: 00000017d468b000 CR4: 00000000000007e0
[1230784.667080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1230784.667122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1230784.667163] Stack:
[1230784.667178] 0000000000000000 0000000000000246 ffff880cf03a2f10 000000007113a35f
[1230784.667229] ffff88016d9f9980 ffff880bdd7c6000 ffff8800256bbcf8 ffffffff8111f7aa
[1230784.667280] 00000000810c54a3 ffff880bdd7c6000 ffff8811d04fe240 ffff8800256bbf28
[1230784.667328] Call Trace:
[1230784.667350] [<ffffffff8111f7aa>] __audit_inode+0x18a/0x3c0
[1230784.667385] [<ffffffff8120de34>] do_last+0xcd4/0x12a0
[1230784.667418] [<ffffffff8120e4c2>] path_openat+0xc2/0x490
[1230784.667453] [<ffffffff81252bf4>] ? locks_free_lock+0x64/0x70
[1230784.667488] [<ffffffff81253b0a>] ? __posix_lock_file+0x20a/0x560
[1230784.667529] [<ffffffff8121063b>] do_filp_open+0x4b/0xb0
[1230784.667562] [<ffffffff8111f60f>] ? __audit_getname+0x9f/0xb0
[1230784.667600] [<ffffffff8121d2b7>] ? __alloc_fd+0xa7/0x130
[1230784.667636] [<ffffffff811fdb43>] do_sys_open+0xf3/0x1f0
[1230784.667670] [<ffffffff811fdc5e>] SyS_open+0x1e/0x20
[1230784.667704] [<ffffffff81697749>] system_call_fastpath+0x16/0x1b
[1230784.667741] Code: 00 00 66 66 66 66 90 55 48 89 e5 41 54 49 89 f4 48 8d 77 3c 53 48 89 fb 48 83 ec 20 65 48 8b 04 25 28 00 00 00 48 89 45 e8
 31 c0 <48> 8b 42 40 48 89 47 20 48 8b 42 28 8b 40 10 89 47 28 0f b7 02
[1230784.667990] RIP [<ffffffff81119d29>] audit_copy_inode+0x29/0xb0
[1230784.669487] RSP <ffff8800256bbc78>
[1230784.670976] CR2: 0000000000000040


[1052537.891368] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040
[1052537.891431] IP: [<ffffffff81119899>] audit_copy_inode+0x29/0xb0
[1052537.891477] PGD c83b04067 PUD c7fae1067 PMD 0
[1052537.891512] Oops: 0000 [#1] SMP
[1052537.891537] Modules linked in: nfsv3 nfs fscache 8021q garp mrp stp llc bonding intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclm
ul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ses enclosure scsi_transport_sas cryptd iTCO_wdt iTCO_vendor_support sg i
pmi_ssif gpio_ich shpchp lpc_ich i7core_edac edac_core pcspkr ipmi_devintf dcdbas ipmi_si acpi_power_meter ipmi_msghandler nfsd auth_rpcgss nfs_a
cl lockd binfmt_misc grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect
crct10dif_pclmul crct10dif_common sysimgblt fb_sys_fops crc32c_intel ttm serio_raw ixgbe drm igb mdio ptp pps_core i2c_algo_bit megaraid_sas i2c_
core dca bnx2 fjes
[1052537.892070] CPU: 7 PID: 126878 Comm: lmtp Tainted: G I ------------ 3.10.0-514.10.2.el7.x86_64 #1
[1052537.892124] Hardware name: Dell Inc. PowerEdge R710/0MD99X, BIOS 6.1.0 10/18/2011
[1052537.892165] task: ffff8816a2a08000 ti: ffff88006eeec000 task.ti: ffff88006eeec000
[1052537.892208] RIP: 0010:[<ffffffff81119899>] [<ffffffff81119899>] audit_copy_inode+0x29/0xb0
[1052537.892258] RSP: 0018:ffff88006eeefc78 EFLAGS: 00010246
[1052537.892288] RAX: 0000000000000000 RBX: ffff880d3c8e1460 RCX: 0000000000000000
[1052537.892326] RDX: 0000000000000000 RSI: ffff880d3c8e149c RDI: ffff880d3c8e1460
[1052537.892368] RBP: ffff88006eeefca8 R08: ffffffff816af9a0 R09: 0000000000000000
[1052537.892408] R10: 0000000000000400 R11: 0000000000000c00 R12: ffff88155f317e00
[1052537.892449] R13: ffff880d3c8e1460 R14: 00000000112fc2cc R15: 0000000000000000
[1052537.892489] FS: 00007fce9d3f9740(0000) GS:ffff880c0fac0000(0000) knlGS:0000000000000000
[1052537.892534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1052537.892563] CR2: 0000000000000040 CR3: 0000000c7fef1000 CR4: 00000000000007e0
[1052537.892606] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1052537.892650] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1052537.892688] Stack:
[1052537.892704] ffff88180f200000 0000000000000246 ffff880c817e0fb0 00000000d54048c9
[1052537.892744] ffff88016db2ee40 ffff880d3c8e1400 ffff88006eeefcf8 ffffffff8111f31a
[1052537.892792] 00000000810c5043 ffff880d3c8e1400 ffff88155f317e00 ffff88006eeeff28
[1052537.892845] Call Trace:
[1052537.892868] [<ffffffff8111f31a>] __audit_inode+0x18a/0x3c0
[1052537.892908] [<ffffffff8120d7e4>] do_last+0xcd4/0x12a0
[1052537.892939] [<ffffffff8120de72>] path_openat+0xc2/0x490
[1052537.892971] [<ffffffff81252594>] ? locks_free_lock+0x64/0x70
[1052537.893001] [<ffffffff812534aa>] ? __posix_lock_file+0x20a/0x560
[1052537.893040] [<ffffffff8120ffeb>] do_filp_open+0x4b/0xb0
[1052537.893077] [<ffffffff8111f17f>] ? __audit_getname+0x9f/0xb0
[1052537.893117] [<ffffffff8121cc67>] ? __alloc_fd+0xa7/0x130
[1052537.893144] [<ffffffff811fd4f3>] do_sys_open+0xf3/0x1f0
[1052537.893169] [<ffffffff811fd60e>] SyS_open+0x1e/0x20
[1052537.893200] [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
[1052537.893230] Code: 00 00 66 66 66 66 90 55 48 89 e5 41 54 49 89 f4 48 8d 77 3c 53 48 89 fb 48 83 ec 20 65 48 8b 04 25 28 00 00 00 48 89 45 e8
 31 c0 <48> 8b 42 40 48 89 47 20 48 8b 42 28 8b 40 10 89 47 28 0f b7 02
[1052537.893550] RIP [<ffffffff81119899>] audit_copy_inode+0x29/0xb0
[1052537.894818] RSP <ffff88006eeefc78>
[1052537.896041] CR2: 0000000000000040
TagsNo tags attached.
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0029891

tigalch (manager)

What about the latest kernel 3.10.0-514.26.2.el7?

~0029892

massimo.marin (reporter)

Unfortunately we had other issue with the last kernel.
the backend's servers were unreachable by frontend and the TCP connection from frontend gone in timeout

~0030045

massimo.marin (reporter)

same issue with kernel 3.10.0-514.26.2.el7.x86_64

[763722.867306] BUG: unable to handle kernel NULL pointer dereference at 0000000000000040
[763722.868824] IP: [<ffffffff81119d39>] audit_copy_inode+0x29/0xb0
[763722.870221] PGD 671f8e067 PUD 2bfbf7067 PMD 0
[763722.871640] Oops: 0000 [#1] SMP
[763722.882021] CPU: 0 PID: 23729 Comm: lmtp Tainted: G I ------------ 3.10.0-514.26.2.el7.x86_64 #1
[763722.885057] task: ffff880cc3ebbec0 ti: ffff8800286b8000 task.ti: ffff8800286b8000
[763722.886576] RIP: 0010:[<ffffffff81119d39>] [<ffffffff81119d39>] audit_copy_inode+0x29/0xb0
[763722.888131] RSP: 0018:ffff8800286bbc78 EFLAGS: 00010246
[763722.889744] RAX: 0000000000000000 RBX: ffff88011a1d5c60 RCX: 0000000000000000
[763722.891241] RDX: 0000000000000000 RSI: ffff88011a1d5c9c RDI: ffff88011a1d5c60
[763722.892674] RBP: ffff8800286bbca8 R08: ffffffff816b1460 R09: 0000000000000000
[763722.894004] R10: 0000000000000400 R11: 0000000000000800 R12: ffff881785feacc0
[763722.895356] R13: ffff88011a1d5c60 R14: 000000000bc30e70 R15: 0000000000000000
[763722.896673] FS: 00007f357c5e3740(0000) GS:ffff88180f200000(0000) knlGS:0000000000000000
[763722.898050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[763722.899544] CR2: 0000000000000040 CR3: 0000000a59e9c000 CR4: 00000000000007f0
[763722.900929] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[763722.902326] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[763722.903789] Stack:
[763722.905190] 0000000000000000 0000000000000246 ffff88012e121f60 000000003be94051
[763722.906517] ffff88016db02180 ffff88011a1d5c00 ffff8800286bbcf8 ffffffff8111f7ba
[763722.907835] 00000000810c54a3 ffff88011a1d5c00 ffff881785feacc0 ffff8800286bbf28
[763722.909240] Call Trace:
[763722.910483] [<ffffffff8111f7ba>] __audit_inode+0x18a/0x3c0
[763722.911741] [<ffffffff8120deb4>] do_last+0xcd4/0x12a0
[763722.912959] [<ffffffff8120e542>] path_openat+0xc2/0x490
[763722.914192] [<ffffffff81252c74>] ? locks_free_lock+0x64/0x70
[763722.915441] [<ffffffff81253b8a>] ? __posix_lock_file+0x20a/0x560
[763722.916822] [<ffffffff812106bb>] do_filp_open+0x4b/0xb0
[763722.918064] [<ffffffff8111f61f>] ? __audit_getname+0x9f/0xb0
[763722.919254] [<ffffffff8121d337>] ? __alloc_fd+0xa7/0x130
[763722.920415] [<ffffffff811fdbc3>] do_sys_open+0xf3/0x1f0
[763722.921521] [<ffffffff811fdcde>] SyS_open+0x1e/0x20
[763722.922817] [<ffffffff81697809>] system_call_fastpath+0x16/0x1b
[763722.924000] Code: 00 00 66 66 66 66 90 55 48 89 e5 41 54 49 89 f4 48 8d 77 3c 53 48 89 fb 48 83 ec 20 65 48 8b 04 25 28 00 00 00 48 89 45 e8
31 c0 <48> 8b 42 40 48 89 47 20 48 8b 42 28 8b 40 10 89 47 28 0f b7 02
[763722.926370] RIP [<ffffffff81119d39>] audit_copy_inode+0x29/0xb0
[763722.927807] RSP <ffff8800286bbc78>
[763722.929481] CR2: 0000000000000040

~0030046

tru (administrator)

-> Hardware name: Dell Inc. PowerEdge R710/0MD99X, BIOS 6.1.0 10/18/2011

Could you upgrade the bios to the latest version and see if that fixes your issue?
http://www.dell.com/support/home/us/en/04/product-support/product/poweredge-r710/drivers is listing version 6.4.0 from 2013
+Notes

-Issue History
Date Modified Username Field Change
2017-08-17 17:05 massimo.marin New Issue
2017-08-18 05:23 tigalch Note Added: 0029891
2017-08-18 06:13 massimo.marin Note Added: 0029892
2017-09-13 12:22 massimo.marin Note Added: 0030045
2017-09-13 12:54 tru Note Added: 0030046
+Issue History