View Issue Details

IDProjectCategoryView StatusLast Update
0015103CentOS-7kernelpublic2018-08-02 10:18
ReporterMohammed Salman 
PrioritynormalSeveritycrashReproducibilityrandom
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0015103: Kernel crash seen due to cgroup
DescriptionHave a Centos 7.3 VM deployed on ESXi. System abruptly crashed and rebooted. Observed a vmcore-dmesg file. Any help regarding the fix or RCA is appreciated.

 8410.965246] conntrack: generic helper won't handle protocol 47. Please consider loading the specific helper module.
[362781.469432] nr_pdflush_threads exported in /proc is scheduled for removal
[362855.836789] blk_update_request: I/O error, dev fd0, sector 0
[516913.460994] hrtimer: interrupt took 153474 ns
[1613776.834348] ------------[ cut here ]------------
[1613776.835112] kernel BUG at kernel/cgroup.c:895!
[1613776.835707] invalid opcode: 0000 [#1] SMP
[1613776.836327] Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 dell_rbu nf_conntrack_netlink nfnetlink br_netfilter bridge stp llc overlay dcdbas binfmt_misc ip_gre ip_tunnel gre vmw_vsock_vmci_transport vsock iptable_raw ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_REDIRECT nf_nat_redirect xt_addrtype iptable_nat xt_CT nf_nat_ipv4 nf_nat ip6table_raw nf_conntrack_ipv6 nf_conntrack_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 xt_limit ip6table_filter xt_connmark ip6_tables xt_conntrack nf_conntrack iptable_filter xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key fuse intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg shpchp vmw_vmci i2c_piix4 parport_pc parport ip_tables ext4 mbcache
[1613776.846623] jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix scsi_transport_spi drm serio_raw mptscsih libata mptbase vmxnet3 i2c_core floppy fjes ecryptfs
[1613776.850663] CPU: 2 PID: 22493 Comm: kworker/2:1 Not tainted 3.10.0-514.26.2.aruba.el7.x86_64 #1
[1613776.851868] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
[1613776.853419] Workqueue: cgroup_destroy css_dput_fn
[1613776.854103] task: ffff880209ab9f60 ti: ffff88000e78c000 task.ti: ffff88000e78c000
[1613776.855203] RIP: 0010:[<ffffffff8110c750>] [<ffffffff8110c750>] cgroup_diput+0xc0/0xf0
[1613776.856318] RSP: 0018:ffff88000e78fd78 EFLAGS: 00010246
[1613776.857060] RAX: 0000000000000000 RBX: ffff88017fbcae40 RCX: dead000000000200
[1613776.858043] RDX: 0000000000000000 RSI: ffff8802371b8b90 RDI: ffff8800362af600
[1613776.859036] RBP: ffff88000e78fda0 R08: ffff88017fbcaed0 R09: dff6b911633c3420
[1613776.860026] R10: dff6b911633c3420 R11: 0000000000000000 R12: ffff8802370bdb40
[1613776.861009] R13: ffff8802371b8b90 R14: ffff88017fbcae98 R15: ffff8802371b8b90
[1613776.861994] FS: 0000000000000000(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
[1613776.863107] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[1613776.863888] CR2: 00007fd238cc0810 CR3: 00000000019be000 CR4: 00000000000007e0
[1613776.864902] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1613776.865906] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1613776.866911] Stack:
[1613776.867239] ffff88017fbcae40 ffff8802370bdb40 ffff8802371b8b90 ffff88017fbcae98
[1613776.868345] 0000000000000080 ffff88000e78fdd0 ffffffff81215d16 ffff88017fbcae40
[1613776.869468] ffff88017fbcae98 ffff88023fc96480 ffff88023fc9b100 ffff88000e78fdf0
[1613776.870622] Call Trace:
[1613776.871010] [<ffffffff81215d16>] dentry_kill+0x146/0x1b0
[1613776.871737] [<ffffffff81215ddc>] dput+0x5c/0xd0
[1613776.872384] [<ffffffff8110a91c>] cgroup_dput.isra.21+0x1c/0x30
[1613776.873185] [<ffffffff8110a94d>] css_dput_fn+0x1d/0x20
[1613776.873908] [<ffffffff810a845b>] process_one_work+0x17b/0x470
[1613776.874692] [<ffffffff810a9296>] worker_thread+0x126/0x410
[1613776.875459] [<ffffffff810a9170>] ? rescuer_thread+0x460/0x460
[1613776.876249] [<ffffffff810b0a4f>] kthread+0xcf/0xe0
[1613776.876924] [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[1613776.877801] [<ffffffff81697798>] ret_from_fork+0x58/0x90
[1613776.878533] [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[1613776.879421] Code: 41 5e 41 5f 5d c3 0f 1f 44 00 00 48 8b 7f 78 48 8b 07 a8 01 74 15 48 81 c7 38 01 00 00 48 c7 c6 70 97 10 81 e8 62 c8 02 00 eb c8 <0f> 0b 49 8b 4e 18 48 c7 c2 33 93 8d 81 be 87 03 00 00 48 c7 c7
[1613776.883306] RIP [<ffffffff8110c750>] cgroup_diput+0xc0/0xf0
Tagscgroup crash panic
abrt_hash
URL

Activities

pgreco

pgreco

2018-07-25 22:08

developer   ~0032358

First, I would recommend updating to 7.5, most likely this has been fixed.
Second, I see that your kernels says 3.10.0-514.26.2.aruba.el7.x86_64, which tells me it is not a standard kernel from centos, but a rebuild. This may also be related to your problems.
icymoon

icymoon

2018-08-02 05:04

reporter   ~0032416

I get the same crash info on a new version, 3.10.0-693.11.1.el7.x86_64
1851 [15639061.912756] kernel BUG at kernel/cgroup.c:895!
1852 [15639061.912773] invalid opcode: 0000 [#1] SMP
1853 [15639061.912790] Modules linked in: ipmi_watchdog ipmi_poweroff 8021q garp mrp stp llc binfmt_misc sb_edac edac_core coretemp intel_rap l iosf_mbi kvm_intel xfs kvm libcrc32c irqbypass crc32_pclmul ghash_clmulni_intel ses aesni_intel lrw enclosure sg gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support mxm_wmi ipmi_si mei_me ipmi_devintf mei lpc_ich i2c_i801 pcspkr ioatdma ipmi_msghandler shpchp acpi_power_meter wmi ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect s ysimgblt fb_sys_fops ttm drm ixgbe mpt3sas igb crct10dif_pclmul crct10dif_common crc32c_intel mdio raid_class ptp i2c_algo_bit scsi_tran sport_sas pps_core i2c_core dca
1854 [15639061.913197] CPU: 3 PID: 173822 Comm: kworker/3:3 Not tainted 3.10.0-693.11.1.el7.x86_64 #1
pgreco

pgreco

2018-08-02 10:18

developer   ~0032418

@icymoon, that is newer (7.4), but still not new enough (7.5).
We need you to be able to reproduce this bug using kernel-3.10.0-862.9.1.el7 or at least kernel-3.10.0-862.6.3.el7.

Issue History

Date Modified Username Field Change
2018-07-25 21:24 Mohammed Salman New Issue
2018-07-25 21:24 Mohammed Salman Tag Attached: cgroup crash panic
2018-07-25 22:08 pgreco Note Added: 0032358
2018-08-02 05:04 icymoon Note Added: 0032416
2018-08-02 10:18 pgreco Note Added: 0032418