View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0015612CentOS-7kernelpublic2018-12-19 16:302020-09-27 03:16
Reporterdennisxrow Assigned To 
PriorityhighSeveritycrashReproducibilityrandom
Status newResolutionopen 
OSCentOS Linux release 7.6.1810 
Summary0015612: kernel BUG at lib/idr.c:1163!
DescriptionHello,
every so often, usually every 3 to 6 days, one of our servers panics and reboots with the panic documented below.

It seem that there actually is an article from redhat about this issue here: https://access.redhat.com/solutions/3492911, with the solution being "Upgrade to kernel-3.10.0-957.el7 from Errata RHSA-2018:3083 or later".

However upgrading to kernel-3.10.0-957 did not resolve the issue.

Please note that the error message from redhat differs with the one in my vmcore-dmesg.txt:
"Kernel BUG at lib/idr.c:1157!" vs. "kernel BUG at lib/idr.c:1163!"
I'm not sure if that matters.

It might relates with https://bugs.centos.org/view.php?id=15578 which I created a few days ago, occuring on the same servers aswell.

Any help / guidance is highly appreciated.
Steps To ReproduceProblem occurs randomly every 3 to 6 days.
Additional Informationvmcore-dmesg.txt:
[597693.759915] ------------[ cut here ]------------
[597693.760678] kernel BUG at lib/idr.c:1163!
[597693.760818] invalid opcode: 0000 [#1] SMP
[597693.760974] Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_addrtype ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter xt_conntrack nf_nat nf_conntrack_netlink nf_conntrack overlay(T) ip_set_hash_ip ip_set nfnetlink rpcsec_gss_krb5 auth_rpcgss tcp_diag nfsv4 inet_diag dns_resolver nfs lockd grace fscache sunrpc ppdev sb_edac iosf_mbi kvm_intel kvm vmw_balloon irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 br_netfilter bridge stp llc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif
[597693.762917] crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmwgfx drm_kms_helper floppy syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm drm ata_piix libata vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
[597693.764319] CPU: 7 PID: 30459 Comm: kworker/7:3 Kdump: loaded Tainted: G ------------ T 3.10.0-957.1.3.el7.x86_64 #1
[597693.765071] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
[597693.765895] Workqueue: events free_work
[597693.766328] task: ffff923dd7631040 ti: ffff923ad25a0000 task.ti: ffff923ad25a0000
[597693.766775] RIP: 0010:[<ffffffff82b76ee1>] [<ffffffff82b76ee1>] ida_simple_remove+0x41/0x50
[597693.767253] RSP: 0018:ffff923ad25a3da0 EFLAGS: 00010286
[597693.767734] RAX: ffff9238920c3800 RBX: 00000000ffffffff RCX: 0000000000002fff
[597693.768225] RDX: 0000000000002fff RSI: 00000000ffffffff RDI: ffffffff83a097c0
[597693.768723] RBP: ffff923ad25a3db8 R08: 0004000100024460 R09: 0002444100024420
[597693.769231] R10: 0002444100024420 R11: 0002440100024380 R12: 0000000000000400
[597693.769747] R13: ffff9239b6a36af0 R14: 0000000000000001 R15: 0000000000000004
[597693.770289] FS: 0000000000000000(0000) GS:ffff923e3fdc0000(0000) knlGS:0000000000000000
[597693.770792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[597693.771354] CR2: 000000c420202000 CR3: 00000005c9ae2000 CR4: 00000000001607e0
[597693.771967] Call Trace:
[597693.772540] [<ffffffff82a302a4>] __mem_cgroup_free+0x234/0x250
[597693.773135] [<ffffffff82a302d5>] free_work+0x15/0x20
[597693.773725] [<ffffffff828b9d4f>] process_one_work+0x17f/0x440
[597693.774322] [<ffffffff828bade6>] worker_thread+0x126/0x3c0
[597693.774925] [<ffffffff828bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[597693.775535] [<ffffffff828c1c31>] kthread+0xd1/0xe0
[597693.776136] [<ffffffff828c1b60>] ? insert_kthread_work+0x40/0x40
[597693.776742] [<ffffffff82f74c37>] ret_from_fork_nospec_begin+0x21/0x21
[597693.777339] [<ffffffff828c1b60>] ? insert_kthread_work+0x40/0x40
[597693.777922] Code: d0 a2 83 e8 12 36 3f 00 89 de 49 89 c5 4c 89 e7 e8 05 fd ff ff 4c 89 ee 48 c7 c7 b8 d0 a2 83 e8 86 32 3f 00 5b 41 5c 41 5d 5d c3 <0f> 0b 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41
57
[597693.779716] RIP [<ffffffff82b76ee1>] ida_simple_remove+0x41/0x50
[597693.780297] RSP <ffff923ad25a3da0>

# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

# uname -r
3.10.0-957.1.3.el7.x86_64

The servers are mainly running docker and kubernetes components.
Tagsbug, crash, kernel
abrt_hash
URL

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2018-12-19 16:30 dennisxrow New Issue
2018-12-19 16:30 dennisxrow Tag Attached: bug
2018-12-19 16:30 dennisxrow Tag Attached: crash
2018-12-19 16:30 dennisxrow Tag Attached: kernel