View Issue Details

IDProjectCategoryView StatusLast Update
0016879CentOS-7crashpublic2019-12-31 13:11
Reporterttong0202 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0016879: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s!
DescriptionDec 27 04:43:55 localhost kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [migration/1:14]
Dec 27 04:43:55 localhost kernel: Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache xt_statistic veth ipt_REJECT nf_reject_ipv4 xt_nat xt_recent ip_vs xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter overlay(T) iptable_filter ip_tables vxlan ip6_udp_tunnel udp_tunnel ip_set nfnetlink bridge stp llc vfat fat skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm dell_smbios iTCO_wdt dell_wmi_descriptor irqbypass dcdbas crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif sg pcspkr joydev i2c_i801 mei_me mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler wmi
Dec 27 04:43:55 localhost kernel: acpi_pad acpi_power_meter xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci tg3 libahci libata megaraid_sas drm_panel_orientation_quirks ptp pps_core nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
Dec 27 04:43:55 localhost kernel: CPU: 1 PID: 14 Comm: migration/1 Kdump: loaded Tainted: G ------------ T 3.10.0-957.el7.x86_64 #1
Dec 27 04:43:55 localhost kernel: Hardware name: Dell Inc. PowerEdge R540/0TKD84, BIOS 2.4.8 11/27/2019
Dec 27 04:43:55 localhost kernel: task: ffff8e84b4191040 ti: ffff8e84b419c000 task.ti: ffff8e84b419c000
Dec 27 04:43:55 localhost kernel: RIP: 0010:[<ffffffffbb12fd8e>] [<ffffffffbb12fd8e>] multi_cpu_stop+0x8e/0x110
Dec 27 04:43:55 localhost kernel: RSP: 0000:ffff8e84b419fd98 EFLAGS: 00000246
Dec 27 04:43:55 localhost kernel: RAX: 0000000000000001 RBX: ffff8e84b2dbfc00 RCX: dead000000000200
Dec 27 04:43:55 localhost kernel: RDX: ffff8e8c1be15ff0 RSI: 0000000000000282 RDI: ffff8e8384bf7ab0
Dec 27 04:43:55 localhost kernel: RBP: ffff8e84b419fdc0 R08: ffff8e8384bf7a50 R09: 0000000000000001
Dec 27 04:43:55 localhost kernel: R10: 000000000000b9ed R11: 0000000000000001 R12: 0000000000000000
Dec 27 04:43:55 localhost kernel: R13: 000000000001ab80 R14: ffff8e84b2d8f000 R15: 0000000200000001
Dec 27 04:43:55 localhost kernel: FS: 0000000000000000(0000) GS:ffff8e8c1be00000(0000) knlGS:0000000000000000
Dec 27 04:43:55 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 27 04:43:55 localhost kernel: CR2: 00007fb1fcff8df0 CR3: 0000000abd410000 CR4: 00000000007607e0
Dec 27 04:43:55 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 27 04:43:55 localhost kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 27 04:43:55 localhost kernel: PKRU: 00000000
Dec 27 04:43:55 localhost kernel: Call Trace:
Dec 27 04:43:55 localhost kernel: [<ffffffffbb12fd00>] ? cpu_stop_should_run+0x50/0x50
Dec 27 04:43:55 localhost kernel: [<ffffffffbb130009>] cpu_stopper_thread+0x99/0x150
Dec 27 04:43:55 localhost kernel: [<ffffffffbb76770f>] ? __schedule+0x3ff/0x890
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0ca5c4>] smpboot_thread_fn+0x144/0x1a0
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0ca480>] ? lg_double_unlock+0x40/0x40
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0c1c31>] kthread+0xd1/0xe0
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0c1b60>] ? insert_kthread_work+0x40/0x40
Dec 27 04:43:55 localhost kernel: [<ffffffffbb774c1d>] ret_from_fork_nospec_begin+0x7/0x21
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0c1b60>] ? insert_kthread_work+0x40/0x40
Dec 27 04:43:55 localhost kernel: Code: 41 ff 4c 24 24 0f 94 c0 84 c0 0f 1f 00 74 17 41 8b 44 24 20 41 8b 54 24 10 83 c0 01 41 89 54 24 24 41 89 44 24 20 83 fb 04 74 0a <89> d8 eb b6 66 0f 1f 44 00 00 4c 89 ef 57 9d 0f 1f 44 00 00 5b
Dec 27 04:43:55 localhost kernel: NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [runc:[1:CHILD]:37047]
Dec 27 04:43:55 localhost kernel: Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache xt_statistic veth ipt_REJECT nf_reject_ipv4 xt_nat xt_recent ip_vs xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter overlay(T) iptable_filter ip_tables vxlan ip6_udp_tunnel udp_tunnel ip_set nfnetlink bridge stp llc vfat fat skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm dell_smbios iTCO_wdt iTCO_vendor_support dell_wmi_descriptor irqbypass dcdbas crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif sg pcspkr joydev i2c_i801 mei_me mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler wmi
Dec 27 04:43:55 localhost kernel: acpi_pad acpi_power_meter xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci tg3 libahci libata megaraid_sas drm_panel_orientation_quirks ptp pps_core nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
Dec 27 04:43:55 localhost kernel: CPU: 26 PID: 37047 Comm: runc:[1:CHILD] Kdump: loaded Tainted: G L ------------ T 3.10.0-957.el7.x86_64 #1
Dec 27 04:43:55 localhost kernel: Hardware name: Dell Inc. PowerEdge R540/0TKD84, BIOS 2.4.8 11/27/2019
Dec 27 04:43:55 localhost kernel: task: ffff8e7bc9f130c0 ti: ffff8e7bc9598000 task.ti: ffff8e7bc9598000
Dec 27 04:43:55 localhost kernel: RIP: 0010:[<ffffffffbb386959>] [<ffffffffbb386959>] __write_lock_failed+0x9/0x20
Dec 27 04:43:55 localhost kernel: RSP: 0018:ffff8e7bc959bcd8 EFLAGS: 00000297
Dec 27 04:43:55 localhost kernel: RAX: ffff8e7bc959bfd8 RBX: ffff8e7bc9a5c400 RCX: 0000000000000000
Dec 27 04:43:55 localhost kernel: RDX: ffff8e7bc959bd5c RSI: 0000000000000001 RDI: ffff8e8382214e3c
Dec 27 04:43:55 localhost kernel: RBP: ffff8e7bc959bcd8 R08: 000000000001f140 R09: ffffffffbb6ec6f2
Dec 27 04:43:55 localhost kernel: R10: ffff8e841cf5f140 R11: fffffa4a2ffa9600 R12: 0000000000000006
Dec 27 04:43:55 localhost kernel: R13: ffff8e7bc9a5c4f0 R14: ffff8e84202eae00 R15: ffff8e841eda9be8
Dec 27 04:43:55 localhost kernel: FS: 0000000001231880(0063) GS:ffff8e841cf40000(0000) knlGS:0000000000000000
Dec 27 04:43:55 localhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 27 04:43:55 localhost kernel: CR2: 00007f8a2da16248 CR3: 0000000009fce000 CR4: 00000000007607e0
Dec 27 04:43:55 localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 27 04:43:55 localhost kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 27 04:43:55 localhost kernel: PKRU: 55555554
Dec 27 04:43:55 localhost kernel: Call Trace:
Dec 27 04:43:55 localhost kernel: [<ffffffffbb76a27a>] _raw_write_lock_bh+0x2a/0x30
Dec 27 04:43:55 localhost kernel: [<ffffffffbb6e039a>] xfrm_policy_flush+0x3a/0x2f0
Dec 27 04:43:55 localhost kernel: [<ffffffffbb6e06a8>] xfrm_policy_fini+0x58/0x180
Dec 27 04:43:55 localhost kernel: [<ffffffffbb6e0a70>] xfrm_net_init+0x240/0x370
Dec 27 04:43:55 localhost kernel: [<ffffffffbb62dd94>] ops_init+0x44/0x150
Dec 27 04:43:55 localhost kernel: [<ffffffffbb62df43>] setup_net+0xa3/0x160
Dec 27 04:43:55 localhost kernel: [<ffffffffbb62e6e5>] copy_net_ns+0xb5/0x180
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0c6e39>] create_new_namespaces+0xf9/0x180
Dec 27 04:43:55 localhost kernel: [<ffffffffbb0c707a>] unshare_nsproxy_namespaces+0x5a/0xc0
Dec 27 04:43:55 localhost kernel: [<ffffffffbb096b83>] SyS_unshare+0x173/0x2e0
Dec 27 04:43:55 localhost kernel: [<ffffffffbb774ddb>] system_call_fastpath+0x22/0x27
Dec 27 04:43:55 localhost kernel: Code: 00 00 0f 1f 44 00 00 41 ff e7 e8 07 00 00 00 f3 90 0f ae e8 eb f9 4c 89 3c 24 c3 90 90 90 90 90 90 90 55 48 89 e5 f0 ff 07 f3 90 <83> 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 90 66 2e 0f 1f 84 00 00 00
Dec 27 04:43:56 localhost etcd: the clock difference against peer 1d6a226c8fbc6731 is too high [1.859563421s > 1s] (prober "ROUND_TRIPPER_RAFT_MESSAGE")
Dec 27 04:43:56 localhost etcd: the clock difference against peer 1d6a226c8fbc6731 is too high [1.861943733s > 1s] (prober "ROUND_TRIPPER_SNAPSHOT")
Dec 27 04:44:06 localhost kube-proxy: E1227 04:44:06.645819 40792 proxier.go:964] Failed to delete kube-system/kube-dns-service:dns endpoint connections, error: error deleting conntrack entries for UDP peer {10.254.0.2, 172.30.92.26}, error: error looking for path of conntrack: exec: "conntrack": executable file not found in $PATH
Tagssoft lockup
abrt_hash
URL

Activities

TrevorH

TrevorH

2019-12-31 13:11

manager   ~0035924

7.6 is unsupported. Run `yum update` to get to 7.7 plus hte latest fixes, reboot and see if the problem happens again.

Issue History

Date Modified Username Field Change
2019-12-31 10:41 ttong0202 New Issue
2019-12-31 10:41 ttong0202 Tag Attached: soft lockup
2019-12-31 13:11 TrevorH Note Added: 0035924