2017-07-25 14:43 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0012577CentOS-7kernelpublic2017-02-16 09:53
Reporterdandim 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusnewResolutionopen 
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0012577: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
DescriptionSystem freezed after this kernel error. I could not connect to Console or via SSH to VM. I had to reboot VM.
Additional InformationNMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [check_nrpe:1081]
  Code: c1 07 89 c2 c1 ea 10 66 39 c2 75 01 c3 55 83 e2 fe 0f b7 f2 48 89 e5 b8 00 80 00 00 eb 0d 66 0f 1f 44 00 00 f3 90 83 e8 01 74 0a <0f> b7 0f 66 39 ca 75 f1 5d c3 0f 1f 80 00 0
   [<ffffffff816967c9>] system_call_fastpath+0x16/0x1b
   [<ffffffff8155615e>] SyS_connect+0xe/0x10
   [<ffffffff8121ccd7>] ? __fd_install+0x47/0x60
   [<ffffffff815521c0>] ? sock_alloc_file+0xa0/0x140
   [<ffffffff81555357>] SYSC_connect+0xe7/0x120
   [<ffffffff815eba18>] inet_stream_connect+0x38/0x50
   [<ffffffff811de456>] ? kmem_cache_alloc_trace+0x1d6/0x200
   [<ffffffff8166b202>] ? netlbl_domhsh_search+0x12/0xb0
   [<ffffffff815eb765>] __inet_stream_connect+0xb5/0x330
   [<ffffffff815d4890>] tcp_v4_connect+0x2b0/0x4e0
   [<ffffffff815bae9b>] inet_hash_connect+0x3b/0x40
   [<ffffffff815b9ff0>] ? inet_unhash+0xb0/0xb0
   [<ffffffff815b9e40>] ? inet_ehashfn+0xc0/0xc0
   [<ffffffff815bac46>] __inet_hash_connect+0x1b6/0x3d0
  Call Trace:
   ffffffff815b9e40 ffffffff815b9ff0 ffffffff81aa2340 ffffffff0000b119
   ffff88007a73cd80 ffffffff81a9cd80 ffffffff0000b119 ffffc9000035b4f0
   ffff8800065abd70 ffffffff815bac46 03fffffffe06a736 7b3b049ed01771e1
  Stack:
  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  CR2: 00002ad4842708b0 CR3: 00000000780b6000 CR4: 00000000000407f0
  CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  FS: 00002ad483714a80(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
  R13: ffff8800065abc78 R14: ffff8800065abcd0 R15: ffffffff8140cc32
  R10: ffff88007a73cd80 R11: 0000000057acec73 R12: 0000000000000080
  RBP: ffff8800065abcd0 R08: ffffffff81a9cd80 R09: ffffc9000035b4f0
  RDX: 0000000000007c2a RSI: 0000000000007c2a RDI: ffffc9000035b4f0
  RAX: 0000000000004919 RBX: 000000000000b118 RCX: 0000000000009180
  RSP: 0018:ffff8800065abcd0 EFLAGS: 00000202
  RIP: 0010:[<ffffffff8168db37>] [<ffffffff8168db37>] _raw_spin_lock+0x37/0x50
  task: ffff88007c3a5e20 ti: ffff8800065a8000 task.ti: ffff8800065a8000
  Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
  CPU: 0 PID: 1081 Comm: check_nrpe Not tainted 3.10.0-514.2.2.el7.x86_64 #1
  Modules linked in: iptable_nat vmw_vsock_vmci_transport nf_nat_ipv4 nf_nat vsock ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter
  NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [check_nrpe:1081]
TagsNo tags attached.
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0028585

dandim (reporter)

After update to kernel version 3.10.0-514.6.1.el7 problem repeated! I had to boot to kernel version 3.10.0-327.36.3.el7. With this version everything looks good. I looks like that problem is in kernel/VMXNET3 NIC or VMWARE ESXi 5.5.

Log with kernel 3.10.0-514.6.1.el7 looks like:

úno 16 01:31:23 QUANTI-LB01 kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 67s! [nginx:9204]
úno 16 01:31:19 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:31:14 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:31:09 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:31:04 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:59 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:54 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:49 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:44 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:39 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:34 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:29 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:24 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:19 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:14 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:09 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:30:04 QUANTI-LB01 kernel: vmxnet3 0000:0b:00.0 ens192: tx hang
úno 16 01:29:59 QUANTI-LB01 kernel: Code: 00 02 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 01 c3 55 83 e2 fe 0f b7 f2 48 89 e5 b8 00 80 00 00 eb 0d 66 0f 1f 44 00 00 f3 90 <83> e8 01 74 0a 0f b7 0f 66 39 ca 75 f1 5d c3 66 66 66 90 66 66
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff816968c9>] system_call_fastpath+0x16/0x1b
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff8155614e>] SyS_connect+0xe/0x10
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff81555347>] SYSC_connect+0xe7/0x120
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815ebb08>] inet_stream_connect+0x38/0x50
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff810b12cc>] ? add_wait_queue+0x3c/0x50
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815eb855>] __inet_stream_connect+0xb5/0x330
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815d4980>] tcp_v4_connect+0x2b0/0x4e0
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815baf6b>] inet_hash_connect+0x3b/0x40
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815ba0c0>] ? inet_unhash+0xb0/0xb0
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815b9f10>] ? inet_ehashfn+0xc0/0xc0
úno 16 01:29:59 QUANTI-LB01 kernel: [<ffffffff815bad16>] __inet_hash_connect+0x1b6/0x3d0
úno 16 01:29:59 QUANTI-LB01 kernel: Call Trace:
úno 16 01:29:59 QUANTI-LB01 kernel: ffffffff815b9f10 ffffffff815ba0c0 ffffffff81aa2340 ffff88000000b119
úno 16 01:29:59 QUANTI-LB01 kernel: ffff88007bb1b640 ffffffff81a9cd80 1d15b4190000b119 ffffc9000035b4f0
úno 16 01:29:59 QUANTI-LB01 kernel: ffff88007c3e7d70 ffffffff815bad16 03fffffffe06a736 0d9cce57c331b0d0
úno 16 01:29:59 QUANTI-LB01 kernel: Stack:
úno 16 01:29:59 QUANTI-LB01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
úno 16 01:29:59 QUANTI-LB01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
úno 16 01:29:59 QUANTI-LB01 kernel: CR2: 00007f2ba2821000 CR3: 000000007a7c4000 CR4: 00000000000407e0
úno 16 01:29:59 QUANTI-LB01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
úno 16 01:29:59 QUANTI-LB01 kernel: FS: 00007f1088175740(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
úno 16 01:29:59 QUANTI-LB01 kernel: R13: 0000000000000000 R14: ffff88007c3e7d24 R15: ffff88007f808400
úno 16 01:29:59 QUANTI-LB01 kernel: R10: ffff88007bb1b640 R11: 000000006ec01afb R12: ffff88007c3e7c80
úno 16 01:29:59 QUANTI-LB01 kernel: RBP: ffff88007c3e7cd0 R08: ffffffff81a9cd80 R09: ffffc9000035b4f0
úno 16 01:29:59 QUANTI-LB01 kernel: RDX: 0000000000007c1a RSI: 0000000000007c1a RDI: ffffc9000035b4f0
úno 16 01:29:59 QUANTI-LB01 kernel: RAX: 000000000000436b RBX: 00000011d1339f11 RCX: 0000000000004180
úno 16 01:29:59 QUANTI-LB01 kernel: RSP: 0018:ffff88007c3e7cd0 EFLAGS: 00000206
úno 16 01:29:59 QUANTI-LB01 kernel: RIP: 0010:[<ffffffff8168dc12>] [<ffffffff8168dc12>] _raw_spin_lock+0x32/0x50
úno 16 01:29:59 QUANTI-LB01 kernel: task: ffff88007b82af10 ti: ffff88007c3e4000 task.ti: ffff88007c3e4000
úno 16 01:29:59 QUANTI-LB01 kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
úno 16 01:29:59 QUANTI-LB01 kernel: CPU: 1 PID: 9204 Comm: nginx Tainted: G W L ------------ 3.10.0-514.6.1.el7.x86_64 #1
úno 16 01:29:59 QUANTI-LB01 kernel: Modules linked in: iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp cor
+Notes

-Issue History
Date Modified Username Field Change
2017-01-04 09:26 dandim New Issue
2017-02-16 09:53 dandim Note Added: 0028585
+Issue History