0003213: Panic and ksoftird with high load

CentOS Bug Tracker

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0003213

CentOS-5

kernel

public

2008-10-27 03:13

2008-11-11 14:46

Reporter

gondim

Priority

normal

Severity

crash

Reproducibility

random

Status

resolved

Resolution

fixed

Platform

OS Version

Product Version

5.2

Target Version

Fixed in Version

5.2

Summary

0003213: Panic and ksoftird with high load

Description

When I upgraded the kernel to 2.6.18-92.1.13.el5 problems began to appear and some kernel panic. In version 2.6.18-92.1.10.el5 these problems do not exist.
The problem begins with the ksoftirqd using much CPU and appears in the logs this:

Oct 26 02:01:56 zeus kernel: BUG: soft lockup - CPU#1 stuck for 10s! [iptables:30452]
Oct 26 02:01:56 zeus kernel:
Oct 26 02:01:56 zeus kernel: Pid: 30452, comm: iptables
Oct 26 02:01:56 zeus kernel: EIP: 0060:[<f8c53500>] CPU: 1
Oct 26 02:01:56 zeus kernel: EIP is at ipt_do_table+0xad/0x2d9 [ip_tables]
Oct 26 02:01:56 zeus kernel: EFLAGS: 00200246 Tainted: G (2.6.18-92.1.13.el5PAE 0000001)
Oct 26 02:01:56 zeus kernel: EAX: 00000098 EBX: e15dd820 ECX: 00000000 EDX: ec322001
Oct 26 02:01:56 zeus kernel: ESI: f9231bb0 EDI: f8c5c86c EBP: c072bedc DS: 007b ES: 007b
Oct 26 02:01:56 zeus kernel: CR0: 8005003b CR2: 09d01000 CR3: 376dfd60 CR4: 000006f0
Oct 26 02:01:56 zeus kernel: [<f8c5c017>] ipt_route_hook+0x17/0x1c [iptable_mangle]
Oct 26 02:01:56 zeus kernel: [<c05c35a4>] nf_iterate+0x30/0x61
Oct 26 02:01:56 zeus kernel: [<c05cc24b>] ip_finish_output+0x0/0x1db
Oct 26 02:01:56 zeus kernel: [<c05c36ca>] nf_hook_slow+0x3a/0x90
Oct 26 02:01:56 zeus kernel: [<c05cc24b>] ip_finish_output+0x0/0x1db
Oct 26 02:01:56 zeus kernel: [<c05cd530>] ip_output+0x82/0x266
Oct 26 02:01:56 zeus kernel: [<c05cc24b>] ip_finish_output+0x0/0x1db
Oct 26 02:01:56 zeus kernel: [<c05c9d13>] ip_forward+0x1d9/0x22e
Oct 26 02:01:56 zeus kernel: [<c05c8b58>] ip_rcv+0x475/0x4af
Oct 26 02:01:56 zeus kernel: [<c05ae3a2>] netif_receive_skb+0x2b1/0x325
Oct 26 02:01:56 zeus kernel: [<f8a0d908>] e1000_receive_skb+0x103/0x120 [e1000e]
Oct 26 02:01:56 zeus kernel: [<f8a105c7>] e1000_clean_rx_irq+0x21d/0x2bc [e1000e]
Oct 26 02:01:56 zeus kernel: [<f8a0ed6a>] e1000_clean+0x63/0x215 [e1000e]
Oct 26 02:01:56 zeus kernel: [<c05afd42>] net_rx_action+0x92/0x175
Oct 26 02:01:56 zeus kernel: [<c042abd6>] __do_softirq+0x5a/0xbb
Oct 26 02:01:56 zeus kernel: [<c0407451>] do_softirq+0x52/0x9d
Oct 26 02:01:56 zeus kernel: [<c04059bf>] apic_timer_interrupt+0x1f/0x24
Oct 26 02:01:56 zeus kernel: [<f8c52160>] get_counters+0x91/0xb5 [ip_tables]
Oct 26 02:01:56 zeus kernel: [<f8c52f0c>] do_ipt_set_ctl+0x254/0x600 [ip_tables]
Oct 26 02:01:56 zeus kernel: [<c0417a86>] do_flush_tlb_all+0x0/0x5a
Oct 26 02:01:56 zeus kernel: [<c0417a86>] do_flush_tlb_all+0x0/0x5a
Oct 26 02:01:56 zeus kernel: [<c05c3f9c>] nf_sockopt+0xac/0xda
Oct 26 02:01:56 zeus kernel: [<c05c3fee>] nf_setsockopt+0x11/0x16
Oct 26 02:01:56 zeus kernel: [<c05ce74a>] ip_setsockopt+0x9f1/0xa10
Oct 26 02:01:56 zeus kernel: [<c04059bf>] apic_timer_interrupt+0x1f/0x24
Oct 26 02:01:56 zeus kernel: [<c045797b>] get_page_from_freelist+0x2ba/0x333
Oct 26 02:01:56 zeus kernel: [<c0457a4b>] __alloc_pages+0x57/0x297
Oct 26 02:01:56 zeus kernel: [<c0464c19>] anon_vma_prepare+0x11/0xa5
Oct 26 02:01:56 zeus kernel: [<c0460044>] __handle_mm_fault+0xb0a/0xb62
Oct 26 02:01:56 zeus kernel: [<c04e4f8f>] vsnprintf+0x41f/0x45d
Oct 26 02:01:56 zeus kernel: [<c0461902>] vma_adjust+0x19d/0x37b
Oct 26 02:01:56 zeus kernel: [<c05a70d2>] sock_common_setsockopt+0x12/0x16
Oct 26 02:01:56 zeus kernel: [<c05a585c>] sys_setsockopt+0x76/0x95
Oct 26 02:01:56 zeus kernel: [<c05a6a7a>] sys_socketcall+0x15c/0x19e
Oct 26 02:01:56 zeus kernel: [<c0404eff>] syscall_call+0x7/0xb
Oct 26 02:01:56 zeus kernel: =======================

Additional Information

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
stepping : 11
cpu MHz : 1596.000
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
bogomips : 4799.52

total used free shared buffers cached
Mem: 4073936 1470780 2603156 0 175620 1093568
-/+ buffers/cache: 201592 3872344
Swap: 2031608 0 2031608

Notes
(0008193) gondim (reporter) 2008-10-27 11:28	We are Internet Service Provider with more than 2,000 customers. (root@zeus)[~]# iptables -L -n -t filter\|wc -l 2448 (root@zeus)[~]# iptables -L -n -t nat\|wc -l 2315 (root@zeus)[~]# iptables -L -n -t mangle\|wc -l 4546

(0008262) gondim (reporter) 2008-11-11 14:10	Hi all, This problem was resolved in version: 2.6.18-92.1.17.el5. :)

(0008263) gondim (reporter) 2008-11-11 14:11	Hi all, This problem was resolved in version: 2.6.18-92.1.17.el5. :)

(0008265) range (administrator) 2008-11-11 14:46	Has been fixed in 2.6.18-92.1.17

Issue History
Date Modified	Username	Field	Change
2008-10-27 03:13	gondim	New Issue
2008-10-27 11:28	gondim	Note Added: 0008193
2008-11-11 14:10	gondim	Note Added: 0008262
2008-11-11 14:11	gondim	Note Added: 0008263
2008-11-11 14:11	gondim	Status	new => feedback
2008-11-11 14:46	range	Status	feedback => resolved
2008-11-11 14:46	range	Fixed in Version	=> 5.2
2008-11-11 14:46	range	Resolution	open => fixed
2008-11-11 14:46	range	Note Added: 0008265

Relationships