View Issue Details

IDProjectCategoryView StatusLast Update
0005813CentOS-6kernelpublic2014-06-02 14:14
Reporterjiandal 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Product Version6.2 
Target VersionFixed in Version6.4 
Summary0005813: Higher than normal ksoftirqd CPU usage
DescriptionI have a Dual CPU Xeon L5420 server
8GB RAM
SSD
HDD

Any kernel version >7.1 is producing higher than normal ksoftirqd CPU usage, up to ~30% at times.


top output:


 8103 mysql 20 0 5330m 3.1g 6884 S 94.7 40.5 1430:08 mysqld
    9 root 20 0 0 0 0 S 23.9 0.0 495:49.30 ksoftirqd/1
   25 root 20 0 0 0 0 S 13.9 0.0 237:24.25 ksoftirqd/5
25926 nginx 20 0 386m 58m 38m S 4.3 0.7 1:38.91 php-fpm
26677 nginx 20 0 385m 57m 38m S 3.3 0.7 1:34.78 php-fpm
16609 nginx 20 0 394m 63m 44m S 2.7 0.8 2:35.59 php-fpm
25019 nginx 20 0 396m 63m 41m S 2.7 0.8 1:48.12 php-fpm
    4 root 20 0 0 0 0 S 1.0 0.0 704:40.21 ksoftirqd/0
   33 root 20 0 0 0 0 S 1.0 0.0 125:50.17 ksoftirqd/7
 7670 varnish 20 0 1556m 316m 268m S 1.0 4.0 14:05.77 varnishd
16713 nginx 20 0 383m 58m 40m S 1.0 0.7 2:36.83 php-fpm
   29 root 20 0 0 0 0 S 0.7 0.0 50:45.33 ksoftirqd/6
 6682 nginx 10 -10 36868 7196 800 S 0.7 0.1 13:13.75 nginx

# uname -r
2.6.32-220.23.1.el6.x86_64

# lspci
00:00.0 Host bridge: Intel Corporation 5100 Chipset Memory Controller Hub (rev 90)
00:02.0 PCI bridge: Intel Corporation 5100 Chipset PCI Express x8 Port 2-3 (rev 90)
00:04.0 PCI bridge: Intel Corporation 5100 Chipset PCI Express x16 Port 4-7 (rev 90)
00:08.0 System peripheral: Intel Corporation 5100 Chipset DMA Engine (rev 90)
00:10.0 Host bridge: Intel Corporation 5100 Chipset FSB Registers (rev 90)
00:10.1 Host bridge: Intel Corporation 5100 Chipset FSB Registers (rev 90)
00:10.2 Host bridge: Intel Corporation 5100 Chipset FSB Registers (rev 90)
00:11.0 Host bridge: Intel Corporation 5100 Chipset Reserved Registers (rev 90)
00:13.0 Host bridge: Intel Corporation 5100 Chipset Reserved Registers (rev 90)
00:15.0 Host bridge: Intel Corporation 5100 Chipset DDR Channel 0 Registers (rev 90)
00:16.0 Host bridge: Intel Corporation 5100 Chipset DDR Channel 1 Registers (rev 90)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 (rev 02)
00:1c.5 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 6 (rev 02)
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
04:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
06:01.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics Innovation) Z7/Z9 (XG20 core)
Steps To ReproduceAny kernel > 7.1
Additional InformationPlease let me know any other things you require. I have reverted back to 7.1 previously and ksoftirqd CPU is lower, ~0.5%.

Thank you.
TagsNo tags attached.

Activities

bmalynovytch

bmalynovytch

2012-07-10 14:24

reporter   ~0015390

Hi,

Could you provide a "vmstat 1 30" output during load please ?

Regards,

Benjamin
jiandal

jiandal

2012-07-10 14:28

reporter   ~0015391

# vmstat 1 30
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 4 0 46328 2192996 247292 1516740 4 5 147 191 7 12 33 1 65 1 0
 5 0 46328 2190012 247292 1516740 0 0 0 4 5361 964 60 0 40 0 0
 5 0 46328 2189512 247292 1516756 0 0 0 8 6128 1766 63 1 36 0 0
 3 0 46328 2188232 247364 1516672 0 0 0 576 5833 2188 57 1 37 5 0
 2 0 46328 2188140 247364 1516740 0 0 0 8 4186 1820 35 0 65 0 0
 1 0 46328 2191108 247364 1516744 0 0 0 4 2786 1884 18 1 81 0 0
 2 0 46328 2190868 247364 1516744 0 0 0 556 3046 1411 26 0 74 0 0
 2 0 46328 2190372 247364 1516744 0 0 0 0 2883 954 25 0 74 0 0
 1 0 46328 2189256 247364 1516744 0 0 0 4032 2414 1426 16 0 83 1 0
 3 0 46328 2189132 247372 1516740 0 0 0 44 2944 1196 24 0 76 0 0
 3 0 46328 2185040 247372 1516744 0 0 0 52 4536 2437 41 1 59 0 0
 2 0 46328 2203404 247372 1516744 0 0 0 20 4128 2470 32 1 66 0 0
 1 0 46328 2197516 247372 1516744 0 0 0 16 2912 2133 20 1 79 0 0
 2 0 46328 2197864 247372 1516744 0 0 0 40 2554 1987 18 0 82 0 0
 2 0 46328 2199492 247372 1516744 0 0 0 40 3007 2645 18 1 81 0 0
 2 0 46328 2202352 247372 1516744 0 0 0 28 4332 2740 31 1 68 0 0
 2 0 46328 2204056 247372 1516744 0 0 0 136 3651 2301 26 0 72 2 0
 3 0 46328 2193508 247376 1516740 0 0 0 10720 5888 3229 49 1 49 1 0
 5 0 46328 2191764 247376 1516744 0 0 0 4 5544 2616 48 0 52 0 0
 3 0 46328 2190276 247376 1516748 0 0 0 8 5249 2124 49 0 51 0 0
 2 0 46328 2191408 247380 1516744 0 0 0 36 3498 2138 26 1 73 0 0
 3 0 46328 2193276 247380 1516748 0 0 0 8 2594 1676 19 0 80 0 0
 2 0 46328 2189676 247380 1516748 0 0 0 28 3650 2820 26 1 74 0 0
 3 0 46328 2190296 247380 1516748 0 0 0 16 3770 1863 31 1 69 0 0
 6 0 46328 2191296 247380 1516748 0 0 0 8 5513 2019 49 1 50 0 0
 6 0 46328 2186336 247380 1516748 0 0 0 24 6892 1763 73 1 26 0 0
 7 0 46328 2186892 247384 1516748 0 0 0 20 7409 2300 73 0 27 0 0
11 0 46328 2182172 247388 1516744 0 0 0 8280 9041 2462 94 1 5 0 0
10 0 46328 2183884 247388 1516748 0 0 0 4 7864 1907 88 0 12 0 0
10 0 46328 2179040 247388 1516752 0 0 0 68 8247 2385 89 0 10 0 0


Thank you
bmalynovytch

bmalynovytch

2012-07-10 15:07

reporter   ~0015393

I thought you had a bottleneck that we could see with vmstat, but I can't see any there.

Is your server under heavy network load ? (large amount of small packets for exemple ? could be tons of empty UDP packets)

If not any of what I mentioned, I won't be very helpful, sorry :(

Regards,

Benjamin
jiandal

jiandal

2012-07-10 15:12

reporter   ~0015394

I don't think it's related to network, so could be a kernel bug.

# netstat -an |grep :80 |wc -l
1168
# netstat -an | grep 80 | grep ESTA | wc
    140 840 12460

Thanks for helping anyway!
jiandal

jiandal

2012-07-12 17:49

reporter   ~0015417

Possibly related:
https://bugzilla.redhat.com/show_bug.cgi?id=836964 I have yet to try the nohz=off fix
smartgig

smartgig

2012-08-29 13:31

reporter   ~0015717

Please fix this bug.
This bug is killing my dedicated servers ...
bmalynovytch

bmalynovytch

2012-08-29 14:28

reporter   ~0015718

Did you try the "nohz=off" trick ?
It fixed the problem with some of my VMs.
smartgig

smartgig

2012-08-29 17:19

reporter   ~0015721

with nohz=off, the number of threads increases from 265 to 380 .

This bug is reported on 2.6.32 kernels. I think centos always has had stable releases, this is very annoying me that I've migrated two centos 5.x servers and 1 debian 6.x server to centos 6.3 and all of them have this critical problem.
This bug increases IRQ load drastically.

Please fix it asap.
jiandal

jiandal

2012-08-29 17:23

reporter   ~0015722

This seems like a upstream issue.

Sadly I am unable to add any possible solutions to this - we didn't end up using the nohz=off setting since our servers were particularly under load. We're now migrated onto newer E3 servers.
Nerigal

Nerigal

2012-09-12 20:44

reporter   ~0015767

Could you provide a "cat /proc/interrupts"

thanks
smartgig

smartgig

2012-09-13 10:24

reporter   ~0015769

Here you're mine result :

root@sv: ~# cat /proc/interrupts


           CPU0 CPU1 CPU2 CPU3
  0: 3198 0 3 161 IO-APIC-edge timer
  1: 1 1 0 0 IO-APIC-edge i8042
  3: 0 1 1 0 IO-APIC-edge
  4: 1 0 1 0 IO-APIC-edge
  7: 0 0 0 0 IO-APIC-edge parport0
  8: 0 0 0 1 IO-APIC-edge rtc0
  9: 0 0 0 0 IO-APIC-fasteoi acpi
 16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
 17: 2838 2831 25519161 11478634 IO-APIC-fasteoi uhci_hcd:usb4, ata_piix
 18: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8, ata_piix
 22: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
 23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
 27: 56 352 57 199996941 PCI-MSI-edge eth0
 28: 4 392790 92526 15 PCI-MSI-edge eth1
NMI: 243422 380412 284569 363464 Non-maskable interrupts
LOC: 258106853 366869962 326776605 394366789 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 243422 380412 284569 363464 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 30592641 19589376 24714908 16575473 Rescheduling interrupts
CAL: 10812984 37461217 231292 49643745 Function call interrupts
TLB: 4360277 5380098 4315899 5281866 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 3236 3236 3236 3236 Machine check polls
ERR: 0
MIS: 0
Nerigal

Nerigal

2012-09-13 13:45

reporter   ~0015771

yeah "same" result as me. Take a look at the line

LOC: 258106853 366869962 326776605 394366789 Local timer interrupts

for me this value never been so high before Centos 6.x
i dont have the answer how to solve it because this is very low level coding but
reading this could give you hint where to search deeper

http://stackoverflow.com/questions/10567214/what-are-linux-local-timer-interrupts
sigtrap

sigtrap

2012-09-28 17:24

reporter   ~0015857

Are there any updates on a possible resolution to this issue? I'm having the same issue, but on a 1 core VM after upgrading from CentOS 5.8 to CentOS 6.2; as of right now, 3 separate systems are experiencing this issue.

% top
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    4 root 20 0 0 0 0 R 44.7 0.0 108:10.03 ksoftirqd/0

% vmstat 2
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 3 0 72 440060 111340 834172 0 0 4 24 79 63 0 3 95 0 2
 4 0 72 440060 111340 834192 0 0 0 2 667 131 0 50 0 0 50
 3 0 72 440060 111344 834188 0 0 0 48 686 157 0 47 10 0 43
11 0 72 440060 111344 834192 0 0 0 0 734 146 0 51 0 0 49
 3 0 72 440084 111344 834192 0 0 0 0 837 208 1 50 0 0 49
 5 0 72 440084 111352 834192 0 0 0 18 738 134 0 50 0 0 50
 7 0 72 440084 111352 834192 0 0 0 0 736 152 0 50 0 0 49
 4 0 72 440084 111352 834192 0 0 0 0 677 131 0 51 0 0 49

% cat /proc/interrupts
           CPU0
  0: 11141844 xen-percpu-virq timer0
  1: 0 xen-percpu-ipi resched0
  2: 0 xen-percpu-ipi callfunc0
  3: 0 xen-percpu-virq debug0
  4: 0 xen-percpu-ipi callfuncsingle0
  5: 788 xen-dyn-event xenbus
  6: 58 xen-dyn-event hvc_console
  7: 414227 xen-dyn-event blkif
  8: 36272 xen-dyn-event blkif
  9: 459 xen-dyn-event blkif
 10: 7447465 xen-dyn-event eth0
NMI: 0 Non-maskable interrupts
LOC: 0 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
PND: 0 Performance pending work
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 0 Machine check polls
ERR: 0
MIS: 0
diegozaka

diegozaka

2012-10-02 03:35

reporter   ~0015867

Having the same issue on a HP Proliant dual L5420 using centos 6.2, kernel ver 2.6.32-279. Had to downgrade to the stock kernel 2.6.32-220
toracat

toracat

2012-10-02 16:50

manager   ~0015871

Would any of you be able to try the latest stable kernel from kernel.org and see if the issue is resolved there? This can be done by installing kernel-ml from ELRepo:

http://elrepo.org/tiki/kernel-ml

Because it installs in parallel with the existing kernel, it's easy to install/uninstall without any conflicts. Currently the latest is kernel 3.5.4.
crashatau

crashatau

2012-10-19 02:16

reporter   ~0015961

I can confirm this was happening on my HPC cluster.
ksoftirqd was interrupting the CPUs on an irregular basis and hobbling CPU throughput. You can imagine MPI jobs on multiple nodes perform purely when one node suddenly halts processing for a second.

The "nohz=off" trick significantly improved performance by 50%.

Currently running :
Linux 2.6.32-279.2.1.el6.x86_64 x86_64 x86_64 x86_64 GNU/Linux
CentOS 6.3

I'm also going to roll back to the stock kernel to see if this gives even better performance. If I have time I will try the new kernel from Toracat.
crashatau

crashatau

2012-10-19 02:43

reporter   ~0015962

The stock kernel had the same performance as the 2.6.32-279.2.1.el6.x86_64 with the nohz=off fix.

My hardware is SGI H2106-G7 compute nodes...
AMD Opteron(TM) Processor 6238 @ 2.6Ghz (4 x 12 Cores)
128GB RAM
u235

u235

2012-10-23 05:27

reporter   ~0015977

I'm having this issue as of 2.6.32-220.13.1.el6.x86_64. Relevant hardware overview:
Dual Xeon L5420 (stepping 10)
32GB RAM

toracat, I briefly tested the latest mainline kernel (3.6.3-1 as of this writing). I don't see this issue with 3.6.3-1. I ran some quick tests on various kernels by booting and running 6 cpuburn threads for 10 minutes. Here is the cpu time of the ksoftirqd processes and the output of /proc/interrupts for the following kernels:
2.6.32-220.4.1.el6.x86_64 (unaffected)
2.6.32-220.13.1.el6.x86_64 (affected)
2.6.32-279.9.1.el6.x86_64 (affected)
3.6.3-1.el6.elrepo.x86_64 (unaffected)

--------<2.6.32-220.4.1.el6.x86_64>--------
root 4 2 0 19:22 ? 00:00:00 [ksoftirqd/0]
root 9 2 0 19:22 ? 00:00:00 [ksoftirqd/1]
root 13 2 0 19:22 ? 00:00:00 [ksoftirqd/2]
root 17 2 0 19:22 ? 00:00:00 [ksoftirqd/3]
root 21 2 0 19:22 ? 00:00:00 [ksoftirqd/4]
root 25 2 0 19:22 ? 00:00:00 [ksoftirqd/5]
root 29 2 0 19:22 ? 00:00:00 [ksoftirqd/6]
root 33 2 0 19:22 ? 00:00:00 [ksoftirqd/7]

           CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
  0: 147 2 0 1 0 1 1 0 IO-APIC-edge timer
  1: 1 1 1 2 0 1 1 1 IO-APIC-edge i8042
  4: 1 2 1 2 1 2 0 2 IO-APIC-edge serial
  8: 0 0 0 0 1 0 0 0 IO-APIC-edge rtc0
  9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
 12: 57 21 54 19 20 17 21 19 IO-APIC-edge i8042
 18: 49 53 48 52 53 45 43 36 IO-APIC-fasteoi radeon
 20: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
 23: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5
 37: 2858 1828 2004 1832 1826 1840 1838 1849 IO-APIC-fasteoi aacraid
 52: 1 0 0 0 1 0 0 1 PCI-MSI-edge ioat-msix
 53: 1 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
 54: 0 0 0 0 1 0 0 0 PCI-MSI-edge ioat-msix
 55: 0 0 1 0 0 0 0 0 PCI-MSI-edge ioat-msix
 56: 0 0 0 0 1 0 0 0 PCI-MSI-edge eth0
 57: 121 2 441 2 835 1 270 2 PCI-MSI-edge eth0-rx-0
 58: 198 0 154 2 296 4 687 3 PCI-MSI-edge eth0-rx-1
 59: 403 5 511 3 171 19 609 13 PCI-MSI-edge eth0-rx-2
 60: 365 8 237 0 193 5 208 2 PCI-MSI-edge eth0-rx-3
 61: 102 4 142 6 271 8 139 0 PCI-MSI-edge eth0-tx-0
 62: 163 16 127 12 152 6 243 5 PCI-MSI-edge eth0-tx-1
 63: 170 2 222 4 149 2 142 3 PCI-MSI-edge eth0-tx-2
 64: 202 4 334 4 235 3 71 3 PCI-MSI-edge eth0-tx-3
 65: 1 0 0 0 0 0 0 0 PCI-MSI-edge eth1
 66: 202 11 123 12 592 11 144 138 PCI-MSI-edge eth1-rx-0
 67: 355 6 433 4 344 1 97 1 PCI-MSI-edge eth1-rx-1
 68: 142 11 183 3 128 1 221 2 PCI-MSI-edge eth1-rx-2
 69: 327 13 120 7 127 7 111 1 PCI-MSI-edge eth1-rx-3
 70: 209 2 107 5 137 1 211 3 PCI-MSI-edge eth1-tx-0
 71: 39 7 202 8 225 13 286 19 PCI-MSI-edge eth1-tx-1
 72: 447 10 199 142 109 12 229 10 PCI-MSI-edge eth1-tx-2
 73: 207 2 206 3 129 12 161 8 PCI-MSI-edge eth1-tx-3
NMI: 205 149 338 340 150 236 348 352 Non-maskable interrupts
LOC: 190039 144914 304318 306373 144734 235286 312890 318355 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 205 149 338 340 150 236 348 352 Performance monitoring interrupts
PND: 0 0 0 0 0 0 0 0 Performance pending work
RES: 14400 19165 3156 3599 25883 13112 903 169 Rescheduling interrupts
CAL: 2307 5697 21631 18348 2720 2895 16087 38350 Function call interrupts
TLB: 316 390 817 820 471 413 857 874 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 5 5 5 5 5 5 5 5 Machine check polls
ERR: 0
MIS: 0
--------</2.6.32-220.4.1.el6.x86_64>--------

--------<2.6.32-220.13.1.el6.x86_64>--------
root 4 2 13 20:28 ? 00:00:53 [ksoftirqd/0]
root 9 2 8 20:28 ? 00:00:34 [ksoftirqd/1]
root 13 2 0 20:28 ? 00:00:00 [ksoftirqd/2]
root 17 2 0 20:28 ? 00:00:02 [ksoftirqd/3]
root 21 2 9 20:28 ? 00:00:39 [ksoftirqd/4]
root 25 2 11 20:28 ? 00:00:44 [ksoftirqd/5]
root 29 2 0 20:28 ? 00:00:02 [ksoftirqd/6]
root 33 2 0 20:28 ? 00:00:01 [ksoftirqd/7]

           CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
  0: 147 1 0 0 1 0 0 0 IO-APIC-edge timer
  1: 0 1 1 1 1 2 1 1 IO-APIC-edge i8042
  4: 2 3 0 1 1 1 2 2 IO-APIC-edge serial
  8: 1 0 0 0 0 0 0 0 IO-APIC-edge rtc0
  9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
 12: 17 18 15 16 14 14 17 13 IO-APIC-edge i8042
 18: 38 33 32 41 35 43 36 45 IO-APIC-fasteoi radeon
 20: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
 23: 899 903 902 940 26977 907 26954 924 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5
 37: 1846 1855 1864 1810 2011 2583 2018 1822 IO-APIC-fasteoi aacraid
 52: 1 0 0 0 1 0 0 1 PCI-MSI-edge ioat-msix
 53: 0 0 1 0 0 0 0 0 PCI-MSI-edge ioat-msix
 54: 0 0 0 0 0 0 1 0 PCI-MSI-edge ioat-msix
 55: 0 1 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
 56: 0 1 0 0 0 0 0 0 PCI-MSI-edge eth0
 57: 60 9 18 6 119 4 114 12 PCI-MSI-edge eth0-rx-0
 58: 8 7 675 77 9 3 4 9 PCI-MSI-edge eth0-rx-1
 59: 229 28 7 14 60 4 49 11 PCI-MSI-edge eth0-rx-2
 60: 84 4 39 2 37 11 15 0 PCI-MSI-edge eth0-rx-3
 61: 30 5 56 9 40 14 225 11 PCI-MSI-edge eth0-tx-0
 62: 93 4 63 6 4 14 38 10 PCI-MSI-edge eth0-tx-1
 63: 44 5 61 3 24 2 64 8 PCI-MSI-edge eth0-tx-2
 64: 44 3 28 3 97 2 15 0 PCI-MSI-edge eth0-tx-3
 65: 0 0 0 1 0 0 0 0 PCI-MSI-edge eth1
 66: 31 1 51 3 82 16 78 7 PCI-MSI-edge eth1-rx-0
 67: 68 1 12 3 185 2 78 4 PCI-MSI-edge eth1-rx-1
 68: 56 3 33 1 56 1 39 11 PCI-MSI-edge eth1-rx-2
 69: 37 4 11 3 121 3 29 3 PCI-MSI-edge eth1-rx-3
 70: 46 5 61 3 37 4 46 2 PCI-MSI-edge eth1-tx-0
 71: 24 8 41 6 51 5 104 2 PCI-MSI-edge eth1-tx-1
 72: 58 27 88 9 23 9 22 23 PCI-MSI-edge eth1-tx-2
 73: 27 3 82 7 6 3 60 3 PCI-MSI-edge eth1-tx-3
NMI: 202 213 349 342 164 160 340 345 Non-maskable interrupts
LOC: 185025 195663 310762 306282 150909 150085 301990 307228 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 202 213 349 342 164 160 340 345 Performance monitoring interrupts
PND: 0 0 0 0 0 0 0 0 Performance pending work
RES: 19050 13272 328 1765 17461 17051 2489 1291 Rescheduling interrupts
CAL: 5583 3453 20145 21123 2142 2315 17195 16306 Function call interrupts
TLB: 381 400 865 736 376 372 808 798 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 2 2 2 2 Machine check polls
ERR: 0
MIS: 0
--------</2.6.32-220.13.1.el6.x86_64>--------

--------<2.6.32-279.9.1.el6.x86_64>--------
root 4 2 5 20:43 ? 00:00:28 [ksoftirqd/0]
root 9 2 10 20:43 ? 00:00:50 [ksoftirqd/1]
root 13 2 0 20:43 ? 00:00:01 [ksoftirqd/2]
root 17 2 0 20:43 ? 00:00:00 [ksoftirqd/3]
root 21 2 11 20:43 ? 00:00:57 [ksoftirqd/4]
root 25 2 11 20:43 ? 00:00:55 [ksoftirqd/5]
root 29 2 0 20:43 ? 00:00:00 [ksoftirqd/6]
root 33 2 0 20:43 ? 00:00:00 [ksoftirqd/7]

           CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
  0: 147 0 0 1 1 0 0 0 IO-APIC-edge timer
  1: 0 1 1 1 1 2 1 1 IO-APIC-edge i8042
  4: 1 4 2 0 0 2 2 1 IO-APIC-edge serial
  8: 1 0 0 0 0 0 0 0 IO-APIC-edge rtc0
  9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
 12: 17 18 16 14 14 15 15 15 IO-APIC-edge i8042
 18: 40 30 40 45 44 36 29 37 IO-APIC-fasteoi radeon
 20: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
 22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
 23: 24829 527 24709 529 492 2275 503 2290 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5
 37: 1834 1824 1834 1812 2857 1825 2054 1826 IO-APIC-fasteoi aacraid
 52: 1 0 1 0 0 0 1 0 PCI-MSI-edge ioat-msix
 53: 0 0 0 0 1 0 0 0 PCI-MSI-edge ioat-msix
 54: 0 0 1 0 0 0 0 0 PCI-MSI-edge ioat-msix
 55: 0 0 0 0 0 0 1 0 PCI-MSI-edge ioat-msix
 56: 0 0 0 1 0 0 0 0 PCI-MSI-edge eth0
 57: 103 2 29 4 189 2 68 30 PCI-MSI-edge eth0-rx-0
 58: 18 10 452 7 32 71 182 6 PCI-MSI-edge eth0-rx-1
 59: 175 7 195 3 120 4 85 3 PCI-MSI-edge eth0-rx-2
 60: 58 1 42 12 64 2 63 2 PCI-MSI-edge eth0-rx-3
 61: 75 2 8 8 106 1 127 9 PCI-MSI-edge eth0-tx-0
 62: 31 3 4 10 53 7 149 4 PCI-MSI-edge eth0-tx-1
 63: 64 6 77 1 38 1 49 2 PCI-MSI-edge eth0-tx-2
 64: 48 7 89 9 56 4 37 13 PCI-MSI-edge eth0-tx-3
 65: 0 0 0 0 1 0 0 0 PCI-MSI-edge eth1
 66: 31 1 178 3 72 3 19 2 PCI-MSI-edge eth1-rx-0
 67: 123 0 113 4 123 1 58 3 PCI-MSI-edge eth1-rx-1
 68: 55 7 56 6 27 0 96 4 PCI-MSI-edge eth1-rx-2
 69: 185 4 16 2 30 6 10 3 PCI-MSI-edge eth1-rx-3
 70: 80 4 52 5 63 2 42 1 PCI-MSI-edge eth1-tx-0
 71: 23 15 153 8 31 3 26 3 PCI-MSI-edge eth1-tx-1
 72: 16 44 24 5 125 4 63 5 PCI-MSI-edge eth1-tx-2
 73: 118 4 47 4 26 6 69 4 PCI-MSI-edge eth1-tx-3
NMI: 248 171 340 347 132 184 345 346 Non-maskable interrupts
LOC: 228290 157124 304049 309534 126075 170864 307000 308774 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 248 171 340 347 132 184 345 346 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RES: 13184 16974 1868 284 21760 25834 458 498 Rescheduling interrupts
CAL: 3922 3298 26488 17388 2046 4886 14364 19357 Function call interrupts
TLB: 422 382 852 802 340 402 778 782 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 2 2 2 2 Machine check polls
ERR: 0
MIS: 0
--------</2.6.32-279.9.1.el6.x86_64>--------

--------<3.6.3-1.el6.elrepo.x86_64>--------
root 3 2 0 20:01 ? 00:00:00 [ksoftirqd/0]
root 13 2 0 20:01 ? 00:00:00 [ksoftirqd/1]
root 18 2 0 20:01 ? 00:00:00 [ksoftirqd/2]
root 23 2 0 20:01 ? 00:00:00 [ksoftirqd/3]
root 28 2 0 20:01 ? 00:00:00 [ksoftirqd/4]
root 33 2 0 20:01 ? 00:00:00 [ksoftirqd/5]
root 38 2 0 20:01 ? 00:00:00 [ksoftirqd/6]
root 43 2 0 20:01 ? 00:00:00 [ksoftirqd/7]

            CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
   0: 152 0 0 0 0 0 0 0 IO-APIC-edge timer
   1: 1 3 1 2 2 3 2 0 IO-APIC-edge i8042
   4: 3 1 0 3 2 1 1 2 IO-APIC-edge serial
   8: 0 0 0 0 0 0 0 1 IO-APIC-edge rtc0
   9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
  12: 44 57 54 52 46 54 56 46 IO-APIC-edge i8042
  18: 42 38 38 34 33 39 40 38 IO-APIC-fasteoi radeon
  20: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
  21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
  22: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
  23: 85 341 85 342 85 86 86 86 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5
  37: 1883 1877 1884 1885 1891 5492 1876 5474 IO-APIC-fasteoi aacraid
  68: 1 0 0 0 1 0 0 1 PCI-MSI-edge ioat-msix
  69: 0 0 1 0 0 0 0 0 PCI-MSI-edge ioat-msix
  70: 0 0 0 0 0 0 1 0 PCI-MSI-edge ioat-msix
  71: 0 1 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
  72: 0 0 0 0 0 0 0 1 PCI-MSI-edge eth0
  73: 737 12 3 3 8 3 22 3 PCI-MSI-edge eth0-rx-0
  74: 314 3 6 4 2 4 1 1 PCI-MSI-edge eth0-rx-1
  75: 6 1 605 7 11 1 8 5 PCI-MSI-edge eth0-rx-2
  76: 5 5 2 1 263 1 2 4 PCI-MSI-edge eth0-rx-3
  77: 6 2 10 4 8 6 265 6 PCI-MSI-edge eth0-tx-0
  78: 0 4 254 7 4 2 1 1 PCI-MSI-edge eth0-tx-1
  79: 5 11 9 3 258 3 6 8 PCI-MSI-edge eth0-tx-2
  80: 5 2 0 1 1 3 396 5 PCI-MSI-edge eth0-tx-3
  81: 0 0 0 0 0 1 0 0 PCI-MSI-edge eth1
  82: 5 3 0 340 5 5 3 3 PCI-MSI-edge eth1-rx-0
  83: 2 0 4 5 3 278 2 3 PCI-MSI-edge eth1-rx-1
  84: 5 262 2 3 3 1 5 3 PCI-MSI-edge eth1-rx-2
  85: 5 264 2 3 6 12 2 6 PCI-MSI-edge eth1-rx-3
  86: 7 8 9 6 5 263 5 16 PCI-MSI-edge eth1-tx-0
  87: 2 2 4 250 5 7 2 5 PCI-MSI-edge eth1-tx-1
  88: 6 8 9 48 4 8 9 292 PCI-MSI-edge eth1-tx-2
  89: 5 9 3 1 1 3 3 250 PCI-MSI-edge eth1-tx-3
 NMI: 7 345 364 364 362 24 366 363 Non-maskable interrupts
 LOC: 23968 301151 317191 318696 314449 36637 322704 320473 Local timer interrupts
 SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
 PMI: 7 345 364 364 362 24 366 363 Performance monitoring interrupts
 IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
 RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries
 RES: 144234 2883 1045 2463 2334 2362 1198 1296 Rescheduling interrupts
 CAL: 1795 2301 11385 34009 1816 2897 35682 11844 Function call interrupts
 TLB: 0 0 0 0 0 0 0 0 TLB shootdowns
 TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
 THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
 MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
 MCP: 2 2 2 2 2 2 2 2 Machine check polls
 ERR: 0
 MIS: 0
--------</3.6.3-1.el6.elrepo.x86_64>--------

Sorry for all the text, but hopefully it's helpful? Please let me know if there's anything else I could try that may prove useful. Cheers.
opoplawski

opoplawski

2012-10-23 16:22

reporter   ~0015978

One problem with the nohz=off workaround is that it appears to break load average computation.
toracat

toracat

2012-10-23 16:54

manager   ~0015979

@u235,

Your result with the mainline kernel 3.6.3 is looking good. I suggest that, with that info on hand, you file a bug report upstream at http://bugzilla.redhat.com .

Hopefully the issue gets fixed upstream so that it comes down to the CentOS kernel.
u235

u235

2012-10-26 22:56

reporter   ~0015986

Bug filed with RedHat: https://bugzilla.redhat.com/show_bug.cgi?id=870573
drunreal

drunreal

2012-11-15 16:32

reporter   ~0016028

None of our AMD systems demonstrate this problem. Only our Intel systems. Perhaps a clue...
AlanBartlett

AlanBartlett

2012-11-15 17:54

reporter   ~0016029

@drunreal -- That may prove to be significant. Is Comment 12 to the Upstream bug report (https://bugzilla.redhat.com/show_bug.cgi?id=870573#c12) from you? If not, please also add your evidence.
NetCaptive

NetCaptive

2012-12-12 04:06

reporter   ~0016131

Having this issue on an AMD based HP DL380 (G3 i think -- oldie but goodie).

%lspci
00:03.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
00:04.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
00:04.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
00:04.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
00:07.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:07.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:08.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
00:08.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:00.0 USB controller: Advanced Micro Devices [AMD] AMD-8111 USB OHCI (rev 0b)
01:00.1 USB controller: Advanced Micro Devices [AMD] AMD-8111 USB OHCI (rev 0b)
01:02.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 01)
01:02.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 01)
01:03.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Rage XL (rev 27)
02:04.0 RAID bus controller: Compaq Computer Corporation Smart Array 64xx (rev 01)
04:09.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
04:09.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
04:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
04:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
05:07.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
05:07.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) (rev 01)

%uname -r
2.6.32-220.13.1.el6.x86_64

%top
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    9 root 20 0 0 0 0 S 49.7 0.0 41888:57 ksoftirqd/1
    4 root 20 0 0 0 0 S 30.2 0.0 13789:30 ksoftirqd/0
   13 root 20 0 0 0 0 S 0.0 0.0 801:13.84 ksoftirqd/2

%vmstat 1 30
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 1 0 0 5005032 286052 10149884 0 0 108 96 0 0 24 1 75 0 0
 1 0 0 5005120 286052 10149884 0 0 0 0 1279 237 24 1 75 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1364 263 23 2 76 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1363 292 27 1 72 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1253 215 24 1 75 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1267 252 26 2 72 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1214 224 26 1 73 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1334 262 24 1 75 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1242 240 24 2 75 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1289 255 25 1 74 0 0
 1 0 0 5005320 286052 10149884 0 0 0 0 1231 241 27 1 71 0 0
 1 0 0 5005032 286052 10149884 0 0 0 0 1584 509 21 2 77 0 0
 1 0 0 5005032 286052 10149884 0 0 0 0 1330 269 29 1 70 0 0
 1 0 0 5005056 286052 10149884 0 0 0 0 1265 230 22 1 77 0 0
 2 0 0 5005064 286052 10149884 0 0 0 0 1237 217 24 1 75 0 0
 1 0 0 5005064 286052 10149884 0 0 0 0 1254 243 25 1 74 0 0
 2 0 0 5005064 286052 10149884 0 0 0 24 1378 328 25 1 74 0 0
 1 0 0 5005080 286052 10149884 0 0 0 0 1201 227 25 1 74 0 0
 2 0 0 5005088 286052 10149884 0 0 0 0 1200 240 22 1 77 0 0
 1 0 0 5005088 286052 10149884 0 0 0 0 1159 207 25 1 74 0 0
 1 0 0 5005116 286052 10149884 0 0 0 0 1161 209 27 1 72 0 0
 1 0 0 5005116 286052 10149884 0 0 0 0 1157 212 22 1 76 0 0
 1 0 0 5005116 286052 10149884 0 0 0 0 1235 254 23 1 77 0 0
 1 0 0 5005116 286052 10149884 0 0 0 0 1303 292 28 1 71 0 0
 1 0 0 5005116 286052 10149884 0 0 0 0 1236 238 29 1 71 0 0
 1 0 0 5005116 286052 10149884 0 0 0 0 1245 247 21 1 79 0 0
 1 0 0 5004992 286052 10149884 0 0 0 0 1255 251 27 1 72 0 0
 1 0 0 5004992 286052 10149884 0 0 0 0 1277 278 22 1 77 0 0
 1 0 0 5004992 286052 10149884 0 0 0 0 1242 248 26 1 73 0 0
 1 0 0 5004992 286052 10149884 0 0 0 0 1240 262 22 1 77 0 0

%cat /proc/interrupts
           CPU0 CPU1 CPU2 CPU3
  0: 142 0 0 0 IO-APIC-edge timer
  1: 0 0 0 8 IO-APIC-edge i8042
  8: 0 0 0 0 IO-APIC-edge rtc0
  9: 0 0 0 0 IO-APIC-fasteoi acpi
 12: 1 0 0 135 IO-APIC-edge i8042
 14: 0 2 0 113 IO-APIC-edge pata_amd
 15: 0 0 0 0 IO-APIC-edge pata_amd
 17: 0 0 0 0 IO-APIC-fasteoi hpilo
 19: 0 0 0 0 IO-APIC-fasteoi ohci_hcd:usb1, ohci_hcd:usb2
 24: 107198300 650487 5002562 14291 IO-APIC-fasteoi cciss0
 34: 440883 7378280 1532221884 26 IO-APIC-fasteoi eth0
NMI: 7293 7820 63331 236450 Non-maskable interrupts
LOC: 1088561492 603847137 367510055 1356207289 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 7293 7820 63331 236450 Performance monitoring interrupts
PND: 0 0 0 0 Performance pending work
RES: 173273751 62072783 62427335 21025311 Rescheduling interrupts
CAL: 406413 35308247 8857 12397 Function call interrupts
TLB: 3120087 805997 1365901 486461 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 63274 63274 63274 63274 Machine check polls
ERR: 0
MIS: 0

%cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : AMD Opteron(tm) Processor 275
stepping : 2
cpu MHz : 2204.767
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni cmp_legacy
bogomips : 4409.53
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : AMD Opteron(tm) Processor 275
stepping : 2
cpu MHz : 2204.767
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 2
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni cmp_legacy
bogomips : 4409.21
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 2
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : AMD Opteron(tm) Processor 275
stepping : 2
cpu MHz : 2204.767
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni cmp_legacy
bogomips : 4409.23
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : AMD Opteron(tm) Processor 275
stepping : 2
cpu MHz : 2204.767
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni cmp_legacy
bogomips : 4409.23
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

Updated to 2.6.32-279.14.1 just now, but can't test it until i can schedule a reboot on this box.
jdshewey

jdshewey

2012-12-28 00:59

reporter   ~0016187

I too am experiencing this issue and it does appear to be an upstream problem as I am running Ubuntu. It sounds like the commonality is HP hardware and kernel version. I found this thread: http://forums.fedoraforum.org/archive/index.php/t-70926.html and this bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/575392 for a similar issue. So who among us is running an HP server, what model and what is your RAID configuration?

I'm on an HP ProLiant DL380 G3 with RAID5. Running a 2.8Ghz Intel Chip.
llevet

llevet

2013-02-21 13:25

reporter   ~0016513

Hi, the bug is resolved.

Bug 836964 - ksoftirqd quite busy on some systems.
https://bugzilla.redhat.com/show_bug.cgi?id=836964

Report from Bugzilla :
errata-xmlrpc 2013-02-21 01:31:13 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0496.html



The kernel containt bug fixe :
kernel-2.6.32-358.el6

So, Centos must release this kernel now.

Ludovic.
toracat

toracat

2013-02-21 13:41

manager   ~0016514

kernel-2.6.32-358.el6 is part of RHEL 6.4. It will be in CentOS 6.4 when it's released.
llevet

llevet

2013-02-21 14:14

reporter   ~0016515

So, you can find the patch to the problem :

http://kernel.opensuse.org/cgit/kernel/commit/?id=c5d753a55ac92e09816d410cd17093813f1a904b
toracat

toracat

2013-07-15 18:27

manager   ~0017670

According to the info provided in this bug tracker as well as the following upstream KB article:

https://access.redhat.com/site/solutions/302623

the issue has been resolved in 6.4. Closing as 'resolved'. Feel free to reopen if this was found not to be the case.

Issue History

Date Modified Username Field Change
2012-07-10 12:47 jiandal New Issue
2012-07-10 14:24 bmalynovytch Note Added: 0015390
2012-07-10 14:28 jiandal Note Added: 0015391
2012-07-10 15:07 bmalynovytch Note Added: 0015393
2012-07-10 15:12 jiandal Note Added: 0015394
2012-07-12 17:49 jiandal Note Added: 0015417
2012-08-29 13:31 smartgig Note Added: 0015717
2012-08-29 14:28 bmalynovytch Note Added: 0015718
2012-08-29 17:19 smartgig Note Added: 0015721
2012-08-29 17:23 jiandal Note Added: 0015722
2012-09-12 20:44 Nerigal Note Added: 0015767
2012-09-13 10:24 smartgig Note Added: 0015769
2012-09-13 13:45 Nerigal Note Added: 0015771
2012-09-28 17:24 sigtrap Note Added: 0015857
2012-10-02 03:35 diegozaka Note Added: 0015867
2012-10-02 16:50 toracat Note Added: 0015871
2012-10-19 02:16 crashatau Note Added: 0015961
2012-10-19 02:43 crashatau Note Added: 0015962
2012-10-23 05:27 u235 Note Added: 0015977
2012-10-23 16:22 opoplawski Note Added: 0015978
2012-10-23 16:54 toracat Note Added: 0015979
2012-10-26 22:56 u235 Note Added: 0015986
2012-11-15 16:32 drunreal Note Added: 0016028
2012-11-15 17:54 AlanBartlett Note Added: 0016029
2012-12-12 04:06 NetCaptive Note Added: 0016131
2012-12-28 00:59 jdshewey Note Added: 0016187
2013-02-21 13:25 llevet Note Added: 0016513
2013-02-21 13:41 toracat Note Added: 0016514
2013-02-21 14:14 llevet Note Added: 0016515
2013-07-15 18:27 toracat Note Added: 0017670
2013-07-15 18:27 toracat Status new => resolved
2013-07-15 18:27 toracat Resolution open => fixed
2013-07-15 18:27 toracat Fixed in Version => 6.4