View Issue Details

IDProjectCategoryView StatusLast Update
0014621CentOS-7kernelpublic2018-10-18 19:08
Reporterjeferson.cerutti 
PriorityurgentSeveritymajorReproducibilityalways
Status newResolutionopen 
Platformx86_64OSCentosOS Version7.4.1708 (Core)
Product Version7.4.1708 
Target VersionFixed in Version 
Summary0014621: Disk write speed drops dramatically after kernel upgrade.
Descriptionafter upgrading to the kernel to 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux, dropped to 20MB whereas in the kernel version 3.10.0-693.el7.x86_64 a record is 80MB.
Steps To Reproduceexecute dd if=/dev/zero of=/iotest bs=129 count=8000 in both kernel
Additional Informationlost performance
Tags7.4
abrt_hash
URL

Activities

jeferson.cerutti

jeferson.cerutti

2018-03-25 15:15

reporter  

N3WWN

N3WWN

2018-10-18 19:08

reporter   ~0032941

Hi jeferson.cerutti!

I was able to replicate the performance degradation that you are reporting and have tracked it back to Spectre mitigation which has been added to the kernel.

The performance hit appeared between kernel-3.10.0-693.11.1.el7 and kernel-3.10.0-693.11.6.el7

The primary difference between these two kernels is the addition of the KAISER/KPTI (Kernel Page Table Isolation) patches, which increase security at the cost of syscall/interrupt/exception performance.

The dd tests that you are executing cause a lot of system calls and interrupts to be generated due to the small block size that is being written to the /iotest file.

There are two things that you may be able to leverage in order to decrease the performance hit with the newer KPTI-patched kernels:

1) Optimize your applications to generate fewer system calls and interrupts.

On the system that I was testing this out on, I was able to almost entirely negate the performance hit due to KPTI by increasing the block size from 128 bytes to 1536 bytes. Doing this, it was my block I/O that caused the bottleneck due to a 12x increase in the data stored in the file with an equal number of system calls. I also increased the count from 8000 to 4000000 so the dd would take several seconds to run.

Depending on what application(s) you are using on your systems, you may be able to configure them or modify them to do more work per syscall or interrupt. If you're familiar with databases, think of this as using transactions to combine operations into a single commit versus committing each operation individually.

and

2) If the additional security of a KPTI kernel is not required in your environment, you could consider disabling KPTI at system boot time by adding the 'nopti' parameter to the kernel command line. This does not restore the systems performance to that of the earlier, pre-Spectre-mitigating kernels, but it may minimize the performance decrease.

My testing showed that I could reduce the performance hit about about 2/3 by booting with the 'nopti' parameter. This *may* mean that the 71% decrease in performance that you reported (71 MB/s -> 21 MB/s) *may* improve to only a 23% decrease (71 MB/s -> 55 MB/s) for this dd test case.

-Rich Alloway (Rogue Wave)

See also https://lwn.net/Articles/738975/ and, linked from that page, https://lwn.net/Articles/738997/ for more information.

Issue History

Date Modified Username Field Change
2018-03-25 15:15 jeferson.cerutti New Issue
2018-03-25 15:15 jeferson.cerutti File Added: version7kernel 3.10.0-693.21.1.el7.x86_64.png
2018-03-25 15:15 jeferson.cerutti File Added: version7kernel3.10.0-693.el7.x86_64.png
2018-03-25 15:15 jeferson.cerutti Tag Attached: 7.4
2018-10-18 19:08 N3WWN Note Added: 0032941