View Issue Details

IDProjectCategoryView StatusLast Update
0015723CentOS-7sysstatpublic2019-09-01 16:24
Reporteryoor699 
PriorityhighSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0015723: high disk utilization
Descriptionhi,

After last kernel upgrade on several cPanel servers using NVMe SSD, we get a common high disk utilization according to iostat output, confirmed also by dstat -cd --disk-util (in case iostat would have a caveat : https://brooker.co.za/blog/2014/07/04/iostat-pct.html)
It affects both disks.

Issue is from this last kernel
3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

raid1 is fine :
cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0]
      356576192 blocks [2/2] [UU]
      bitmap: 3/3 pages [12KB], 65536KB chunk

md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
      71678912 blocks [2/2] [UU]

unused devices: <none>

Initial investigation :
- hardware is in perfect state in all servers, they use NVMe SSD for ages.
- such high level of disk utilization has no justification in term of running processes from any user, or any scripts, no iowait also at all
- what is reported by linux may be wrong, but this lets us a visible symptom that we can't trust, neither ignoring it util we have a realiable explanation.

we observed that :
- an upgrade centos on a basic server (with mechanic hdd in this case) is not causing this at reboot the following
- an excessive update of file /.readahead, this is not happening on other cpanel server using cloudlinux by ie
(our servers use no hard raid card)

We tried to disable readahead and reboot in case, but no change at all :
systemctl disable systemd-readahead-collect.service
systemctl disable systemd-readahead-replay.service

We contacted cPanel support, investigated deeply with 2 sysadmin, it appears that the answer will only come from centos 7 kernel developer

Thanks for your attention,
John
Steps To Reproduce3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
NVMe SSD
raid1
check output dstat -cd --disk-util
Tagscentos 7
abrt_hash
URL

Activities

pgreco

pgreco

2019-01-20 11:36

developer   ~0033641

The first step would be to determine which was the last kernel this worked as expected, and if it is in fact the kernel, or another program that was updated during the same period.
yoor699

yoor699

2019-01-20 12:03

reporter   ~0033643

Hello
See screenshot, the 862.14 is the last one working properly


Here is the full list from yum.log update new with kernel install (kernel-3.10.0-957.1.3.el7.x86_64), and issue appeared after :

Dec 08 17:48:43 Updated: libgcc-4.8.5-36.el7.x86_64
Dec 08 17:48:43 Updated: 1:grub2-common-2.02-0.76.el7.centos.noarch
Dec 08 17:48:43 Updated: 32:bind-license-9.9.4-72.el7.noarch
Dec 08 17:48:43 Updated: 1:grub2-pc-modules-2.02-0.76.el7.centos.noarch
Dec 08 17:48:43 Updated: tzdata-2018g-1.el7.noarch
Dec 08 17:48:47 Updated: glibc-common-2.17-260.el7.x86_64
Dec 08 17:48:47 Updated: glibc-2.17-260.el7.x86_64
Dec 08 17:48:48 Updated: bash-4.2.46-31.el7.x86_64
Dec 08 17:48:48 Updated: libsepol-2.5-10.el7.x86_64
Dec 08 17:48:48 Updated: libselinux-2.5-14.1.el7.x86_64
Dec 08 17:48:48 Updated: zlib-1.2.7-18.el7.x86_64
Dec 08 17:48:48 Updated: libcom_err-1.42.9-13.el7.x86_64
Dec 08 17:48:48 Updated: krb5-libs-1.15.1-34.el7.x86_64
Dec 08 17:48:48 Updated: 1:openssl-libs-1.0.2k-16.el7.x86_64
Dec 08 17:48:48 Updated: coreutils-8.22-23.el7.x86_64
Dec 08 17:48:49 Updated: 4:perl-macros-5.16.3-293.el7.x86_64
Dec 08 17:48:49 Updated: 4:perl-libs-5.16.3-293.el7.x86_64
Dec 08 17:48:50 Updated: 4:perl-5.16.3-293.el7.x86_64
Dec 08 17:48:51 Updated: ea-openssl-1.0.2q-1.1.1.cpanel.x86_64
Dec 08 17:48:51 Updated: ea-php70-php-common-7.0.32-2.2.7.cpanel.x86_64
Dec 08 17:48:51 Updated: freetype-2.8-12.el7.x86_64
Dec 08 17:48:51 Updated: libstdc++-4.8.5-36.el7.x86_64
Dec 08 17:48:51 Updated: audit-libs-2.8.4-4.el7.x86_64
Dec 08 17:48:51 Updated: libuuid-2.23.2-59.el7.x86_64
Dec 08 17:48:51 Updated: libjpeg-turbo-1.2.90-6.el7.x86_64
Dec 08 17:48:51 Updated: libblkid-2.23.2-59.el7.x86_64
Dec 08 17:48:51 Updated: libssh2-1.4.3-12.el7.x86_64
Dec 08 17:48:51 Updated: elfutils-libelf-0.172-2.el7.x86_64
Dec 08 17:48:51 Updated: ea-libnghttp2-1.32.0-1.1.3.cpanel.x86_64
Dec 08 17:48:52 Updated: python-libs-2.7.5-76.el7.x86_64
Dec 08 17:48:52 Updated: python-2.7.5-76.el7.x86_64
Dec 08 17:48:53 Updated: libicu-50.1.2-17.el7.x86_64
Dec 08 17:48:53 Updated: GeoIP-1.5.0-13.el7.x86_64
Dec 08 17:48:53 Updated: ea-apr-1.6.3-2.2.4.cpanel.x86_64
Dec 08 17:48:53 Updated: ea-php55-libc-client-2007f-13.13.4.cpanel.x86_64
Dec 08 17:48:54 Updated: centos-release-7-6.1810.2.el7.centos.x86_64
Dec 08 17:48:54 Updated: libxcb-1.13-1.el7.x86_64
Dec 08 17:48:54 Updated: libmount-2.23.2-59.el7.x86_64
Dec 08 17:48:54 Updated: glib2-2.56.1-2.el7.x86_64
Dec 08 17:48:54 Updated: libsemanage-2.5-14.el7.x86_64
Dec 08 17:48:54 Updated: 1:net-snmp-libs-5.7.2-37.el7.x86_64
Dec 08 17:48:54 Updated: file-libs-5.11-35.el7.x86_64
Dec 08 17:48:54 Updated: file-5.11-35.el7.x86_64
Dec 08 17:48:54 Updated: 1:findutils-4.5.11-6.el7.x86_64
Dec 08 17:48:54 Updated: ea-apr-util-1.6.1-4.4.4.cpanel.x86_64
Dec 08 17:48:54 Updated: 32:bind-libs-9.9.4-72.el7.x86_64
Dec 08 17:48:55 Updated: binutils-2.27-34.base.el7.x86_64
Dec 08 17:48:55 Updated: mesa-libglapi-18.0.5-3.el7.x86_64
Dec 08 17:48:55 Updated: lm_sensors-libs-3.4.0-6.20160601gitf9185e5.el7.x86_64
Dec 08 17:48:55 Updated: setup-2.8.71-10.el7.noarch
Dec 08 17:48:55 Updated: 2:shadow-utils-4.1.5.1-25.el7.x86_64
Dec 08 17:48:55 Updated: libuuid-devel-2.23.2-59.el7.x86_64
Dec 08 17:48:55 Updated: ea-php72-php-pdo-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:48:55 Updated: e2fsprogs-libs-1.42.9-13.el7.x86_64
Dec 08 17:48:55 Updated: libcom_err-devel-1.42.9-13.el7.x86_64
Dec 08 17:48:55 Updated: zlib-devel-1.2.7-18.el7.x86_64
Dec 08 17:48:56 Updated: libselinux-utils-2.5-14.1.el7.x86_64
Dec 08 17:48:56 Updated: 2:tar-1.26-35.el7.x86_64
Dec 08 17:48:56 Updated: 2:ethtool-4.8-9.el7.x86_64
Dec 08 17:48:56 Installed: 1:libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 08 17:48:56 Updated: libwayland-server-1.15.0-1.el7.x86_64
Dec 08 17:48:56 Updated: freetype-devel-2.8-12.el7.x86_64
Dec 08 17:48:56 Updated: libblkid-devel-2.23.2-59.el7.x86_64
Dec 08 17:48:56 Updated: ea-apache24-tools-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_cgid-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_mpm_event-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy_http-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy_wstunnel-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-config-runtime-1.0-141.141.1.cpanel.noarch
Dec 08 17:48:56 Updated: ea-apache24-config-1.0-141.141.1.cpanel.noarch
Dec 08 17:48:56 Updated: ea-apache24-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_ssl-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_bwlimited-1.4-46.48.89.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy_fcgi-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_dbd-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_authn_dbm-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Installed: libsemanage-python-2.5-14.el7.x86_64
Dec 08 17:48:56 Updated: harfbuzz-1.7.5-2.el7.x86_64
Dec 08 17:48:56 Updated: libcroco-0.6.12-4.el7.x86_64
Dec 08 17:48:57 Updated: libxcb-devel-1.13-1.el7.x86_64
Dec 08 17:48:57 Updated: grubby-8.28-25.el7.x86_64
Dec 08 17:48:57 Updated: python-perf-3.10.0-957.1.3.el7.x86_64
Dec 08 17:48:57 Updated: libselinux-python-2.5-14.1.el7.x86_64
Dec 08 17:48:57 Updated: python-linux-procfs-0.4.9-4.el7.noarch
Dec 08 17:48:57 Updated: python-urlgrabber-3.10-9.el7.noarch
Dec 08 17:48:57 Installed: audit-libs-python-2.8.4-4.el7.x86_64
Dec 08 17:48:57 Installed: python-IPy-0.75-6.el7.noarch
Dec 08 17:48:57 Installed: python-ply-3.4-11.el7.noarch
Dec 08 17:48:57 Updated: systemtap-sdt-devel-3.3-3.el7.x86_64
Dec 08 17:48:57 Updated: gdb-7.6.1-114.el7.x86_64
Dec 08 17:48:57 Updated: ea-nghttp2-1.32.0-1.1.3.cpanel.x86_64
Dec 08 17:48:57 Updated: jasper-libs-1.900.1-33.el7.x86_64
Dec 08 17:48:58 Updated: libstdc++-devel-4.8.5-36.el7.x86_64
Dec 08 17:48:58 Installed: setools-libs-3.3.8-4.el7.x86_64
Dec 08 17:49:02 Updated: 1:perl-IO-Zlib-1.10-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Locale-Maketext-Simple-0.21-293.el7.noarch
Dec 08 17:49:02 Updated: perl-CPAN-1.9800-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Module-Loaded-0.08-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Object-Accessor-0.42-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Pod-Escapes-1.04-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Module-CoreList-2.76.02-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Package-Constants-0.02-293.el7.noarch
Dec 08 17:49:02 Updated: perl-Time-Piece-1.20.1-293.el7.x86_64
Dec 08 17:49:02 Updated: 1:openssl-1.0.2k-16.el7.x86_64
Dec 08 17:49:02 Updated: logrotate-3.8.6-17.el7.x86_64
Dec 08 17:49:02 Updated: libkadm5-1.15.1-34.el7.x86_64
Dec 08 17:49:02 Updated: libtirpc-0.2.4-0.15.el7.x86_64
Dec 08 17:49:03 Updated: libss-1.42.9-13.el7.x86_64
Dec 08 17:49:03 Updated: gnutls-3.3.29-8.el7.x86_64
Dec 08 17:49:03 Updated: cpp-4.8.5-36.el7.x86_64
Dec 08 17:49:03 Updated: kmod-libs-20-23.el7.x86_64
Dec 08 17:49:03 Updated: 2:vim-minimal-7.4.160-5.el7.x86_64
Dec 08 17:49:03 Updated: libsepol-devel-2.5-10.el7.x86_64
Dec 08 17:49:03 Updated: libselinux-devel-2.5-14.1.el7.x86_64
Dec 08 17:49:03 Updated: krb5-devel-1.15.1-34.el7.x86_64
Dec 08 17:49:04 Updated: 1:openssl-devel-1.0.2k-16.el7.x86_64
Dec 08 17:49:04 Updated: libssh2-devel-1.4.3-12.el7.x86_64
Dec 08 17:49:04 Updated: ea-libcurl-7.62.0-2.2.2.cpanel.x86_64
Dec 08 17:49:04 Updated: perl-Git-1.8.3.1-20.el7.noarch
Dec 08 17:49:05 Updated: git-1.8.3.1-20.el7.x86_64
Dec 08 17:49:10 Updated: linux-firmware-20180911-69.git85c5d90.el7.noarch
Dec 08 17:49:10 Updated: libgomp-4.8.5-36.el7.x86_64
Dec 08 17:49:10 Installed: fribidi-1.0.2-1.el7.x86_64
Dec 08 17:49:10 Installed: checkpolicy-2.5-8.el7.x86_64
Dec 08 17:49:10 Updated: libfastjson-0.99.4-3.el7.x86_64
Dec 08 17:49:10 Updated: libteam-1.27-5.el7.x86_64
Dec 08 17:49:10 Installed: libsmartcols-2.23.2-59.el7.x86_64
Dec 08 17:49:10 Updated: 7:device-mapper-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:10 Updated: kpartx-0.4.9-123.el7.x86_64
Dec 08 17:49:10 Updated: procps-ng-3.3.10-23.el7.x86_64
Dec 08 17:49:10 Updated: util-linux-2.23.2-59.el7.x86_64
Dec 08 17:49:10 Updated: 7:device-mapper-libs-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:10 Updated: cryptsetup-libs-2.0.3-3.el7.x86_64
Dec 08 17:49:10 Updated: dracut-033-554.el7.x86_64
Dec 08 17:49:10 Updated: kmod-20-23.el7.x86_64
Dec 08 17:49:11 Updated: elfutils-libs-0.172-2.el7.x86_64
Dec 08 17:49:11 Updated: systemd-libs-219-62.el7.x86_64
Dec 08 17:49:11 Updated: 1:dbus-libs-1.10.24-12.el7.x86_64
Dec 08 17:49:12 Updated: systemd-219-62.el7.x86_64
Dec 08 17:49:12 Updated: 1:dbus-1.10.24-12.el7.x86_64
Dec 08 17:49:12 Updated: elfutils-default-yama-scope-0.172-2.el7.noarch
Dec 08 17:49:12 Updated: systemd-sysv-219-62.el7.x86_64
Dec 08 17:49:12 Updated: initscripts-9.49.46-1.el7.x86_64
Dec 08 17:49:12 Installed: 1:grub2-tools-minimal-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:12 Updated: libdrm-2.4.91-3.el7.x86_64
Dec 08 17:49:12 Updated: 7:device-mapper-event-libs-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:12 Updated: cronie-anacron-1.4.11-20.el7_6.x86_64
Dec 08 17:49:12 Updated: cronie-1.4.11-20.el7_6.x86_64
Dec 08 17:49:12 Updated: 7:device-mapper-event-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:13 Updated: 7:lvm2-libs-2.02.180-10.el7_6.2.x86_64
Dec 08 17:49:13 Updated: mesa-libgbm-18.0.5-3.el7.x86_64
Dec 08 17:49:13 Updated: polkit-0.112-18.el7.x86_64
Dec 08 17:49:13 Installed: libcgroup-0.41-20.el7.x86_64
Dec 08 17:49:13 Updated: elfutils-0.172-2.el7.x86_64
Dec 08 17:49:13 Updated: dracut-network-033-554.el7.x86_64
Dec 08 17:49:13 Updated: ipset-libs-6.38-3.el7_6.x86_64
Dec 08 17:49:13 Updated: libwayland-client-1.15.0-1.el7.x86_64
Dec 08 17:49:13 Updated: nss-pem-1.0.3-5.el7.x86_64
Dec 08 17:49:13 Updated: libcurl-7.29.0-51.el7.x86_64
Dec 08 17:49:13 Updated: curl-7.29.0-51.el7.x86_64
Dec 08 17:49:13 Updated: rpm-libs-4.11.3-35.el7.x86_64
Dec 08 17:49:13 Updated: rpm-4.11.3-35.el7.x86_64
Dec 08 17:49:13 Updated: openldap-2.4.44-20.el7.x86_64
Dec 08 17:49:13 Updated: policycoreutils-2.5-29.el7.x86_64
Dec 08 17:49:14 Installed: 1:grub2-tools-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:14 Installed: 1:grub2-tools-extra-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:14 Updated: selinux-policy-3.13.1-229.el7_6.6.noarch
Dec 08 17:49:14 Updated: redhat-rpm-config-9.1.0-87.el7.centos.noarch
Dec 08 17:49:14 Updated: rpm-build-libs-4.11.3-35.el7.x86_64
Dec 08 17:49:14 Updated: rpm-python-4.11.3-35.el7.x86_64
Dec 08 17:49:14 Updated: yum-plugin-fastestmirror-1.1.31-50.el7.noarch
Dec 08 17:49:14 Updated: yum-3.4.3-161.el7.centos.noarch
Dec 08 17:49:14 Updated: 1:grub2-pc-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:14 Installed: policycoreutils-python-2.5-29.el7.x86_64
Dec 08 17:49:14 Updated: 1:net-snmp-agent-libs-5.7.2-37.el7.x86_64
Dec 08 17:49:14 Updated: 1:net-snmp-5.7.2-37.el7.x86_64
Dec 08 17:49:14 Updated: efivar-libs-36-11.el7.x86_64
Dec 08 17:49:15 Updated: kernel-headers-3.10.0-957.1.3.el7.x86_64
Dec 08 17:49:15 Updated: glibc-headers-2.17-260.el7.x86_64
Dec 08 17:49:15 Updated: glibc-devel-2.17-260.el7.x86_64
Dec 08 17:49:15 Updated: perl-ExtUtils-Install-1.58-293.el7.noarch
Dec 08 17:49:15 Updated: 4:perl-devel-5.16.3-293.el7.x86_64
Dec 08 17:49:15 Updated: perl-ExtUtils-Embed-1.30-293.el7.noarch
Dec 08 17:49:15 Updated: 1:perl-ExtUtils-CBuilder-0.28.2.6-293.el7.noarch
Dec 08 17:49:16 Updated: gcc-4.8.5-36.el7.x86_64
Dec 08 17:49:17 Updated: kbd-misc-1.15.5-15.el7.noarch
Dec 08 17:49:17 Updated: libreport-filesystem-2.1.11-42.el7.centos.x86_64
Dec 08 17:49:17 Updated: xorg-x11-proto-devel-2018.4-1.el7.noarch
Dec 08 17:49:17 Updated: libX11-common-1.6.5-2.el7.noarch
Dec 08 17:49:17 Updated: libX11-1.6.5-2.el7.x86_64
Dec 08 17:49:17 Installed: 1:libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 08 17:49:17 Updated: mesa-libGL-18.0.5-3.el7.x86_64
Dec 08 17:49:17 Updated: gdk-pixbuf2-2.36.12-3.el7.x86_64
Dec 08 17:49:17 Updated: tkinter-2.7.5-76.el7.x86_64
Dec 08 17:49:17 Installed: 1:libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 08 17:49:17 Updated: mesa-libEGL-18.0.5-3.el7.x86_64
Dec 08 17:49:17 Installed: dejavu-fonts-common-2.33-6.el7.noarch
Dec 08 17:49:18 Installed: dejavu-sans-fonts-2.33-6.el7.noarch
Dec 08 17:49:20 Updated: fontconfig-2.13.0-4.3.el7.x86_64
Dec 08 17:49:20 Updated: cairo-1.15.12-3.el7.x86_64
Dec 08 17:49:20 Updated: pango-1.42.4-1.el7.x86_64
Dec 08 17:49:20 Updated: tzdata-java-2018g-1.el7.noarch
Dec 08 17:49:20 Updated: kbd-legacy-1.15.5-15.el7.noarch
Dec 08 17:49:20 Updated: 2:vim-filesystem-7.4.160-5.el7.x86_64
Dec 08 17:49:21 Updated: glibc-2.17-260.el7.i686
Dec 08 17:49:21 Updated: zlib-1.2.7-18.el7.i686
Dec 08 17:49:21 Updated: libgcc-4.8.5-36.el7.i686
Dec 08 17:49:22 Updated: 2:vim-common-7.4.160-5.el7.x86_64
Dec 08 17:49:22 Updated: 2:vim-enhanced-7.4.160-5.el7.x86_64
Dec 08 17:49:22 Updated: kbd-1.15.5-15.el7.x86_64
Dec 08 17:49:25 Updated: 1:java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64
Dec 08 17:49:25 Updated: librsvg2-2.40.20-1.el7.x86_64
Dec 08 17:49:25 Updated: fontconfig-devel-2.13.0-4.3.el7.x86_64
Dec 08 17:49:26 Updated: ghostscript-9.07-31.el7_6.3.x86_64
Dec 08 17:49:26 Updated: python-tools-2.7.5-76.el7.x86_64
Dec 08 17:49:27 Updated: libX11-devel-1.6.5-2.el7.x86_64
Dec 08 17:49:27 Updated: mdadm-4.1-rc1_2.el7.x86_64
Dec 08 17:49:27 Updated: gcc-c++-4.8.5-36.el7.x86_64
Dec 08 17:49:27 Updated: perl-core-5.16.3-293.el7.x86_64
Dec 08 17:49:27 Updated: glibc-static-2.17-260.el7.x86_64
Dec 08 17:49:27 Updated: efibootmgr-17-2.el7.x86_64
Dec 08 17:49:27 Updated: ea-php72-php-snmp-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:49:28 Updated: 32:bind-9.9.4-72.el7.x86_64
Dec 08 17:49:28 Installed: 1:grub2-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:28 Updated: yum-utils-1.1.31-50.el7.noarch
Dec 08 17:49:28 Updated: rpm-build-4.11.3-35.el7.x86_64
Dec 08 17:49:28 Updated: scl-utils-build-20130529-19.el7.x86_64
Dec 08 17:49:35 Updated: selinux-policy-targeted-3.13.1-229.el7_6.6.noarch
Dec 08 17:49:35 Updated: 1:grub2-efi-x64-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:35 Updated: sudo-1.8.23-3.el7.x86_64
Dec 08 17:49:35 Updated: rng-tools-6.3.1-3.el7.x86_64
Dec 08 17:49:35 Updated: ipset-6.38-3.el7_6.x86_64
Dec 08 17:49:35 Updated: kexec-tools-2.0.15-21.el7.x86_64
Dec 08 17:49:35 Updated: tuned-2.10.0-6.el7.noarch
Dec 08 17:49:35 Updated: 7:lvm2-2.02.180-10.el7_6.2.x86_64
Dec 08 17:49:35 Updated: sysstat-10.1.5-17.el7.x86_64
Dec 08 17:49:35 Updated: rdma-core-17.2-3.el7.x86_64
Dec 08 17:49:38 Installed: kernel-3.10.0-957.1.3.el7.x86_64
Dec 08 17:49:38 Updated: dnsmasq-2.76-7.el7.x86_64
Dec 08 17:49:38 Updated: ea-php54-php-fpm-5.4.45-56.56.16.cpanel.x86_64
Dec 08 17:49:38 Updated: rpcbind-0.2.0-47.el7.x86_64
Dec 08 17:49:39 Updated: audit-2.8.4-4.el7.x86_64
Dec 08 17:49:39 Updated: ea-php71-php-fpm-7.1.24-1.1.6.cpanel.x86_64
Dec 08 17:49:39 Updated: rsyslog-8.24.0-34.el7.x86_64
Dec 08 17:49:39 Updated: nscd-2.17-260.el7.x86_64
Dec 08 17:49:40 Updated: hwdata-0.252-9.1.el7.x86_64
Dec 08 17:49:40 Updated: net-tools-2.0-0.24.20131004git.el7.x86_64
Dec 08 17:49:40 Updated: at-3.1.13-24.el7.x86_64
Dec 08 17:49:40 Updated: teamd-1.27-5.el7.x86_64
Dec 08 17:49:40 Updated: 1:NetworkManager-libnm-1.12.0-8.el7_6.x86_64
Dec 08 17:49:40 Updated: libgudev1-219-62.el7.x86_64
Dec 08 17:49:40 Updated: dracut-config-rescue-033-554.el7.x86_64
Dec 08 17:49:40 Updated: lftp-4.4.8-11.el7.x86_64
Dec 08 17:49:40 Updated: e2fsprogs-1.42.9-13.el7.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_authn_socache-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_unique_id-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_suphp-0.7.2-26.26.10.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_expires-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_reqtimeout-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_suexec-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_env-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_remoteip-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_headers-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_asis-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_version-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: libmount-devel-2.23.2-59.el7.x86_64
Dec 08 17:49:43 Updated: e2fsprogs-devel-1.42.9-13.el7.x86_64
Dec 08 17:49:43 Updated: ea-php72-php-odbc-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-php72-php-mysqlnd-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:49:43 Updated: 32:bind-devel-9.9.4-72.el7.x86_64
Dec 08 17:49:43 Updated: 32:bind-utils-9.9.4-72.el7.x86_64
Dec 08 17:49:43 Updated: 1:net-snmp-utils-5.7.2-37.el7.x86_64
Dec 08 17:49:43 Updated: gobject-introspection-1.56.1-1.el7.x86_64
Dec 08 17:49:43 Updated: ea-php55-php-imap-5.5.38-39.39.16.cpanel.x86_64
Dec 08 17:49:43 Updated: GeoIP-devel-1.5.0-13.el7.x86_64
Dec 08 17:49:43 Updated: 32:bind-libs-lite-9.9.4-72.el7.x86_64
Dec 08 17:49:46 Updated: GeoIP-data-1.5.0-13.el7.noarch
Dec 08 17:49:46 Updated: pyOpenSSL-0.13.1-4.el7.x86_64
Dec 08 17:49:46 Updated: python-firewall-0.5.3-5.el7.noarch
Dec 08 17:49:46 Updated: python-devel-2.7.5-76.el7.x86_64
Dec 08 17:49:46 Updated: python2-psutil-2.2.1-4.el7.x86_64
Dec 08 17:49:46 Updated: xfsprogs-4.5.0-18.el7.x86_64
Dec 08 17:49:46 Updated: libjpeg-turbo-devel-1.2.90-6.el7.x86_64
Dec 08 17:49:46 Updated: wget-1.14-18.el7.x86_64
Dec 08 17:49:46 Updated: nvme-cli-1.6-1.el7.x86_64
Dec 08 17:49:46 Updated: 1:xorg-x11-font-utils-7.5-21.el7.x86_64
Dec 08 17:49:47 Updated: libXfont-1.5.4-1.el7.x86_64
Dec 08 17:49:47 Updated: perl-HTTP-Daemon-6.01-8.el7.noarch
Dec 08 17:49:49 Updated: zsh-5.0.2-31.el7.x86_64
Dec 08 17:49:49 Updated: man-db-2.6.3-11.el7.x86_64
Dec 08 17:49:49 Updated: 2:nmap-ncat-6.40-16.el7.x86_64
Dec 08 17:49:49 Updated: iprutils-2.4.16.1-1.el7.x86_64
Dec 08 17:49:49 Updated: lsof-4.87-6.el7.x86_64
Dec 08 17:49:49 Updated: flex-2.5.37-6.el7.x86_64
Dec 08 17:49:49 Updated: strace-4.12-9.el7.x86_64
Dec 08 17:49:49 Updated: scl-utils-20130529-19.el7.x86_64
Dec 08 17:49:49 Updated: iptables-1.4.21-28.el7.x86_64
Dec 08 17:49:49 Updated: bison-3.0.4-2.el7.x86_64
Dec 08 17:49:49 Updated: 1:dmidecode-3.1-2.el7.x86_64
Dec 08 17:49:50 Updated: 1:grub2-efi-x64-modules-2.02-0.76.el7.centos.noarch
Dec 08 17:49:50 Updated: firewalld-filesystem-0.5.3-5.el7.noarch
Dec 08 17:49:50 Updated: 1:gnu-efi-3.0.8-2.el7.x86_64
Dec 08 17:49:50 Updated: elfutils-libelf-0.172-2.el7.i686
Dec 08 17:49:50 Updated: elfutils-libs-0.172-2.el7.i686
Dec 08 17:49:50 Updated: libsepol-2.5-10.el7.i686
Dec 08 17:49:50 Updated: libselinux-2.5-14.1.el7.i686
Dec 08 17:49:50 Updated: systemd-libs-219-62.el7.i686
Dec 08 17:49:50 Updated: rdma-core-17.2-3.el7.i686
Dec 08 17:49:50 Updated: libstdc++-4.8.5-36.el7.i686
Dec 08 17:49:50 Updated: audit-libs-2.8.4-4.el7.i686



note that firewalld, and selinux are always disabled at default server setup

kernel-862-14.JPG (32,642 bytes)
kernel-862-14.JPG (32,642 bytes)
pgreco

pgreco

2019-01-20 15:22

developer   ~0033645

Ok, I see you updated from 862.14.4 to 957.1.3. I don't expect the regression to be between 957 and 957.1.3 but just to be sure, can you check with boot 957?
yoor699

yoor699

2019-01-20 16:00

reporter   ~0033646

see screenshot, we installed 957, and booted on it

issue is still here

kernel-957.JPG (43,622 bytes)
kernel-957.JPG (43,622 bytes)
yoor699

yoor699

2019-01-27 04:39

reporter   ~0033699

Any news ?

Soon, i will have a virgin nvme server available, i will reset it with steps to observe when issue appears and be sure it is linked to nvme technology use with centos :
- install centos 7
- install cpanel
vojtech.kurka

vojtech.kurka

2019-01-31 20:19

reporter   ~0033747

Hi, we encountered exacty the same behaviour. We have 4 physical machines in two pairs, every pair is identical HW with nvme drives. We upgraded only 1 node in each pair and the updated node shows 99% disk util, while the old kernel shows about 10-20%. The workload is the same. The util jumped up just after the kernel upgrade.

Affected kernel:
  3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Not affected kernel (the old one):
 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

iostat output examples, new kernel (3.10.0-957.1.3):

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 213.00 30.00 1235.00 0.37 32.87 53.81 1.21 0.20 0.10 0.20 0.77 97.40
nvme1n1 0.00 187.00 27.00 1188.00 0.32 32.57 55.45 1.20 0.20 0.22 0.20 0.80 97.20

old kernel (3.10.0-862.14.4):

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 1332.00 210.00 2665.00 3.04 101.46 74.44 0.92 0.35 0.25 0.36 0.05 13.00
nvme1n1 0.00 1298.00 164.00 3100.00 2.33 101.70 65.27 1.23 0.38 0.25 0.39 0.05 15.00

See the "svctm" column and the "%util" column. The NVMe drives are "IntelĀ® SSD DC P3600 Series"
yoor699

yoor699

2019-01-31 21:11

reporter   ~0033748

Glad to hear to hear that, kernel is well the cause.

Something in kernel don't behave properly regarding NVMe SSDs disks, at least a bug somewhere you must find with your team.

Load is not high as you can see, but the bug may cause premature wear of disks, or else that crash server, who knows....
hostimanadmin

hostimanadmin

2019-01-31 23:07

reporter   ~0033749

Hello everyone!

I'm have some server with config like:

- CPU e3-1270v6
- HDD NVME samsung 970 pro 1tb

First time I'm installed centos 7.6 from iso and have same problem with high disk utilization. Replacing the disk with another did not work, the problem continued to reproduce itself.

For next time, I'm reinstall iso from old iso with centos 7.5 and I saw, the problem has gone away. So, after 1-2 hours I'm update OS to 7.6 and kernel to 3.10.0-957.1.3 and after reboot the problem with disk util reappeared.

Then I boot server from prev kernel 3.10.0-862 and problem HAS BEEN FIXED!

TLDR; The problem with disk util is the place to be(!) and is associated with the 3.10.0-957.1.3 version of the kernel!
hostimanadmin

hostimanadmin

2019-01-31 23:07

reporter   ~0033750

p.s. sorry for my bad english
yoor699

yoor699

2019-02-01 04:47

reporter   ~0033751

hostimanadmin,

Such specs is server from OVH, this is exactly where are coming from my servers... Your SSD is Samsung, in my case is 480 SSD ones from Intel...
Issue has nothing to do with HW, at worst a bios setting, but I tried changing few settings, and nothing came up.

Let the professional troubleshoot this kernel thing, it may be simple or complex...
yoor699

yoor699

2019-02-01 07:40

reporter   ~0033752

I wish to correct :
Your SSD is Samsung, in my case is 480 SSD
by
Your SSD is Samsung this is SATA interface, in my case is 480 SSD but with NVME

This means that issue is worst than thought, SSD drives would be all affected by this bug...
hostimanadmin

hostimanadmin

2019-02-01 08:30

reporter   ~0033753

yoor69,

my samsung is not sata disk, it is m.2 nvme 1tb:
https://www.samsung.com/ru/memory-storage/970-pro-nvme-m2-ssd-/MZ-V7P1T0BW/
yoor699

yoor699

2019-02-12 07:45

reporter   ~0033809

Any ETA for this issue ?
arrfab

arrfab

2019-02-12 08:00

administrator   ~0033810

Just a simple reminder : as CentOS only rebuilds the RHEL sources (including kernel), there is no real kernel developer at the CentOS side, so you can't expect an ETA (per definition).
Some people can have a look but the better would be to see if the problem also happens with RHEL, and so also file a bug report against kernel on https://bugzilla.redhat.com .. that's where the problem will be fixed if enough RHEL customers will report the same issue and so ask for a proper fix (or if someone from community identifies the issue and so come with a patch for it)
Have you searched already if RHEL is affected by the same bug/issue ? We have no hardware with NVME disks in the centos infra to just even try to reproduce that behaviour :-(
yoor699

yoor699

2019-02-12 08:27

reporter   ~0033811

Well, i was not expecting such answer, I thought we could get real solutions on your website, and fortunately I follow up with asking any news, or this would end to a death bug report or close.

This is now done :
https://bugzilla.redhat.com/show_bug.cgi?id=1676402
yoor699

yoor699

2019-02-12 09:47

reporter   ~0033812

Just now, we have a free server, we performed a reinstall to see what we can observe...

We observe that issue is not gone, but has strongly reduced :
- issue appear less permanent, we osberve instead random disk utilization spikes

We used both commands to observe :
dstat -cd --disk-util
iostat -x 1

Issue is far from being solved in a convenient manner
yoor699

yoor699

2019-05-05 06:27

reporter   ~0034425

issue is stil lnot solved on
[root@rep9 ~]# uname -a
Linux rep9.hosting1976.fr 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29 14:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

I don't know what they are doing, they have switched the issue to internal :
https://bugzilla.redhat.com/show_bug.cgi?id=1676402

This is really unacceptable for a such major issue
TrevorH

TrevorH

2019-05-05 09:54

manager   ~0034428

All kernel bugs are marked as private by Redhat automatically. It's standard practice. The reporter of the bug has access to it an can add email address to it to allow others to access it too.
vojtech.kurka

vojtech.kurka

2019-05-06 11:17

reporter   ~0034433

I do have access to the upstream bugreport: This bug is already fixed in kernel kernel-3.10.0-1036.el7
You have to wait until kernel-3.10.0-1036.el7 is available in CentOS.
pgreco

pgreco

2019-05-06 13:32

developer   ~0034434

@vojtech.kurka can you add pablo@fliagreco.com.ar to the cc?
also, that kernel won't be available for us until 7.7, unless they backport the fix to z-stream
yoor699

yoor699

2019-06-02 05:43

reporter   ~0034577

This is a disaster, redhat has not fixed yet issue, openvz7 is now contaminated with same issue on last kernel :

http://www.webhostingtalk.com/showthread.php?t=1766652&p=10146560#post10146560
finist

finist

2019-06-13 16:32

reporter   ~0034654

Hi guys, i also don't have access to the RedHat bug.

Can you please post here the the mainstream commits which fix this issue?
Or ask RedHat developers for it in the bug comment?
Or add me to the bug's CC? khorenko at virtuozzo dot com

Thank you!
TrevorH

TrevorH

2019-06-13 16:46

manager   ~0034655

All Redhat bugzilla entries that are kernel related are automatically marked as private. To get access to them you need to have your email address added to the ticket by someone that already has access. That usually means you need to ask the person who initially reported the problem or perhaps someone who has already asked them and been added. No-one from CentOS has access to this bug report unless they have been explicitly added by the original reporter.
yoor699

yoor699

2019-06-13 16:50

reporter   ~0034656

I am the original reporter, and I'm also out of redhat ticket, they simply put private, and prevent anyone from following issue !

this is disgusting
vojtech.kurka

vojtech.kurka

2019-06-13 16:51

reporter   ~0034657

@finist: I'm the original reporter, I just added you to the bug's CC.

Vojtech
yoor699

yoor699

2019-06-13 16:54

reporter   ~0034658

@vojtech.kurka

Are you sure you understand what "finist" talk about ? (cc at redhat bug) and also that the reporter of this bug is me, not you...
vojtech.kurka

vojtech.kurka

2019-06-13 16:55

reporter   ~0034659

To be clear, I'm talking about this upstream bugreport: https://bugzilla.redhat.com/show_bug.cgi?id=1671551

"Fixed In Version: kernel-3.10.0-1036.el7"
yoor699

yoor699

2019-06-13 17:05

reporter   ~0034660

I don't see the relation with this ticket where the issue is dealt by redhat with ID where i opened there the request :
https://bugzilla.redhat.com/show_bug.cgi?id=1676402

Last update is :
I have requested to port the fix to 7.6.z on BZ1669684, so the fix will land on 7.6.z(kernel-3.10.0-957 based kernel)
finally if everything is fine.

see message at link :
https://bugzilla.redhat.com/show_bug.cgi?id=1669684

You mention :
https://bugzilla.redhat.com/show_bug.cgi?id=1671551
and added CC 'finist'

2 questions :
- what is relation with this issue ?
- if you are redhat with special access, can you add me cc on id=1669684 with : admin AT yoorshop.fr
toracat

toracat

2019-06-13 22:11

manager   ~0034661

A test version of the kernel-plus package that has the patch provided in RHBZ #1671551 is now available from:

https://people.centos.org/toracat/kernel/7/plus/bug15723/

Please give it a try. Feedback welcome.
finist

finist

2019-06-14 12:21

reporter   ~0034664

@vojtech.kurka, thank you for adding me to RedHat's bug #1671551.
David Milburn (RedHat dev) was very kind and provided there the patch which should fix the issue,
so i've built an OpenVZ kernel with the patch.

Please, someone who suffer from the problem, check the test kernel, so we can push the fix into the official kernel.
@yoor699, may be you can?

The test kernel link:
http://fe.virtuozzo.com/fbe1cecc91408e1df092e6f38c70d71a/

i've also filed a bug for the issue: https://bugs.openvz.org/browse/OVZ-7105

Thank you.
jmbreuer

jmbreuer

2019-07-03 08:47

reporter   ~0034771

Is the fix being upstreamed / is there a corresponding kernel bug ticket?

The RedHat bug with (presumably) a patch is not public, and I'm seeing the exact same issue on a non-CentOS/RHEL system with a source-built mainline kernel.
finist

finist

2019-07-10 09:57

reporter   ~0034798

> The RedHat bug with (presumably) a patch is not public <skipped>

The patch for RHEL kernel (untested):
https://www.mail-archive.com/devel@openvz.org/msg36709.html

Sorry, don't know the corresponding patch in mainstream, there are differences.
yoor699

yoor699

2019-07-11 20:12

reporter   ~0034809

Issue is still unsolved after months !

we can't rely anymore on redhat, this is simple
finist

finist

2019-07-12 07:35

reporter   ~0034813

> Issue is still unsolved after months !
>
> we can't rely anymore on redhat, this is simple

Come on, this is funny: RedHat handled the bug, it provided the patch, it's included into future RHEL release - the issue is fixed by RedHat!
Why don't they publish a kernel like in case of zero day security? Just because the issue is not that important.
i'm sure RedHat customers who suffer from that issue - could easily get a kpatch-based live patch.

So what is your complain? That you don't pay RedHat, but don't want to rebuild the kernel with provided patch by yourself either?

And people definitely don't suffer - i've built a kernel and requested to test it month ago - no reaction...
dpeca

dpeca

2019-09-01 01:31

reporter   ~0035051

This is not related to CentOS / RedHat only.
The same issue we have on Debian 10, that has kernel 4.19.0-5-amd64
toracat

toracat

2019-09-01 16:24

manager   ~0035052

The fix is now in kernel-3.10.0-1062.el7 which was released in the CentOS 7 CR repo.

From the changelog:
[block] blk-mq: fix IO accounting in case of none io scheduler (Ming Lei)

Issue History

Date Modified Username Field Change
2019-01-20 06:05 yoor699 New Issue
2019-01-20 06:05 yoor699 Tag Attached: centos 7
2019-01-20 11:36 pgreco Note Added: 0033641
2019-01-20 12:03 yoor699 File Added: kernel-862-14.JPG
2019-01-20 12:03 yoor699 Note Added: 0033643
2019-01-20 15:22 pgreco Note Added: 0033645
2019-01-20 16:00 yoor699 File Added: kernel-957.JPG
2019-01-20 16:00 yoor699 Note Added: 0033646
2019-01-27 04:39 yoor699 Note Added: 0033699
2019-01-31 20:19 vojtech.kurka Note Added: 0033747
2019-01-31 21:11 yoor699 Note Added: 0033748
2019-01-31 23:07 hostimanadmin Note Added: 0033749
2019-01-31 23:07 hostimanadmin Note Added: 0033750
2019-02-01 04:47 yoor699 Note Added: 0033751
2019-02-01 07:40 yoor699 Note Added: 0033752
2019-02-01 08:30 hostimanadmin Note Added: 0033753
2019-02-12 07:45 yoor699 Note Added: 0033809
2019-02-12 08:00 arrfab Note Added: 0033810
2019-02-12 08:27 yoor699 Note Added: 0033811
2019-02-12 09:47 yoor699 Note Added: 0033812
2019-05-05 06:27 yoor699 Note Added: 0034425
2019-05-05 09:54 TrevorH Note Added: 0034428
2019-05-06 11:17 vojtech.kurka Note Added: 0034433
2019-05-06 13:32 pgreco Note Added: 0034434
2019-06-02 05:43 yoor699 Note Added: 0034577
2019-06-13 16:32 finist Note Added: 0034654
2019-06-13 16:46 TrevorH Note Added: 0034655
2019-06-13 16:50 yoor699 Note Added: 0034656
2019-06-13 16:51 vojtech.kurka Note Added: 0034657
2019-06-13 16:54 yoor699 Note Added: 0034658
2019-06-13 16:55 vojtech.kurka Note Added: 0034659
2019-06-13 17:05 yoor699 Note Added: 0034660
2019-06-13 22:11 toracat Note Added: 0034661
2019-06-13 22:12 toracat Status new => assigned
2019-06-14 12:21 finist Note Added: 0034664
2019-07-03 08:47 jmbreuer Note Added: 0034771
2019-07-10 09:57 finist Note Added: 0034798
2019-07-11 20:12 yoor699 Note Added: 0034809
2019-07-12 07:35 finist Note Added: 0034813
2019-09-01 01:31 dpeca Note Added: 0035051
2019-09-01 16:24 toracat Note Added: 0035052