View Issue Details

IDProjectCategoryView StatusLast Update
0015723CentOS-7sysstatpublic2019-02-12 09:47
Reporteryoor699 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0015723: high disk utilization
Descriptionhi,

After last kernel upgrade on several cPanel servers using NVMe SSD, we get a common high disk utilization according to iostat output, confirmed also by dstat -cd --disk-util (in case iostat would have a caveat : https://brooker.co.za/blog/2014/07/04/iostat-pct.html)
It affects both disks.

Issue is from this last kernel
3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

raid1 is fine :
cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0]
      356576192 blocks [2/2] [UU]
      bitmap: 3/3 pages [12KB], 65536KB chunk

md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
      71678912 blocks [2/2] [UU]

unused devices: <none>

Initial investigation :
- hardware is in perfect state in all servers, they use NVMe SSD for ages.
- such high level of disk utilization has no justification in term of running processes from any user, or any scripts, no iowait also at all
- what is reported by linux may be wrong, but this lets us a visible symptom that we can't trust, neither ignoring it util we have a realiable explanation.

we observed that :
- an upgrade centos on a basic server (with mechanic hdd in this case) is not causing this at reboot the following
- an excessive update of file /.readahead, this is not happening on other cpanel server using cloudlinux by ie
(our servers use no hard raid card)

We tried to disable readahead and reboot in case, but no change at all :
systemctl disable systemd-readahead-collect.service
systemctl disable systemd-readahead-replay.service

We contacted cPanel support, investigated deeply with 2 sysadmin, it appears that the answer will only come from centos 7 kernel developer

Thanks for your attention,
John
Steps To Reproduce3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
NVMe SSD
raid1
check output dstat -cd --disk-util
Tagscentos 7
abrt_hash
URL

Activities

pgreco

pgreco

2019-01-20 11:36

developer   ~0033641

The first step would be to determine which was the last kernel this worked as expected, and if it is in fact the kernel, or another program that was updated during the same period.
yoor699

yoor699

2019-01-20 12:03

reporter   ~0033643

Hello
See screenshot, the 862.14 is the last one working properly


Here is the full list from yum.log update new with kernel install (kernel-3.10.0-957.1.3.el7.x86_64), and issue appeared after :

Dec 08 17:48:43 Updated: libgcc-4.8.5-36.el7.x86_64
Dec 08 17:48:43 Updated: 1:grub2-common-2.02-0.76.el7.centos.noarch
Dec 08 17:48:43 Updated: 32:bind-license-9.9.4-72.el7.noarch
Dec 08 17:48:43 Updated: 1:grub2-pc-modules-2.02-0.76.el7.centos.noarch
Dec 08 17:48:43 Updated: tzdata-2018g-1.el7.noarch
Dec 08 17:48:47 Updated: glibc-common-2.17-260.el7.x86_64
Dec 08 17:48:47 Updated: glibc-2.17-260.el7.x86_64
Dec 08 17:48:48 Updated: bash-4.2.46-31.el7.x86_64
Dec 08 17:48:48 Updated: libsepol-2.5-10.el7.x86_64
Dec 08 17:48:48 Updated: libselinux-2.5-14.1.el7.x86_64
Dec 08 17:48:48 Updated: zlib-1.2.7-18.el7.x86_64
Dec 08 17:48:48 Updated: libcom_err-1.42.9-13.el7.x86_64
Dec 08 17:48:48 Updated: krb5-libs-1.15.1-34.el7.x86_64
Dec 08 17:48:48 Updated: 1:openssl-libs-1.0.2k-16.el7.x86_64
Dec 08 17:48:48 Updated: coreutils-8.22-23.el7.x86_64
Dec 08 17:48:49 Updated: 4:perl-macros-5.16.3-293.el7.x86_64
Dec 08 17:48:49 Updated: 4:perl-libs-5.16.3-293.el7.x86_64
Dec 08 17:48:50 Updated: 4:perl-5.16.3-293.el7.x86_64
Dec 08 17:48:51 Updated: ea-openssl-1.0.2q-1.1.1.cpanel.x86_64
Dec 08 17:48:51 Updated: ea-php70-php-common-7.0.32-2.2.7.cpanel.x86_64
Dec 08 17:48:51 Updated: freetype-2.8-12.el7.x86_64
Dec 08 17:48:51 Updated: libstdc++-4.8.5-36.el7.x86_64
Dec 08 17:48:51 Updated: audit-libs-2.8.4-4.el7.x86_64
Dec 08 17:48:51 Updated: libuuid-2.23.2-59.el7.x86_64
Dec 08 17:48:51 Updated: libjpeg-turbo-1.2.90-6.el7.x86_64
Dec 08 17:48:51 Updated: libblkid-2.23.2-59.el7.x86_64
Dec 08 17:48:51 Updated: libssh2-1.4.3-12.el7.x86_64
Dec 08 17:48:51 Updated: elfutils-libelf-0.172-2.el7.x86_64
Dec 08 17:48:51 Updated: ea-libnghttp2-1.32.0-1.1.3.cpanel.x86_64
Dec 08 17:48:52 Updated: python-libs-2.7.5-76.el7.x86_64
Dec 08 17:48:52 Updated: python-2.7.5-76.el7.x86_64
Dec 08 17:48:53 Updated: libicu-50.1.2-17.el7.x86_64
Dec 08 17:48:53 Updated: GeoIP-1.5.0-13.el7.x86_64
Dec 08 17:48:53 Updated: ea-apr-1.6.3-2.2.4.cpanel.x86_64
Dec 08 17:48:53 Updated: ea-php55-libc-client-2007f-13.13.4.cpanel.x86_64
Dec 08 17:48:54 Updated: centos-release-7-6.1810.2.el7.centos.x86_64
Dec 08 17:48:54 Updated: libxcb-1.13-1.el7.x86_64
Dec 08 17:48:54 Updated: libmount-2.23.2-59.el7.x86_64
Dec 08 17:48:54 Updated: glib2-2.56.1-2.el7.x86_64
Dec 08 17:48:54 Updated: libsemanage-2.5-14.el7.x86_64
Dec 08 17:48:54 Updated: 1:net-snmp-libs-5.7.2-37.el7.x86_64
Dec 08 17:48:54 Updated: file-libs-5.11-35.el7.x86_64
Dec 08 17:48:54 Updated: file-5.11-35.el7.x86_64
Dec 08 17:48:54 Updated: 1:findutils-4.5.11-6.el7.x86_64
Dec 08 17:48:54 Updated: ea-apr-util-1.6.1-4.4.4.cpanel.x86_64
Dec 08 17:48:54 Updated: 32:bind-libs-9.9.4-72.el7.x86_64
Dec 08 17:48:55 Updated: binutils-2.27-34.base.el7.x86_64
Dec 08 17:48:55 Updated: mesa-libglapi-18.0.5-3.el7.x86_64
Dec 08 17:48:55 Updated: lm_sensors-libs-3.4.0-6.20160601gitf9185e5.el7.x86_64
Dec 08 17:48:55 Updated: setup-2.8.71-10.el7.noarch
Dec 08 17:48:55 Updated: 2:shadow-utils-4.1.5.1-25.el7.x86_64
Dec 08 17:48:55 Updated: libuuid-devel-2.23.2-59.el7.x86_64
Dec 08 17:48:55 Updated: ea-php72-php-pdo-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:48:55 Updated: e2fsprogs-libs-1.42.9-13.el7.x86_64
Dec 08 17:48:55 Updated: libcom_err-devel-1.42.9-13.el7.x86_64
Dec 08 17:48:55 Updated: zlib-devel-1.2.7-18.el7.x86_64
Dec 08 17:48:56 Updated: libselinux-utils-2.5-14.1.el7.x86_64
Dec 08 17:48:56 Updated: 2:tar-1.26-35.el7.x86_64
Dec 08 17:48:56 Updated: 2:ethtool-4.8-9.el7.x86_64
Dec 08 17:48:56 Installed: 1:libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 08 17:48:56 Updated: libwayland-server-1.15.0-1.el7.x86_64
Dec 08 17:48:56 Updated: freetype-devel-2.8-12.el7.x86_64
Dec 08 17:48:56 Updated: libblkid-devel-2.23.2-59.el7.x86_64
Dec 08 17:48:56 Updated: ea-apache24-tools-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_cgid-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_mpm_event-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy_http-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy_wstunnel-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-config-runtime-1.0-141.141.1.cpanel.noarch
Dec 08 17:48:56 Updated: ea-apache24-config-1.0-141.141.1.cpanel.noarch
Dec 08 17:48:56 Updated: ea-apache24-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_ssl-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_bwlimited-1.4-46.48.89.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_proxy_fcgi-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_dbd-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Updated: ea-apache24-mod_authn_dbm-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:48:56 Installed: libsemanage-python-2.5-14.el7.x86_64
Dec 08 17:48:56 Updated: harfbuzz-1.7.5-2.el7.x86_64
Dec 08 17:48:56 Updated: libcroco-0.6.12-4.el7.x86_64
Dec 08 17:48:57 Updated: libxcb-devel-1.13-1.el7.x86_64
Dec 08 17:48:57 Updated: grubby-8.28-25.el7.x86_64
Dec 08 17:48:57 Updated: python-perf-3.10.0-957.1.3.el7.x86_64
Dec 08 17:48:57 Updated: libselinux-python-2.5-14.1.el7.x86_64
Dec 08 17:48:57 Updated: python-linux-procfs-0.4.9-4.el7.noarch
Dec 08 17:48:57 Updated: python-urlgrabber-3.10-9.el7.noarch
Dec 08 17:48:57 Installed: audit-libs-python-2.8.4-4.el7.x86_64
Dec 08 17:48:57 Installed: python-IPy-0.75-6.el7.noarch
Dec 08 17:48:57 Installed: python-ply-3.4-11.el7.noarch
Dec 08 17:48:57 Updated: systemtap-sdt-devel-3.3-3.el7.x86_64
Dec 08 17:48:57 Updated: gdb-7.6.1-114.el7.x86_64
Dec 08 17:48:57 Updated: ea-nghttp2-1.32.0-1.1.3.cpanel.x86_64
Dec 08 17:48:57 Updated: jasper-libs-1.900.1-33.el7.x86_64
Dec 08 17:48:58 Updated: libstdc++-devel-4.8.5-36.el7.x86_64
Dec 08 17:48:58 Installed: setools-libs-3.3.8-4.el7.x86_64
Dec 08 17:49:02 Updated: 1:perl-IO-Zlib-1.10-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Locale-Maketext-Simple-0.21-293.el7.noarch
Dec 08 17:49:02 Updated: perl-CPAN-1.9800-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Module-Loaded-0.08-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Object-Accessor-0.42-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Pod-Escapes-1.04-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Module-CoreList-2.76.02-293.el7.noarch
Dec 08 17:49:02 Updated: 1:perl-Package-Constants-0.02-293.el7.noarch
Dec 08 17:49:02 Updated: perl-Time-Piece-1.20.1-293.el7.x86_64
Dec 08 17:49:02 Updated: 1:openssl-1.0.2k-16.el7.x86_64
Dec 08 17:49:02 Updated: logrotate-3.8.6-17.el7.x86_64
Dec 08 17:49:02 Updated: libkadm5-1.15.1-34.el7.x86_64
Dec 08 17:49:02 Updated: libtirpc-0.2.4-0.15.el7.x86_64
Dec 08 17:49:03 Updated: libss-1.42.9-13.el7.x86_64
Dec 08 17:49:03 Updated: gnutls-3.3.29-8.el7.x86_64
Dec 08 17:49:03 Updated: cpp-4.8.5-36.el7.x86_64
Dec 08 17:49:03 Updated: kmod-libs-20-23.el7.x86_64
Dec 08 17:49:03 Updated: 2:vim-minimal-7.4.160-5.el7.x86_64
Dec 08 17:49:03 Updated: libsepol-devel-2.5-10.el7.x86_64
Dec 08 17:49:03 Updated: libselinux-devel-2.5-14.1.el7.x86_64
Dec 08 17:49:03 Updated: krb5-devel-1.15.1-34.el7.x86_64
Dec 08 17:49:04 Updated: 1:openssl-devel-1.0.2k-16.el7.x86_64
Dec 08 17:49:04 Updated: libssh2-devel-1.4.3-12.el7.x86_64
Dec 08 17:49:04 Updated: ea-libcurl-7.62.0-2.2.2.cpanel.x86_64
Dec 08 17:49:04 Updated: perl-Git-1.8.3.1-20.el7.noarch
Dec 08 17:49:05 Updated: git-1.8.3.1-20.el7.x86_64
Dec 08 17:49:10 Updated: linux-firmware-20180911-69.git85c5d90.el7.noarch
Dec 08 17:49:10 Updated: libgomp-4.8.5-36.el7.x86_64
Dec 08 17:49:10 Installed: fribidi-1.0.2-1.el7.x86_64
Dec 08 17:49:10 Installed: checkpolicy-2.5-8.el7.x86_64
Dec 08 17:49:10 Updated: libfastjson-0.99.4-3.el7.x86_64
Dec 08 17:49:10 Updated: libteam-1.27-5.el7.x86_64
Dec 08 17:49:10 Installed: libsmartcols-2.23.2-59.el7.x86_64
Dec 08 17:49:10 Updated: 7:device-mapper-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:10 Updated: kpartx-0.4.9-123.el7.x86_64
Dec 08 17:49:10 Updated: procps-ng-3.3.10-23.el7.x86_64
Dec 08 17:49:10 Updated: util-linux-2.23.2-59.el7.x86_64
Dec 08 17:49:10 Updated: 7:device-mapper-libs-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:10 Updated: cryptsetup-libs-2.0.3-3.el7.x86_64
Dec 08 17:49:10 Updated: dracut-033-554.el7.x86_64
Dec 08 17:49:10 Updated: kmod-20-23.el7.x86_64
Dec 08 17:49:11 Updated: elfutils-libs-0.172-2.el7.x86_64
Dec 08 17:49:11 Updated: systemd-libs-219-62.el7.x86_64
Dec 08 17:49:11 Updated: 1:dbus-libs-1.10.24-12.el7.x86_64
Dec 08 17:49:12 Updated: systemd-219-62.el7.x86_64
Dec 08 17:49:12 Updated: 1:dbus-1.10.24-12.el7.x86_64
Dec 08 17:49:12 Updated: elfutils-default-yama-scope-0.172-2.el7.noarch
Dec 08 17:49:12 Updated: systemd-sysv-219-62.el7.x86_64
Dec 08 17:49:12 Updated: initscripts-9.49.46-1.el7.x86_64
Dec 08 17:49:12 Installed: 1:grub2-tools-minimal-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:12 Updated: libdrm-2.4.91-3.el7.x86_64
Dec 08 17:49:12 Updated: 7:device-mapper-event-libs-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:12 Updated: cronie-anacron-1.4.11-20.el7_6.x86_64
Dec 08 17:49:12 Updated: cronie-1.4.11-20.el7_6.x86_64
Dec 08 17:49:12 Updated: 7:device-mapper-event-1.02.149-10.el7_6.2.x86_64
Dec 08 17:49:13 Updated: 7:lvm2-libs-2.02.180-10.el7_6.2.x86_64
Dec 08 17:49:13 Updated: mesa-libgbm-18.0.5-3.el7.x86_64
Dec 08 17:49:13 Updated: polkit-0.112-18.el7.x86_64
Dec 08 17:49:13 Installed: libcgroup-0.41-20.el7.x86_64
Dec 08 17:49:13 Updated: elfutils-0.172-2.el7.x86_64
Dec 08 17:49:13 Updated: dracut-network-033-554.el7.x86_64
Dec 08 17:49:13 Updated: ipset-libs-6.38-3.el7_6.x86_64
Dec 08 17:49:13 Updated: libwayland-client-1.15.0-1.el7.x86_64
Dec 08 17:49:13 Updated: nss-pem-1.0.3-5.el7.x86_64
Dec 08 17:49:13 Updated: libcurl-7.29.0-51.el7.x86_64
Dec 08 17:49:13 Updated: curl-7.29.0-51.el7.x86_64
Dec 08 17:49:13 Updated: rpm-libs-4.11.3-35.el7.x86_64
Dec 08 17:49:13 Updated: rpm-4.11.3-35.el7.x86_64
Dec 08 17:49:13 Updated: openldap-2.4.44-20.el7.x86_64
Dec 08 17:49:13 Updated: policycoreutils-2.5-29.el7.x86_64
Dec 08 17:49:14 Installed: 1:grub2-tools-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:14 Installed: 1:grub2-tools-extra-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:14 Updated: selinux-policy-3.13.1-229.el7_6.6.noarch
Dec 08 17:49:14 Updated: redhat-rpm-config-9.1.0-87.el7.centos.noarch
Dec 08 17:49:14 Updated: rpm-build-libs-4.11.3-35.el7.x86_64
Dec 08 17:49:14 Updated: rpm-python-4.11.3-35.el7.x86_64
Dec 08 17:49:14 Updated: yum-plugin-fastestmirror-1.1.31-50.el7.noarch
Dec 08 17:49:14 Updated: yum-3.4.3-161.el7.centos.noarch
Dec 08 17:49:14 Updated: 1:grub2-pc-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:14 Installed: policycoreutils-python-2.5-29.el7.x86_64
Dec 08 17:49:14 Updated: 1:net-snmp-agent-libs-5.7.2-37.el7.x86_64
Dec 08 17:49:14 Updated: 1:net-snmp-5.7.2-37.el7.x86_64
Dec 08 17:49:14 Updated: efivar-libs-36-11.el7.x86_64
Dec 08 17:49:15 Updated: kernel-headers-3.10.0-957.1.3.el7.x86_64
Dec 08 17:49:15 Updated: glibc-headers-2.17-260.el7.x86_64
Dec 08 17:49:15 Updated: glibc-devel-2.17-260.el7.x86_64
Dec 08 17:49:15 Updated: perl-ExtUtils-Install-1.58-293.el7.noarch
Dec 08 17:49:15 Updated: 4:perl-devel-5.16.3-293.el7.x86_64
Dec 08 17:49:15 Updated: perl-ExtUtils-Embed-1.30-293.el7.noarch
Dec 08 17:49:15 Updated: 1:perl-ExtUtils-CBuilder-0.28.2.6-293.el7.noarch
Dec 08 17:49:16 Updated: gcc-4.8.5-36.el7.x86_64
Dec 08 17:49:17 Updated: kbd-misc-1.15.5-15.el7.noarch
Dec 08 17:49:17 Updated: libreport-filesystem-2.1.11-42.el7.centos.x86_64
Dec 08 17:49:17 Updated: xorg-x11-proto-devel-2018.4-1.el7.noarch
Dec 08 17:49:17 Updated: libX11-common-1.6.5-2.el7.noarch
Dec 08 17:49:17 Updated: libX11-1.6.5-2.el7.x86_64
Dec 08 17:49:17 Installed: 1:libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 08 17:49:17 Updated: mesa-libGL-18.0.5-3.el7.x86_64
Dec 08 17:49:17 Updated: gdk-pixbuf2-2.36.12-3.el7.x86_64
Dec 08 17:49:17 Updated: tkinter-2.7.5-76.el7.x86_64
Dec 08 17:49:17 Installed: 1:libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64
Dec 08 17:49:17 Updated: mesa-libEGL-18.0.5-3.el7.x86_64
Dec 08 17:49:17 Installed: dejavu-fonts-common-2.33-6.el7.noarch
Dec 08 17:49:18 Installed: dejavu-sans-fonts-2.33-6.el7.noarch
Dec 08 17:49:20 Updated: fontconfig-2.13.0-4.3.el7.x86_64
Dec 08 17:49:20 Updated: cairo-1.15.12-3.el7.x86_64
Dec 08 17:49:20 Updated: pango-1.42.4-1.el7.x86_64
Dec 08 17:49:20 Updated: tzdata-java-2018g-1.el7.noarch
Dec 08 17:49:20 Updated: kbd-legacy-1.15.5-15.el7.noarch
Dec 08 17:49:20 Updated: 2:vim-filesystem-7.4.160-5.el7.x86_64
Dec 08 17:49:21 Updated: glibc-2.17-260.el7.i686
Dec 08 17:49:21 Updated: zlib-1.2.7-18.el7.i686
Dec 08 17:49:21 Updated: libgcc-4.8.5-36.el7.i686
Dec 08 17:49:22 Updated: 2:vim-common-7.4.160-5.el7.x86_64
Dec 08 17:49:22 Updated: 2:vim-enhanced-7.4.160-5.el7.x86_64
Dec 08 17:49:22 Updated: kbd-1.15.5-15.el7.x86_64
Dec 08 17:49:25 Updated: 1:java-1.8.0-openjdk-headless-1.8.0.191.b12-1.el7_6.x86_64
Dec 08 17:49:25 Updated: librsvg2-2.40.20-1.el7.x86_64
Dec 08 17:49:25 Updated: fontconfig-devel-2.13.0-4.3.el7.x86_64
Dec 08 17:49:26 Updated: ghostscript-9.07-31.el7_6.3.x86_64
Dec 08 17:49:26 Updated: python-tools-2.7.5-76.el7.x86_64
Dec 08 17:49:27 Updated: libX11-devel-1.6.5-2.el7.x86_64
Dec 08 17:49:27 Updated: mdadm-4.1-rc1_2.el7.x86_64
Dec 08 17:49:27 Updated: gcc-c++-4.8.5-36.el7.x86_64
Dec 08 17:49:27 Updated: perl-core-5.16.3-293.el7.x86_64
Dec 08 17:49:27 Updated: glibc-static-2.17-260.el7.x86_64
Dec 08 17:49:27 Updated: efibootmgr-17-2.el7.x86_64
Dec 08 17:49:27 Updated: ea-php72-php-snmp-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:49:28 Updated: 32:bind-9.9.4-72.el7.x86_64
Dec 08 17:49:28 Installed: 1:grub2-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:28 Updated: yum-utils-1.1.31-50.el7.noarch
Dec 08 17:49:28 Updated: rpm-build-4.11.3-35.el7.x86_64
Dec 08 17:49:28 Updated: scl-utils-build-20130529-19.el7.x86_64
Dec 08 17:49:35 Updated: selinux-policy-targeted-3.13.1-229.el7_6.6.noarch
Dec 08 17:49:35 Updated: 1:grub2-efi-x64-2.02-0.76.el7.centos.x86_64
Dec 08 17:49:35 Updated: sudo-1.8.23-3.el7.x86_64
Dec 08 17:49:35 Updated: rng-tools-6.3.1-3.el7.x86_64
Dec 08 17:49:35 Updated: ipset-6.38-3.el7_6.x86_64
Dec 08 17:49:35 Updated: kexec-tools-2.0.15-21.el7.x86_64
Dec 08 17:49:35 Updated: tuned-2.10.0-6.el7.noarch
Dec 08 17:49:35 Updated: 7:lvm2-2.02.180-10.el7_6.2.x86_64
Dec 08 17:49:35 Updated: sysstat-10.1.5-17.el7.x86_64
Dec 08 17:49:35 Updated: rdma-core-17.2-3.el7.x86_64
Dec 08 17:49:38 Installed: kernel-3.10.0-957.1.3.el7.x86_64
Dec 08 17:49:38 Updated: dnsmasq-2.76-7.el7.x86_64
Dec 08 17:49:38 Updated: ea-php54-php-fpm-5.4.45-56.56.16.cpanel.x86_64
Dec 08 17:49:38 Updated: rpcbind-0.2.0-47.el7.x86_64
Dec 08 17:49:39 Updated: audit-2.8.4-4.el7.x86_64
Dec 08 17:49:39 Updated: ea-php71-php-fpm-7.1.24-1.1.6.cpanel.x86_64
Dec 08 17:49:39 Updated: rsyslog-8.24.0-34.el7.x86_64
Dec 08 17:49:39 Updated: nscd-2.17-260.el7.x86_64
Dec 08 17:49:40 Updated: hwdata-0.252-9.1.el7.x86_64
Dec 08 17:49:40 Updated: net-tools-2.0-0.24.20131004git.el7.x86_64
Dec 08 17:49:40 Updated: at-3.1.13-24.el7.x86_64
Dec 08 17:49:40 Updated: teamd-1.27-5.el7.x86_64
Dec 08 17:49:40 Updated: 1:NetworkManager-libnm-1.12.0-8.el7_6.x86_64
Dec 08 17:49:40 Updated: libgudev1-219-62.el7.x86_64
Dec 08 17:49:40 Updated: dracut-config-rescue-033-554.el7.x86_64
Dec 08 17:49:40 Updated: lftp-4.4.8-11.el7.x86_64
Dec 08 17:49:40 Updated: e2fsprogs-1.42.9-13.el7.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_authn_socache-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_unique_id-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_suphp-0.7.2-26.26.10.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_expires-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_reqtimeout-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_suexec-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_env-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_remoteip-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_headers-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_asis-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-apache24-mod_version-2.4.37-1.1.2.cpanel.x86_64
Dec 08 17:49:43 Updated: libmount-devel-2.23.2-59.el7.x86_64
Dec 08 17:49:43 Updated: e2fsprogs-devel-1.42.9-13.el7.x86_64
Dec 08 17:49:43 Updated: ea-php72-php-odbc-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:49:43 Updated: ea-php72-php-mysqlnd-7.2.12-1.1.6.cpanel.x86_64
Dec 08 17:49:43 Updated: 32:bind-devel-9.9.4-72.el7.x86_64
Dec 08 17:49:43 Updated: 32:bind-utils-9.9.4-72.el7.x86_64
Dec 08 17:49:43 Updated: 1:net-snmp-utils-5.7.2-37.el7.x86_64
Dec 08 17:49:43 Updated: gobject-introspection-1.56.1-1.el7.x86_64
Dec 08 17:49:43 Updated: ea-php55-php-imap-5.5.38-39.39.16.cpanel.x86_64
Dec 08 17:49:43 Updated: GeoIP-devel-1.5.0-13.el7.x86_64
Dec 08 17:49:43 Updated: 32:bind-libs-lite-9.9.4-72.el7.x86_64
Dec 08 17:49:46 Updated: GeoIP-data-1.5.0-13.el7.noarch
Dec 08 17:49:46 Updated: pyOpenSSL-0.13.1-4.el7.x86_64
Dec 08 17:49:46 Updated: python-firewall-0.5.3-5.el7.noarch
Dec 08 17:49:46 Updated: python-devel-2.7.5-76.el7.x86_64
Dec 08 17:49:46 Updated: python2-psutil-2.2.1-4.el7.x86_64
Dec 08 17:49:46 Updated: xfsprogs-4.5.0-18.el7.x86_64
Dec 08 17:49:46 Updated: libjpeg-turbo-devel-1.2.90-6.el7.x86_64
Dec 08 17:49:46 Updated: wget-1.14-18.el7.x86_64
Dec 08 17:49:46 Updated: nvme-cli-1.6-1.el7.x86_64
Dec 08 17:49:46 Updated: 1:xorg-x11-font-utils-7.5-21.el7.x86_64
Dec 08 17:49:47 Updated: libXfont-1.5.4-1.el7.x86_64
Dec 08 17:49:47 Updated: perl-HTTP-Daemon-6.01-8.el7.noarch
Dec 08 17:49:49 Updated: zsh-5.0.2-31.el7.x86_64
Dec 08 17:49:49 Updated: man-db-2.6.3-11.el7.x86_64
Dec 08 17:49:49 Updated: 2:nmap-ncat-6.40-16.el7.x86_64
Dec 08 17:49:49 Updated: iprutils-2.4.16.1-1.el7.x86_64
Dec 08 17:49:49 Updated: lsof-4.87-6.el7.x86_64
Dec 08 17:49:49 Updated: flex-2.5.37-6.el7.x86_64
Dec 08 17:49:49 Updated: strace-4.12-9.el7.x86_64
Dec 08 17:49:49 Updated: scl-utils-20130529-19.el7.x86_64
Dec 08 17:49:49 Updated: iptables-1.4.21-28.el7.x86_64
Dec 08 17:49:49 Updated: bison-3.0.4-2.el7.x86_64
Dec 08 17:49:49 Updated: 1:dmidecode-3.1-2.el7.x86_64
Dec 08 17:49:50 Updated: 1:grub2-efi-x64-modules-2.02-0.76.el7.centos.noarch
Dec 08 17:49:50 Updated: firewalld-filesystem-0.5.3-5.el7.noarch
Dec 08 17:49:50 Updated: 1:gnu-efi-3.0.8-2.el7.x86_64
Dec 08 17:49:50 Updated: elfutils-libelf-0.172-2.el7.i686
Dec 08 17:49:50 Updated: elfutils-libs-0.172-2.el7.i686
Dec 08 17:49:50 Updated: libsepol-2.5-10.el7.i686
Dec 08 17:49:50 Updated: libselinux-2.5-14.1.el7.i686
Dec 08 17:49:50 Updated: systemd-libs-219-62.el7.i686
Dec 08 17:49:50 Updated: rdma-core-17.2-3.el7.i686
Dec 08 17:49:50 Updated: libstdc++-4.8.5-36.el7.i686
Dec 08 17:49:50 Updated: audit-libs-2.8.4-4.el7.i686



note that firewalld, and selinux are always disabled at default server setup

kernel-862-14.JPG (32,642 bytes)
kernel-862-14.JPG (32,642 bytes)
pgreco

pgreco

2019-01-20 15:22

developer   ~0033645

Ok, I see you updated from 862.14.4 to 957.1.3. I don't expect the regression to be between 957 and 957.1.3 but just to be sure, can you check with boot 957?
yoor699

yoor699

2019-01-20 16:00

reporter   ~0033646

see screenshot, we installed 957, and booted on it

issue is still here

kernel-957.JPG (43,622 bytes)
kernel-957.JPG (43,622 bytes)
yoor699

yoor699

2019-01-27 04:39

reporter   ~0033699

Any news ?

Soon, i will have a virgin nvme server available, i will reset it with steps to observe when issue appears and be sure it is linked to nvme technology use with centos :
- install centos 7
- install cpanel
vojtech.kurka

vojtech.kurka

2019-01-31 20:19

reporter   ~0033747

Hi, we encountered exacty the same behaviour. We have 4 physical machines in two pairs, every pair is identical HW with nvme drives. We upgraded only 1 node in each pair and the updated node shows 99% disk util, while the old kernel shows about 10-20%. The workload is the same. The util jumped up just after the kernel upgrade.

Affected kernel:
  3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Not affected kernel (the old one):
 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

iostat output examples, new kernel (3.10.0-957.1.3):

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 213.00 30.00 1235.00 0.37 32.87 53.81 1.21 0.20 0.10 0.20 0.77 97.40
nvme1n1 0.00 187.00 27.00 1188.00 0.32 32.57 55.45 1.20 0.20 0.22 0.20 0.80 97.20

old kernel (3.10.0-862.14.4):

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
nvme0n1 0.00 1332.00 210.00 2665.00 3.04 101.46 74.44 0.92 0.35 0.25 0.36 0.05 13.00
nvme1n1 0.00 1298.00 164.00 3100.00 2.33 101.70 65.27 1.23 0.38 0.25 0.39 0.05 15.00

See the "svctm" column and the "%util" column. The NVMe drives are "IntelĀ® SSD DC P3600 Series"
yoor699

yoor699

2019-01-31 21:11

reporter   ~0033748

Glad to hear to hear that, kernel is well the cause.

Something in kernel don't behave properly regarding NVMe SSDs disks, at least a bug somewhere you must find with your team.

Load is not high as you can see, but the bug may cause premature wear of disks, or else that crash server, who knows....
hostimanadmin

hostimanadmin

2019-01-31 23:07

reporter   ~0033749

Hello everyone!

I'm have some server with config like:

- CPU e3-1270v6
- HDD NVME samsung 970 pro 1tb

First time I'm installed centos 7.6 from iso and have same problem with high disk utilization. Replacing the disk with another did not work, the problem continued to reproduce itself.

For next time, I'm reinstall iso from old iso with centos 7.5 and I saw, the problem has gone away. So, after 1-2 hours I'm update OS to 7.6 and kernel to 3.10.0-957.1.3 and after reboot the problem with disk util reappeared.

Then I boot server from prev kernel 3.10.0-862 and problem HAS BEEN FIXED!

TLDR; The problem with disk util is the place to be(!) and is associated with the 3.10.0-957.1.3 version of the kernel!
hostimanadmin

hostimanadmin

2019-01-31 23:07

reporter   ~0033750

p.s. sorry for my bad english
yoor699

yoor699

2019-02-01 04:47

reporter   ~0033751

hostimanadmin,

Such specs is server from OVH, this is exactly where are coming from my servers... Your SSD is Samsung, in my case is 480 SSD ones from Intel...
Issue has nothing to do with HW, at worst a bios setting, but I tried changing few settings, and nothing came up.

Let the professional troubleshoot this kernel thing, it may be simple or complex...
yoor699

yoor699

2019-02-01 07:40

reporter   ~0033752

I wish to correct :
Your SSD is Samsung, in my case is 480 SSD
by
Your SSD is Samsung this is SATA interface, in my case is 480 SSD but with NVME

This means that issue is worst than thought, SSD drives would be all affected by this bug...
hostimanadmin

hostimanadmin

2019-02-01 08:30

reporter   ~0033753

yoor69,

my samsung is not sata disk, it is m.2 nvme 1tb:
https://www.samsung.com/ru/memory-storage/970-pro-nvme-m2-ssd-/MZ-V7P1T0BW/
yoor699

yoor699

2019-02-12 07:45

reporter   ~0033809

Any ETA for this issue ?
arrfab

arrfab

2019-02-12 08:00

administrator   ~0033810

Just a simple reminder : as CentOS only rebuilds the RHEL sources (including kernel), there is no real kernel developer at the CentOS side, so you can't expect an ETA (per definition).
Some people can have a look but the better would be to see if the problem also happens with RHEL, and so also file a bug report against kernel on https://bugzilla.redhat.com .. that's where the problem will be fixed if enough RHEL customers will report the same issue and so ask for a proper fix (or if someone from community identifies the issue and so come with a patch for it)
Have you searched already if RHEL is affected by the same bug/issue ? We have no hardware with NVME disks in the centos infra to just even try to reproduce that behaviour :-(
yoor699

yoor699

2019-02-12 08:27

reporter   ~0033811

Well, i was not expecting such answer, I thought we could get real solutions on your website, and fortunately I follow up with asking any news, or this would end to a death bug report or close.

This is now done :
https://bugzilla.redhat.com/show_bug.cgi?id=1676402
yoor699

yoor699

2019-02-12 09:47

reporter   ~0033812

Just now, we have a free server, we performed a reinstall to see what we can observe...

We observe that issue is not gone, but has strongly reduced :
- issue appear less permanent, we osberve instead random disk utilization spikes

We used both commands to observe :
dstat -cd --disk-util
iostat -x 1

Issue is far from being solved in a convenient manner

Issue History

Date Modified Username Field Change
2019-01-20 06:05 yoor699 New Issue
2019-01-20 06:05 yoor699 Tag Attached: centos 7
2019-01-20 11:36 pgreco Note Added: 0033641
2019-01-20 12:03 yoor699 File Added: kernel-862-14.JPG
2019-01-20 12:03 yoor699 Note Added: 0033643
2019-01-20 15:22 pgreco Note Added: 0033645
2019-01-20 16:00 yoor699 File Added: kernel-957.JPG
2019-01-20 16:00 yoor699 Note Added: 0033646
2019-01-27 04:39 yoor699 Note Added: 0033699
2019-01-31 20:19 vojtech.kurka Note Added: 0033747
2019-01-31 21:11 yoor699 Note Added: 0033748
2019-01-31 23:07 hostimanadmin Note Added: 0033749
2019-01-31 23:07 hostimanadmin Note Added: 0033750
2019-02-01 04:47 yoor699 Note Added: 0033751
2019-02-01 07:40 yoor699 Note Added: 0033752
2019-02-01 08:30 hostimanadmin Note Added: 0033753
2019-02-12 07:45 yoor699 Note Added: 0033809
2019-02-12 08:00 arrfab Note Added: 0033810
2019-02-12 08:27 yoor699 Note Added: 0033811
2019-02-12 09:47 yoor699 Note Added: 0033812