0017170: I/O completion latency Issue for centos8

ID	Project	Category	View Status	Date Submitted	Last Update

0017170	CentOS-8	kernel	public	2020-03-20 10:32	2020-03-20 12:31

Reporter	ufo19890607
Priority	urgent	Severity	major	Reproducibility	always
Status	new	Resolution	open
Platform	Intel x86	OS	centos	OS Version	CentOS 8
Product Version	8.0.1905
Target Version		Fixed in Version

Summary	0017170: I/O completion latency Issue for centos8
Description	Hi, guys. I found a serious I/O performance issue for centos8 in vhost environment. The host OS is centos7.4(kernel-3.10.0-693), and vhost OS is centos8(4.18.0-80.el8.x86_64). I used Intel NVMe P4510 devices which are passthough to the vhost. I just run fio test on vhost, the fio config is as follow: [global] ioengine=libaio direct=1 thread=0 norandommap=1 randrepeat=0 group_reporting log_avg_msec=10 rw=randwrite iomem_align=4096 bs=4k numjobs=1 iodepth=1 group_reporting=1 time_based runtime=60 size=4000G [testfile1] filename=/dev/nvme0n1 Below is the fio output. testfile1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-44 096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process testfile1: (groupid=0, jobs=1): err= 0: pid=21208: Fri Mar 20 17:59:12 2020 write: IOPS=36.4k, BW=142MiB/s (149MB/s)(8695MiB/61222msec) slat (usec): min=2, max=594, avg= 2.69, stdev= 1.24 clat (nsec): min=890, max=3806.0M, avg=22689.67, stdev=4539022.46 lat (usec): min=9, max=3806.0k, avg=25.84, stdev=4539.03 clat percentiles (usec): \| 60.00th=[ 12], 65.00th=[ 12], 70.00th=[ 12], 75.00th=[ 13], \| 76.00th=[ 13], 77.00th=[ 13], 78.00th=[ 13], 79.00th=[ 13], \| 80.00th=[ 13], 85.00th=[ 13], 90.00th=[ 13], 95.00th=[ 15], \| 96.00th=[ 16], 97.00th=[ 17], 98.00th=[ 18], 99.00th=[ 21], \| 99.50th=[ 23], 99.90th=[ 40], 99.95th=[ 48], 99.99th=[ 118] bw ( KiB/s): min= 3464, max=248896, per=100.00%, avg=193551.73, stdev=64026.. 99, samples=92 iops : min= 866, max=62224, avg=48387.97, stdev=16006.70, samples=92 lat (nsec) : 1000=0.12% lat (usec) : 2=0.08%, 4=0.02%, 10=0.23%, 20=98.52%, 50=0.99% lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01% We can see the max completion latency is 3 secs ????
Steps To Reproduce	I just run fio on vhost. fio version : 3.7
Tags	performance

ufo19890607

2020-03-20 10:32

reporter

fio_output.txt (2,005 bytes)

testfile1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-44
096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process

testfile1: (groupid=0, jobs=1): err= 0: pid=21208: Fri Mar 20 17:59:12 2020
  write: IOPS=36.4k, BW=142MiB/s (149MB/s)(8695MiB/61222msec)
    slat (usec): min=2, max=594, avg= 2.69, stdev= 1.24
    clat (nsec): min=890, max=3806.0M, avg=22689.67, stdev=4539022.46
     lat (usec): min=9, max=3806.0k, avg=25.84, stdev=4539.03
    clat percentiles (usec):
     | 60.00th=[   12], 65.00th=[   12], 70.00th=[   12], 75.00th=[   13],
     | 76.00th=[   13], 77.00th=[   13], 78.00th=[   13], 79.00th=[   13],
     | 80.00th=[   13], 85.00th=[   13], 90.00th=[   13], 95.00th=[   15],
     | 96.00th=[   16], 97.00th=[   17], 98.00th=[   18], 99.00th=[   21],
     | 99.50th=[   23], 99.90th=[   40], 99.95th=[   48], 99.99th=[  118]
   bw (  KiB/s): min= 3464, max=248896, per=100.00%, avg=193551.73, stdev=64026..
99, samples=92
   iops        : min=  866, max=62224, avg=48387.97, stdev=16006.70, samples=92
  lat (nsec)   : 1000=0.12%
  lat (usec)   : 2=0.08%, 4=0.02%, 10=0.23%, 20=98.52%, 50=0.99%
  lat (usec)   : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%
  cpu          : usr=6.06%, sys=22.71%, ctx=2217720, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2225906,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=8695MiB  
(9117MB), run=61222-61222msec

Disk stats (read/write):
  nvme0n1: ios=52/2225905, merge=0/0, ticks=1/44272, in_queue=21898, util=61.61%

fio_output.txt (2,005 bytes)

Date Modified	Username	Field
2020-03-20 10:32	ufo19890607	New Issue
2020-03-20 10:32	ufo19890607	File Added: fio_output.txt
2020-03-20 11:56	kudeyda	Tag Attached: performance
2020-03-20 12:30	kudeyda	Tag Attached: Full bore plumbing Canberra \| Water repairs Canberra \| Gas installation experts Canberra \|
2020-03-20 12:31	kudeyda	Tag Detached: Full bore plumbing Canberra \| Water repairs Canberra \| Gas installation experts Canberra \|

View Issue Details

Activities

Issue History