View Issue Details

IDProjectCategoryView StatusLast Update
0017170CentOS-8kernelpublic2020-03-20 12:31
Reporterufo19890607 
PriorityurgentSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformIntel x86OScentosOS VersionCentOS 8
Product Version8.0.1905 
Target VersionFixed in Version 
Summary0017170: I/O completion latency Issue for centos8
DescriptionHi, guys. I found a serious I/O performance issue for centos8 in vhost environment. The host OS is centos7.4(kernel-3.10.0-693), and vhost OS is centos8(4.18.0-80.el8.x86_64). I used Intel NVMe P4510 devices which are passthough to the vhost. I just run fio test on vhost, the fio config is as follow:

[global]
ioengine=libaio
direct=1
thread=0
norandommap=1
randrepeat=0
group_reporting
log_avg_msec=10
rw=randwrite
iomem_align=4096
bs=4k
numjobs=1
iodepth=1
group_reporting=1
time_based
runtime=60
size=4000G

[testfile1]
filename=/dev/nvme0n1

Below is the fio output.

testfile1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-44
096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process

testfile1: (groupid=0, jobs=1): err= 0: pid=21208: Fri Mar 20 17:59:12 2020
  write: IOPS=36.4k, BW=142MiB/s (149MB/s)(8695MiB/61222msec)
    slat (usec): min=2, max=594, avg= 2.69, stdev= 1.24
    clat (nsec): min=890, max=3806.0M, avg=22689.67, stdev=4539022.46
     lat (usec): min=9, max=3806.0k, avg=25.84, stdev=4539.03
    clat percentiles (usec):
     | 60.00th=[ 12], 65.00th=[ 12], 70.00th=[ 12], 75.00th=[ 13],
     | 76.00th=[ 13], 77.00th=[ 13], 78.00th=[ 13], 79.00th=[ 13],
     | 80.00th=[ 13], 85.00th=[ 13], 90.00th=[ 13], 95.00th=[ 15],
     | 96.00th=[ 16], 97.00th=[ 17], 98.00th=[ 18], 99.00th=[ 21],
     | 99.50th=[ 23], 99.90th=[ 40], 99.95th=[ 48], 99.99th=[ 118]
   bw ( KiB/s): min= 3464, max=248896, per=100.00%, avg=193551.73, stdev=64026..
99, samples=92
   iops : min= 866, max=62224, avg=48387.97, stdev=16006.70, samples=92
  lat (nsec) : 1000=0.12%
  lat (usec) : 2=0.08%, 4=0.02%, 10=0.23%, 20=98.52%, 50=0.99%
  lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%

We can see the max completion latency is 3 secs ????
Steps To ReproduceI just run fio on vhost.
fio version : 3.7
Tagsperformance

Activities

ufo19890607

ufo19890607

2020-03-20 10:32

reporter  

fio_output.txt (2,005 bytes)
testfile1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-44
096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process

testfile1: (groupid=0, jobs=1): err= 0: pid=21208: Fri Mar 20 17:59:12 2020
  write: IOPS=36.4k, BW=142MiB/s (149MB/s)(8695MiB/61222msec)
    slat (usec): min=2, max=594, avg= 2.69, stdev= 1.24
    clat (nsec): min=890, max=3806.0M, avg=22689.67, stdev=4539022.46
     lat (usec): min=9, max=3806.0k, avg=25.84, stdev=4539.03
    clat percentiles (usec):
     | 60.00th=[   12], 65.00th=[   12], 70.00th=[   12], 75.00th=[   13],
     | 76.00th=[   13], 77.00th=[   13], 78.00th=[   13], 79.00th=[   13],
     | 80.00th=[   13], 85.00th=[   13], 90.00th=[   13], 95.00th=[   15],
     | 96.00th=[   16], 97.00th=[   17], 98.00th=[   18], 99.00th=[   21],
     | 99.50th=[   23], 99.90th=[   40], 99.95th=[   48], 99.99th=[  118]
   bw (  KiB/s): min= 3464, max=248896, per=100.00%, avg=193551.73, stdev=64026..
99, samples=92
   iops        : min=  866, max=62224, avg=48387.97, stdev=16006.70, samples=92
  lat (nsec)   : 1000=0.12%
  lat (usec)   : 2=0.08%, 4=0.02%, 10=0.23%, 20=98.52%, 50=0.99%
  lat (usec)   : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%
  cpu          : usr=6.06%, sys=22.71%, ctx=2217720, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2225906,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=8695MiB  
(9117MB), run=61222-61222msec

Disk stats (read/write):
  nvme0n1: ios=52/2225905, merge=0/0, ticks=1/44272, in_queue=21898, util=61.61%
fio_output.txt (2,005 bytes)

Issue History

Date Modified Username Field Change
2020-03-20 10:32 ufo19890607 New Issue
2020-03-20 10:32 ufo19890607 File Added: fio_output.txt
2020-03-20 11:56 kudeyda Tag Attached: performance
2020-03-20 12:30 kudeyda Tag Attached: Full bore plumbing Canberra | Water repairs Canberra | Gas installation experts Canberra |
2020-03-20 12:31 kudeyda Tag Detached: Full bore plumbing Canberra | Water repairs Canberra | Gas installation experts Canberra |