View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0017170 | CentOS-8 | kernel | public | 2020-03-20 10:32 | 2020-03-20 12:31 |
Reporter | ufo19890607 | Assigned To | |||
Priority | urgent | Severity | major | Reproducibility | always |
Status | new | Resolution | open | ||
Platform | Intel x86 | OS | centos | OS Version | CentOS 8 |
Product Version | 8.0.1905 | ||||
Summary | 0017170: I/O completion latency Issue for centos8 | ||||
Description | Hi, guys. I found a serious I/O performance issue for centos8 in vhost environment. The host OS is centos7.4(kernel-3.10.0-693), and vhost OS is centos8(4.18.0-80.el8.x86_64). I used Intel NVMe P4510 devices which are passthough to the vhost. I just run fio test on vhost, the fio config is as follow: [global] ioengine=libaio direct=1 thread=0 norandommap=1 randrepeat=0 group_reporting log_avg_msec=10 rw=randwrite iomem_align=4096 bs=4k numjobs=1 iodepth=1 group_reporting=1 time_based runtime=60 size=4000G [testfile1] filename=/dev/nvme0n1 Below is the fio output. testfile1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-44 096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process testfile1: (groupid=0, jobs=1): err= 0: pid=21208: Fri Mar 20 17:59:12 2020 write: IOPS=36.4k, BW=142MiB/s (149MB/s)(8695MiB/61222msec) slat (usec): min=2, max=594, avg= 2.69, stdev= 1.24 clat (nsec): min=890, max=3806.0M, avg=22689.67, stdev=4539022.46 lat (usec): min=9, max=3806.0k, avg=25.84, stdev=4539.03 clat percentiles (usec): | 60.00th=[ 12], 65.00th=[ 12], 70.00th=[ 12], 75.00th=[ 13], | 76.00th=[ 13], 77.00th=[ 13], 78.00th=[ 13], 79.00th=[ 13], | 80.00th=[ 13], 85.00th=[ 13], 90.00th=[ 13], 95.00th=[ 15], | 96.00th=[ 16], 97.00th=[ 17], 98.00th=[ 18], 99.00th=[ 21], | 99.50th=[ 23], 99.90th=[ 40], 99.95th=[ 48], 99.99th=[ 118] bw ( KiB/s): min= 3464, max=248896, per=100.00%, avg=193551.73, stdev=64026.. 99, samples=92 iops : min= 866, max=62224, avg=48387.97, stdev=16006.70, samples=92 lat (nsec) : 1000=0.12% lat (usec) : 2=0.08%, 4=0.02%, 10=0.23%, 20=98.52%, 50=0.99% lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01% We can see the max completion latency is 3 secs ???? | ||||
Steps To Reproduce | I just run fio on vhost. fio version : 3.7 | ||||
Tags | performance | ||||
fio_output.txt (2,005 bytes)
testfile1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-44 096B, ioengine=libaio, iodepth=1 fio-3.7 Starting 1 process testfile1: (groupid=0, jobs=1): err= 0: pid=21208: Fri Mar 20 17:59:12 2020 write: IOPS=36.4k, BW=142MiB/s (149MB/s)(8695MiB/61222msec) slat (usec): min=2, max=594, avg= 2.69, stdev= 1.24 clat (nsec): min=890, max=3806.0M, avg=22689.67, stdev=4539022.46 lat (usec): min=9, max=3806.0k, avg=25.84, stdev=4539.03 clat percentiles (usec): | 60.00th=[ 12], 65.00th=[ 12], 70.00th=[ 12], 75.00th=[ 13], | 76.00th=[ 13], 77.00th=[ 13], 78.00th=[ 13], 79.00th=[ 13], | 80.00th=[ 13], 85.00th=[ 13], 90.00th=[ 13], 95.00th=[ 15], | 96.00th=[ 16], 97.00th=[ 17], 98.00th=[ 18], 99.00th=[ 21], | 99.50th=[ 23], 99.90th=[ 40], 99.95th=[ 48], 99.99th=[ 118] bw ( KiB/s): min= 3464, max=248896, per=100.00%, avg=193551.73, stdev=64026.. 99, samples=92 iops : min= 866, max=62224, avg=48387.97, stdev=16006.70, samples=92 lat (nsec) : 1000=0.12% lat (usec) : 2=0.08%, 4=0.02%, 10=0.23%, 20=98.52%, 50=0.99% lat (usec) : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01% cpu : usr=6.06%, sys=22.71%, ctx=2217720, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2225906,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=8695MiB (9117MB), run=61222-61222msec Disk stats (read/write): nvme0n1: ios=52/2225905, merge=0/0, ticks=1/44272, in_queue=21898, util=61.61% |
|
Date Modified | Username | Field | Change |
---|---|---|---|
2020-03-20 10:32 | ufo19890607 | New Issue | |
2020-03-20 10:32 | ufo19890607 | File Added: fio_output.txt | |
2020-03-20 11:56 | kudeyda | Tag Attached: performance | |
2020-03-20 12:30 | kudeyda | Tag Attached: Full bore plumbing Canberra | Water repairs Canberra | Gas installation experts Canberra | | |
2020-03-20 12:31 | kudeyda | Tag Detached: Full bore plumbing Canberra | Water repairs Canberra | Gas installation experts Canberra | |