View Issue Details

IDProjectCategoryView StatusLast Update
0018083CentOS-7kernelpublic2021-02-24 16:43
Reportersvagner Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.9.2009 
Summary0018083: Incorrect inflight ops in the io_acct for md device
DescriptionWe're reporting incorrect metric for inflight io operations for md devices (at least)

```
# for i in 1 2 ; do echo "###iter ${i}" ; awk '{print $3" "$12}' /proc/diskstats ; mkfs.ext3 /dev/md0 ; sleep 10 ; done
###iter 1
sdd 0
sde 0
sdb 0
sdc 0
sda 0
sda1 0
sda2 0
sr0 0
dm-0 0
dm-1 0
md0 21023
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25376 inodes, 101376 blocks
5068 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1952 inodes per group
Superblock backups stored on blocks:
    8193, 24577, 40961, 57345, 73729

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

###iter 2
sdd 0
sde 0
sdb 0
sdc 0
sda 0
sda1 0
sda2 0
sr0 0
dm-0 0
dm-1 0
md0 23121
```

iostats reports huge `avgqu-sz` and 100% util on the idle md device:

```
# iostat -x 1 3
Linux 3.10.0-1160.15.2.el7.x86_64 (localhost.localdomain) 23/02/21 _x86_64_ (4 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
           9.33 0.00 2.27 0.51 0.00 87.89

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdd 0.00 0.00 0.07 0.00 33.87 0.00 932.23 0.00 2.54 2.54 0.56 0.40 0.00
sde 2.54 135.27 68.69 9.99 2890.57 1198.09 103.92 0.10 1.25 0.80 4.33 0.37 2.89
sdb 0.00 0.89 0.05 0.01 0.47 3.56 122.51 0.00 7.47 0.43 33.74 0.61 0.00
sdc 0.00 0.89 0.00 0.01 0.03 3.56 448.98 0.00 28.96 0.72 33.19 1.41 0.00
sda 0.01 0.20 1.06 0.12 57.68 2.99 102.98 0.00 3.20 3.32 2.13 0.44 0.05
dm-0 0.00 0.00 1.04 0.11 57.19 2.18 103.43 0.00 3.26 3.39 2.09 0.39 0.04
dm-1 0.00 0.00 0.02 0.21 0.11 0.83 8.21 0.00 4.82 0.65 5.24 0.23 0.01
md0 0.00 0.00 0.05 0.90 0.50 3.56 8.54 42.70 0.00 0.00 0.00 1050.95 99.99

avg-cpu: %user %nice %system %iowait %steal %idle
           0.00 0.00 0.00 0.00 0.00 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 25361.75 0.00 0.00 0.00 0.00 100.57

avg-cpu: %user %nice %system %iowait %steal %idle
           0.00 0.00 0.00 0.00 0.00 100.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 25146.95 0.00 0.00 0.00 0.00 99.71
```
Steps To Reproduce1. new vm with 3 disks - /dev/sda /dev/sdb /dev/sdc
2. update kernel to 3.10.0-1160.15.2.el7.x86_64
3. install mdadm
4. mdadm --create /dev/md0 --level=raid1 --raid-devices 2 /dev/sdb /dev/sdc
5. Do some write to md device: mkfs.ext3 /dev/md0
6. check /proc/diskstats && iostats: awk '{print $12}' /proc/diskstats && iostat -x 1 2
Additional InformationProblem is with changes in drivers/md/md.c (func md_make_request). We moved stats accounting to the generic_start_io_acct. But before this change, we didn't account in_flights io tasks (part_inc_in_flight). Because we started to account for it now (do we need it for md???) we should decide where we should call generic_end_io_acct. Or we should revert these changes
TagsNo tags attached.
abrt_hash
URL

Activities

TrevorH

TrevorH

2021-02-24 16:18

manager   ~0038263

CentOS is/was a rebuild of the sources used to produce RHEL and does not fix bugs. You would need to report this to RH via bugzilla.redhat.com and if they decide to fix it and release the fixed package then CentOS will rebuild it.
svagner

svagner

2021-02-24 16:35

reporter   ~0038264

Ok. Good to know. Thanks!
svagner

svagner

2021-02-24 16:43

reporter   ~0038265

Just to leave cross-link here - https://bugzilla.redhat.com/show_bug.cgi?id=1932468

Issue History

Date Modified Username Field Change
2021-02-24 15:46 svagner New Issue
2021-02-24 16:18 TrevorH Note Added: 0038263
2021-02-24 16:35 svagner Note Added: 0038264
2021-02-24 16:43 svagner Note Added: 0038265