View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0015844CentOS-7sysstatpublic2019-02-19 00:302019-02-19 00:47
Reporterudosoft 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0015844: iostat does not show md Raid rebuild IO, iotop same behaviour
Descriptioniostat does not show traffic produced by md stat rebuild process (md0 is a mirror of 4x same and added new disk, md127 is a raid5 array)

root@nas01 ~]# iostat -m -x
Linux 3.10.0-957.5.1.el7.x86_64 (xxxx) 02/19/2019 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
           0.09 0.00 0.06 0.09 0.00 99.76

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.04 0.08 0.69 0.14 0.08 0.00 192.09 0.01 7.11 3.23 26.73 2.56 0.21
sdd 0.58 0.08 0.73 0.14 0.07 0.00 172.49 0.01 6.13 2.11 27.50 1.99 0.17
sdc 0.00 19.73 0.01 0.70 0.00 0.14 413.82 0.06 80.63 1.89 82.03 6.99 0.50
sdb 0.04 0.08 0.11 0.14 0.00 0.00 21.19 0.00 16.41 6.71 24.41 5.16 0.13
md0 0.00 0.00 0.21 0.17 0.00 0.00 26.34 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 50.96 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 44.77 0.00 0.00 0.00 0.00 0.00 0.00
md127 0.00 0.00 0.21 0.00 0.00 0.00 8.25 0.00 0.00 0.00 0.00 0.00 0.00

[root@nas01 ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md127 : active raid5 sdc3[6] sda5[4] sdd5[5] sdb5[2]
      1757417472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU]
        resync=DELAYED
      bitmap: 3/5 pages [12KB], 65536KB chunk

md1 : active raid1 sdc2[3] sdb2[2]
      1951744 blocks super 1.2 [2/1] [U_]
        resync=DELAYED

md2 : active raid1 sda2[2] sdd2[0]
      1951744 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdc1[6] sdd1[5] sda1[4] sdb1[2]
      15616000 blocks super 1.2 [4/3] [U_UU]
      [============>........] recovery = 61.4% (9602176/15616000) finish=7.1min speed=14051K/sec

unused devices: <none>
[root@nas01 ~]# iostat -m -x
Linux 3.10.0-957.5.1.el7.x86_64 (xxxx) 02/19/2019 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
           0.09 0.00 0.06 0.09 0.00 99.76

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.04 0.08 0.69 0.14 0.08 0.00 192.09 0.01 7.11 3.23 26.73 2.56 0.21
sdd 0.58 0.08 0.73 0.14 0.07 0.00 172.88 0.01 6.11 2.11 27.50 1.99 0.17
sdc 0.00 19.73 0.01 0.71 0.00 0.14 412.94 0.06 80.23 1.89 81.61 6.99 0.50
sdb 0.04 0.08 0.11 0.14 0.00 0.00 21.19 0.00 16.41 6.71 24.41 5.16 0.13
md0 0.00 0.00 0.21 0.17 0.00 0.00 26.34 0.00 0.00 0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 50.96 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 44.77 0.00 0.00 0.00 0.00 0.00 0.00
md127 0.00 0.00 0.21 0.00 0.00 0.00 8.25 0.00 0.00 0.00 0.00 0.00 0.00

[root@nas01 ~]# iostat -m
Linux 3.10.0-957.5.1.el7.x86_64 (xxxx) 02/19/2019 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
           0.09 0.00 0.06 0.09 0.00 99.76

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.83 0.08 0.00 10373 192
sdd 0.88 0.07 0.00 9927 192
sdc 0.73 0.00 0.15 45 19823
sdb 0.25 0.00 0.00 158 192
md0 0.38 0.00 0.00 472 189
md2 0.00 0.00 0.00 2 0
md1 0.00 0.00 0.00 1 0
md127 0.21 0.00 0.00 113 0

[root@nas01 ~]#
##########################
some minutes later...
##########################

Personalities : [raid1] [raid6] [raid5] [raid4]
md127 : active raid5 sdc3[6] sda5[4] sdd5[5] sdb5[2]
      1757417472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU]
      [>....................] recovery = 0.1% (621568/585805824) finish=749.6min speed=13009K/sec
      bitmap: 3/5 pages [12KB], 65536KB chunk

md1 : active raid1 sdc2[3] sdb2[2]
      1951744 blocks super 1.2 [2/1] [U_]
        resync=DELAYED

md2 : active raid1 sda2[2] sdd2[0]
      1951744 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdc1[6] sdd1[5] sda1[4] sdb1[2]
      15616000 blocks super 1.2 [4/4] [UUUU]

unused devices: <none>
[root@nas01 ~]# iostat -m
Linux 3.10.0-957.5.1.el7.x86_64 (nas01.udohome) 02/19/2019 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
           0.09 0.00 0.06 0.09 0.00 99.76

Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.87 0.08 0.00 11006 193
sdd 1.25 0.12 0.00 16227 193
sdc 1.10 0.00 0.19 45 26124
sdb 0.29 0.01 0.00 790 193
md0 0.38 0.00 0.00 472 189
md2 0.00 0.00 0.00 2 0
md1 0.00 0.00 0.00 1 0
md127 0.21 0.00 0.00 113 0


#############################################################
iotop does not show anything either
Total DISK READ : 0.00 B/s | Total DISK WRITE : 100.98 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 118.46 K/s
  TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
 1946 be/3 root 0.00 B/s 100.98 K/s 0.00 % 3.84 % [jbd2/md0-8]
    1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22
    2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
    3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
 3588 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
    5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
    7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
    8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
    9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
   10 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lru-add-drain]
   11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0]
   12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1]
   13 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
   14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1]
 2063 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:1H]
   16 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H]
 5137 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % smbd --foreground --no-process-group
   18 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kdevtmpfs]
   19 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [netns]
   20 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khungtaskd]
   21 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [writeback]
   22 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kintegrityd]
   23 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
 4120 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [jbd2/md127-8]
   25 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
   26 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kblockd]
   27 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md]
   28 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [edac-poller]
   29 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdogd]
 5150 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lockd]
18977 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md127_resync]
 3837 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
   36 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kswapd0]
   37 be/5 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksmd]
   38 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khugepaged]
   39 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [crypto]
 2055 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rpciod]
   47 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthrotld]
 2056 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [xprtiod]
   50 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kaluad]
 4147 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % auditd
 4148 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % auditd
 4149 be/2 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % audispd
   54 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kpsmoused]
 3593 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md1_raid1]
   56 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ipv6_addrconf]
 5177 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd]
 1034 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ttm_swap]
 5183 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd]
 5185 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd]
 2059 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % lvmetad -f
   69 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [deferwq]
 5190 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd]
 5191 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % python2 -Es /usr/sbin/tuned -l -P
 5192 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd]
 5193 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd]
 4174 be/4 rpc 0.00 B/s 0.00 B/s 0.00 % 0.00 % rpcbind -w
 4178 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % rngd -f
 4181 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % smartd -n -q never
 4182 be/4 polkitd 0.00 B/s 0.00 B/s 0.00 % 0.00 % polkitd --no-debug
########################################
Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
  TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
    1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22
    2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
    3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
 3588 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
    5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
    7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
    8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
    9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
   10 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lru-add-drain]
   11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0]
   12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1]
   13 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1]
   14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1]
 2063 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:1H]
   16 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H]
 5137 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % smbd --foreground --no-process-group
   18 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kdevtmpfs]
   19 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [netns]
   20 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khungtaskd]
   21 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [writeback]
   22 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kintegrityd]
   23 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
 4120 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [jbd2/md127-8]
   25 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
   26 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kblockd]
   27 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md]
   28 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [edac-poller]
   29 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdogd]
 5150 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lockd]
##################################
rebuild is running with
Steps To Reproducecreate an software raid for example raid 1
put data to it
add a new disk (partition) to it with for example mdadmin --manage md0 --add /dev/sdc1
monitor build operation. ckeck iostat

Additional Informationhp microserver
[root@nas01 ~]# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
07:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0)
[root@nas01 ~
Tagsiostat
abrt_hash
URL

Activities

udosoft

udosoft

2019-02-19 00:47

reporter   ~0033866

even if i have increased the rebuild rate:

[root@nas01 ~]# iostat
Linux 3.10.0-957.5.1.el7.x86_64 (nas01.udohome) 02/19/2019 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
           0.09 0.00 0.09 0.09 0.00 99.73

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 2.45 301.69 1.75 41680497 241213
sdd 2.88 341.17 1.75 47135634 241153
sdc 2.69 0.84 414.02 116033 57200637
sdb 1.88 226.05 1.75 31230370 241118
md0 0.45 4.83 1.71 667633 236892
md2 0.00 0.02 0.00 2352 0
md1 0.00 0.01 0.00 1060 0
md127 0.20 0.84 0.00 115857 180

[root@nas01 ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md127 : active raid5 sdc3[6] sda5[4] sdd5[5] sdb5[2]
      1757417472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU]
      [=>...................] recovery = 5.3% (31449460/585805824) finish=157.4min speed=58663K/sec
      bitmap: 3/5 pages [12KB], 65536KB chunk

md1 : active raid1 sdc2[3] sdb2[2]
      1951744 blocks super 1.2 [2/1] [U_]
        resync=DELAYED

md2 : active raid1 sda2[2] sdd2[0]
      1951744 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdc1[6] sdd1[5] sda1[4] sdb1[2]
      15616000 blocks super 1.2 [4/4] [UUUU]
############
iotop:
##################

Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
  TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
    1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22
    2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
    3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
 3588 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset]
    5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H]
    7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
    8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh]
    9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
   ...

Issue History

Date Modified Username Field Change
2019-02-19 00:30 udosoft New Issue
2019-02-19 00:30 udosoft Tag Attached: iostat
2019-02-19 00:47 udosoft Note Added: 0033866