View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0015844 | CentOS-7 | sysstat | public | 2019-02-19 00:30 | 2019-02-19 00:47 |
Reporter | udosoft | ||||
Priority | normal | Severity | minor | Reproducibility | always |
Status | new | Resolution | open | ||
Product Version | 7.6.1810 | ||||
Target Version | Fixed in Version | ||||
Summary | 0015844: iostat does not show md Raid rebuild IO, iotop same behaviour | ||||
Description | iostat does not show traffic produced by md stat rebuild process (md0 is a mirror of 4x same and added new disk, md127 is a raid5 array) root@nas01 ~]# iostat -m -x Linux 3.10.0-957.5.1.el7.x86_64 (xxxx) 02/19/2019 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.09 0.00 0.06 0.09 0.00 99.76 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.04 0.08 0.69 0.14 0.08 0.00 192.09 0.01 7.11 3.23 26.73 2.56 0.21 sdd 0.58 0.08 0.73 0.14 0.07 0.00 172.49 0.01 6.13 2.11 27.50 1.99 0.17 sdc 0.00 19.73 0.01 0.70 0.00 0.14 413.82 0.06 80.63 1.89 82.03 6.99 0.50 sdb 0.04 0.08 0.11 0.14 0.00 0.00 21.19 0.00 16.41 6.71 24.41 5.16 0.13 md0 0.00 0.00 0.21 0.17 0.00 0.00 26.34 0.00 0.00 0.00 0.00 0.00 0.00 md2 0.00 0.00 0.00 0.00 0.00 0.00 50.96 0.00 0.00 0.00 0.00 0.00 0.00 md1 0.00 0.00 0.00 0.00 0.00 0.00 44.77 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.21 0.00 0.00 0.00 8.25 0.00 0.00 0.00 0.00 0.00 0.00 [root@nas01 ~]# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md127 : active raid5 sdc3[6] sda5[4] sdd5[5] sdb5[2] 1757417472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU] resync=DELAYED bitmap: 3/5 pages [12KB], 65536KB chunk md1 : active raid1 sdc2[3] sdb2[2] 1951744 blocks super 1.2 [2/1] [U_] resync=DELAYED md2 : active raid1 sda2[2] sdd2[0] 1951744 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdc1[6] sdd1[5] sda1[4] sdb1[2] 15616000 blocks super 1.2 [4/3] [U_UU] [============>........] recovery = 61.4% (9602176/15616000) finish=7.1min speed=14051K/sec unused devices: <none> [root@nas01 ~]# iostat -m -x Linux 3.10.0-957.5.1.el7.x86_64 (xxxx) 02/19/2019 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.09 0.00 0.06 0.09 0.00 99.76 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.04 0.08 0.69 0.14 0.08 0.00 192.09 0.01 7.11 3.23 26.73 2.56 0.21 sdd 0.58 0.08 0.73 0.14 0.07 0.00 172.88 0.01 6.11 2.11 27.50 1.99 0.17 sdc 0.00 19.73 0.01 0.71 0.00 0.14 412.94 0.06 80.23 1.89 81.61 6.99 0.50 sdb 0.04 0.08 0.11 0.14 0.00 0.00 21.19 0.00 16.41 6.71 24.41 5.16 0.13 md0 0.00 0.00 0.21 0.17 0.00 0.00 26.34 0.00 0.00 0.00 0.00 0.00 0.00 md2 0.00 0.00 0.00 0.00 0.00 0.00 50.96 0.00 0.00 0.00 0.00 0.00 0.00 md1 0.00 0.00 0.00 0.00 0.00 0.00 44.77 0.00 0.00 0.00 0.00 0.00 0.00 md127 0.00 0.00 0.21 0.00 0.00 0.00 8.25 0.00 0.00 0.00 0.00 0.00 0.00 [root@nas01 ~]# iostat -m Linux 3.10.0-957.5.1.el7.x86_64 (xxxx) 02/19/2019 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.09 0.00 0.06 0.09 0.00 99.76 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.83 0.08 0.00 10373 192 sdd 0.88 0.07 0.00 9927 192 sdc 0.73 0.00 0.15 45 19823 sdb 0.25 0.00 0.00 158 192 md0 0.38 0.00 0.00 472 189 md2 0.00 0.00 0.00 2 0 md1 0.00 0.00 0.00 1 0 md127 0.21 0.00 0.00 113 0 [root@nas01 ~]# ########################## some minutes later... ########################## Personalities : [raid1] [raid6] [raid5] [raid4] md127 : active raid5 sdc3[6] sda5[4] sdd5[5] sdb5[2] 1757417472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU] [>....................] recovery = 0.1% (621568/585805824) finish=749.6min speed=13009K/sec bitmap: 3/5 pages [12KB], 65536KB chunk md1 : active raid1 sdc2[3] sdb2[2] 1951744 blocks super 1.2 [2/1] [U_] resync=DELAYED md2 : active raid1 sda2[2] sdd2[0] 1951744 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdc1[6] sdd1[5] sda1[4] sdb1[2] 15616000 blocks super 1.2 [4/4] [UUUU] unused devices: <none> [root@nas01 ~]# iostat -m Linux 3.10.0-957.5.1.el7.x86_64 (nas01.udohome) 02/19/2019 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.09 0.00 0.06 0.09 0.00 99.76 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.87 0.08 0.00 11006 193 sdd 1.25 0.12 0.00 16227 193 sdc 1.10 0.00 0.19 45 26124 sdb 0.29 0.01 0.00 790 193 md0 0.38 0.00 0.00 472 189 md2 0.00 0.00 0.00 2 0 md1 0.00 0.00 0.00 1 0 md127 0.21 0.00 0.00 113 0 ############################################################# iotop does not show anything either Total DISK READ : 0.00 B/s | Total DISK WRITE : 100.98 K/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 118.46 K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1946 be/3 root 0.00 B/s 100.98 K/s 0.00 % 3.84 % [jbd2/md0-8] 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 3588 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 10 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lru-add-drain] 11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0] 12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1] 13 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1] 14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1] 2063 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:1H] 16 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H] 5137 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % smbd --foreground --no-process-group 18 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kdevtmpfs] 19 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [netns] 20 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khungtaskd] 21 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [writeback] 22 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kintegrityd] 23 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 4120 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [jbd2/md127-8] 25 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 26 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kblockd] 27 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md] 28 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [edac-poller] 29 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdogd] 5150 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lockd] 18977 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md127_resync] 3837 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 36 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kswapd0] 37 be/5 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksmd] 38 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khugepaged] 39 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [crypto] 2055 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rpciod] 47 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthrotld] 2056 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [xprtiod] 50 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kaluad] 4147 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % auditd 4148 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % auditd 4149 be/2 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % audispd 54 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kpsmoused] 3593 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md1_raid1] 56 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ipv6_addrconf] 5177 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd] 1034 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ttm_swap] 5183 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd] 5185 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd] 2059 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % lvmetad -f 69 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [deferwq] 5190 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd] 5191 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % python2 -Es /usr/sbin/tuned -l -P 5192 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd] 5193 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [nfsd] 4174 be/4 rpc 0.00 B/s 0.00 B/s 0.00 % 0.00 % rpcbind -w 4178 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % rngd -f 4181 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % smartd -n -q never 4182 be/4 polkitd 0.00 B/s 0.00 B/s 0.00 % 0.00 % polkitd --no-debug ######################################## Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 3588 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 10 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lru-add-drain] 11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0] 12 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1] 13 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1] 14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1] 2063 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:1H] 16 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H] 5137 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % smbd --foreground --no-process-group 18 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kdevtmpfs] 19 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [netns] 20 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khungtaskd] 21 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [writeback] 22 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kintegrityd] 23 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 4120 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [jbd2/md127-8] 25 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 26 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kblockd] 27 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [md] 28 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [edac-poller] 29 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdogd] 5150 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [lockd] ################################## rebuild is running with | ||||
Steps To Reproduce | create an software raid for example raid 1 put data to it add a new disk (partition) to it with for example mdadmin --manage md0 --add /dev/sdc1 monitor build operation. ckeck iostat | ||||
Additional Information | hp microserver [root@nas01 ~]# lspci 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5) 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5) 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5) 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) 00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05) 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05) 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05) 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02) 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 07:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0) [root@nas01 ~ | ||||
Tags | iostat | ||||
abrt_hash | |||||
URL | |||||
even if i have increased the rebuild rate: [root@nas01 ~]# iostat Linux 3.10.0-957.5.1.el7.x86_64 (nas01.udohome) 02/19/2019 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.09 0.00 0.09 0.09 0.00 99.73 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 2.45 301.69 1.75 41680497 241213 sdd 2.88 341.17 1.75 47135634 241153 sdc 2.69 0.84 414.02 116033 57200637 sdb 1.88 226.05 1.75 31230370 241118 md0 0.45 4.83 1.71 667633 236892 md2 0.00 0.02 0.00 2352 0 md1 0.00 0.01 0.00 1060 0 md127 0.20 0.84 0.00 115857 180 [root@nas01 ~]# cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md127 : active raid5 sdc3[6] sda5[4] sdd5[5] sdb5[2] 1757417472 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/3] [U_UU] [=>...................] recovery = 5.3% (31449460/585805824) finish=157.4min speed=58663K/sec bitmap: 3/5 pages [12KB], 65536KB chunk md1 : active raid1 sdc2[3] sdb2[2] 1951744 blocks super 1.2 [2/1] [U_] resync=DELAYED md2 : active raid1 sda2[2] sdd2[0] 1951744 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdc1[6] sdd1[5] sda1[4] sdb1[2] 15616000 blocks super 1.2 [4/4] [UUUU] ############ iotop: ################## Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 22 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 3588 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [bioset] 5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 7 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] ... |
|