CentOS Bug Tracker - CentOS-7
View Issue Details
0012725CentOS-7kernelpublic2017-01-26 15:592017-02-11 18:12
PrioritynormalSeveritymajorReproducibilityhave not tried
PlatformOSOS Version
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0012725: /sys/block/<device>/queue/max_sectors_kb can be unexpectedly reset to default value

We have servers with block devices that we have tuned - and have noticed that running "sosreport" can unexpectedly reset all our /sys/block/<device>/queue/max_sectors_kb parameters to some default value.

In conjunction with multipath, if there are I/O requests larger than this value in flight, it can result paths flapping as the kernel tries and fails to commit the request to disk. Get continual error messages like:

Jan 23 09:01:59 oss1 kernel: blk_cloned_rq_check_limits: over max size limit.
Jan 23 09:01:59 oss1 kernel: device-mapper: multipath: Failing path 8:16.
Jan 23 09:01:59 oss1 multipathd: sdb: mark as failed
Jan 23 09:01:59 oss1 multipathd: ost2: remaining active paths: 1

Once max_sectors_kb is increased again, the request succeeds.

Digging into the problem, it's caused by the following command executed by the sosreport "block" plugin across each block device in turn:

  parted -s /dev/<device> unit s print

Getting gdb out, I see it happens when parted gets to line arch/linux.c:1619

(parted-3.1-28.el7.x86_64, kernel-3.10.0-514.6.1.el7.x86_64,sos-3.3-5.el7.centos.noarch)
Steps To Reproduce1. Create a fresh virtual machine and install a copy of centos 7.3.1511 via CentOS-7-x86_64-Minimal-1611.iso

2. Patch the virtual machine. I have verified this with kernel version 3.10.0-514.6.1.el7.x86_64

3. Shutdown virtual machine.

4. Create a block device on the host machine and add it to the virtual machine as a second hard disk, as disk bus "SCSI" (problem does not seem manifest itself on virtio bus).

5. Power on machine

6. On my system, the new disk appeared as /dev/sda. Increase max_sectors_kb as far as it will go:

  # cat /sys/block/sda/queue/max_sectors_kb
  # cat /sys/block/sda/queue/max_hw_sectors_kb
  # cat /sys/block/sda/queue/max_hw_sectors_kb > /sys/block/sda/queue/max_sectors_kb
  # cat /sys/block/sda/queue/max_sectors_kb

7. Run parted:

  # parted -s /dev/sda unit s print

8. max_sectors_kb has magically returned to 512:

  # cat /sys/block/sda/queue/max_sectors_kb
Additional InformationWhatever's going on here, it triggers udev - so a rule of the form in conjunction with a script that puts max_sectors_kb can workaround this issue... but applications sensitive to time can still be upset.

ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="some_label", RUN+="/usr/local/sbin/tune_block %k"
TagsNo tags attached.
Attached Files

There are no notes attached to this issue.

Issue History
2017-01-26 15:59bodgererNew Issue