View Issue Details

IDProjectCategoryView StatusLast Update
0005488CentOS-6lvm2public2012-06-20 08:04
Reportervinz 
PriorityhighSeverityblockReproducibilityalways
Status newResolutionopen 
PlatformALLOSCentosOS Version6.2
Product Version6.2 
Target VersionFixed in Version 
Summary0005488: LVM hang forever during shutdown/reboot sequence
DescriptionThere is an issue with LVM when used with multipathd and iscsi/iscsid. The shutdown/reboot sequence hangs forever and this can be reproduced by manually executing scripts in /etc/rc6.d
Steps To ReproduceSystem boots with multipathd service started automatically. Then, manually connect to previously registered iscsi multipath LUN by executing /etc/init.d/iscsi start.
Here everything is OK.
Now simulate the shutdown/reboot sequence order like this :
/etc/init.d/multipathd stop
/etc/init.d/iscsi stop
/etc/init.d/iscsid stop

Now every LVM command will hang forever. For example : lvdisplay.

The problem is that by default lvm2-monitor is started/shutdown automatically, and the shutdown happens after stopping the previous services and issues LVM commands => the system hangs on reboot or shutdown
Additional InformationShutting down iscsi before multipathd seems to not trigger the issue.
As a workaround for shutdown/reboot, one can disable lvm2-monitor if not needed. But this will not fix runtime LVM issues.
TagsNo tags attached.

Activities

vinz

vinz

2012-02-03 09:25

reporter   ~0014395

Looking on the internet, it seems all Linux distros have triggered some sort of the issue and it should have been fixed for years.
It mays be related to multipath config, so here is mine :

blacklist {
    devnode "^sda$"
    devnode "^sda1$"
    devnode "^sda2$"
}

devices {
    device {
        vendor "EQLOGIC"
        product "100E-00"
        path_grouping_policy multibus
            getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
              features "1 queue_if_no_path"
              path_checker readsector0
              path_selector "queue-length 0"
              failback immediate
    }
}
cent2012

cent2012

2012-02-17 10:07

reporter   ~0014490

The following workaround fixed the problem for me.
The idea is to stop lvm at shutdown/reboot before iscsi is stoped.
I changed the prefixes (K99) of /etc/rc[06].d/K*lvm2-monitor to K87 this way:

# first delete/deactivate lvm2-monitor
chkconfig lvm2-monitor --del

# patch /etc/init.d/lvm2-monitor
sed -i.original "s/# chkconfig: 12345 02 99/# original was: chkconfig: 12345 02 99\n# patched to avoid shutdown hang with lvm on iscsi lun\n# chkconfig: 12345 02 87/" /etc/init.d/lvm2-monitor

# activate lvm2-monitor again
chkconfig lvm2-monitor --add

# you can see the changes with
diff /etc/init.d/lvm2-monitor{.original,}
# and
ll /etc/rc?.d/*lvm2-monitor

I don't know if there are any problems or side effects with this workaround.
Please report if you know such a problem.

This patch may be overridden if the rpm package lvm2 is updated or reinstalled.
jonesdea

jonesdea

2012-04-02 17:49

reporter   ~0014779

Just noting to confirm the bug and that cent2012's fix does work.
marji

marji

2012-06-20 08:04

reporter   ~0015291

I can confirm this bug is present in Centos 6.2

This is how you can implement the workaround described by user cent2012 above.

1) create a file /etc/chkconfig.d/lvm2-monitor with this content (yes, all lines start with # ):

# This is a workaround for Centos bug http://bugs.centos.org/view.php?id=5488
# The lvm2-monitor needs to shut down before iscsi.
# Changing the exit priority from 99 to 87.
#
# chkconfig: - 02 87


2) run this command:
chkconfig lvm2-monitor resetpriorities

You can verify the exit priorities changed to 87 by running:
find /etc/ -name '*lvm2-monitor'

This should not get overridden when the lvm2 package is updated or reinstalled.
See the OVERRIDE FILES in man 8 chkconfig.

Issue History

Date Modified Username Field Change
2012-02-03 09:04 vinz New Issue
2012-02-03 09:25 vinz Note Added: 0014395
2012-02-17 10:07 cent2012 Note Added: 0014490
2012-04-02 17:49 jonesdea Note Added: 0014779
2012-06-20 08:04 marji Note Added: 0015291