View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0018043||CentOS-7||kernel||public||2021-02-01 06:28||2021-02-01 06:28|
|Priority||high||Severity||crash||Reproducibility||have not tried|
|Summary||0018043: LVM unmounts wrong filesystem on full snapshot due to /proc/mounts error with --bind mount|
|Description||An odd problem in the kernel's mount --bind logic is confusing dmeventd and causing the wrong filesystem to be lazy unmounted when an LVM snapshot fills up, potentially crashing the server, depending on the affected filesystem.|
Use case: performing a full system backup, with one filesystem containing active virtual machine images backing up from an LVM snapshot instead of from the active filesystem.
This is done by first creating a readonly bind mount of the root filesystem, readonly bind mounting other non-snapshot filesystems into that tree, and mounting the LVM snapshot into that tree. Thus a full copy of the server's mounted filesystems is presented to the backup software, with the one filesystem drawing from an LVM snapshot instead of from the active filesystem.
The kernel does not properly present the mounted filesystems in /proc/mounts, making it look like the snapshot is also mounted at the location of its active filesystem.
If the snapshot fills up, not only the snapshot is lazy unmounted, but the active filesystem is also lazy unmounted.
This likely results in a system crash, depending on which filesystem got unnecessarily unmounted.
There's also an old report of this same unresolved bug over at: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787555
|Steps To Reproduce||1. Configure a new CentOS 7 server with LVM, leaving space for an LVM snapshot. The server should have at least two filesystems, in this case / and /srv. Here we'll call the volume group "vg0" and the /srv logical volume "srv".|
2. Create a snapshot of /srv: lvcreate -L1G -s -n snap /dev/mapper/vg0-srv
3. Create the directory /mnt/snap for the backup software to access.
4. Readonly bind mount / onto /mnt/snap: mount -o bind,ro / /mnt/snap
5. Mount the snapshot: mount -o ro /dev/mapper/vg0-snap /mnt/snap/srv
6. View the output of /proc/mounts with the 'mount' command, verify that /dev/mapper/vg0-snap appears to be mounted both on /mnt/snap/srv and on /srv:
/dev/mapper/vg0-snap on /mnt/snap/srv type ext4 (ro,relatime,seclabel,data=ordered)
/dev/mapper/vg0-snap on /srv type ext4 (ro,relatime,seclabel,data=ordered)
Note that it appears that mounting a filesystem onto a tree in a --bind mount misbehaves: --bind (unlike --rbind) is not supposed to recurse through submounts, but submounts performed in the original tree or the bound tree, *after* binding, get reflected in the bound tree or original tree, respectively, as far as /proc/mounts is concerned, and in "real life" also when the new submount is done in the original tree.
Note also that this could potentially create a security exposure if the --bind mount was used in the creation of a container.
7. Create enough activity on /srv to fill up the COW backing store for the snapshot.
8. Observe dmeventd doing a lazy unmount of /mnt/snap/srv *and* /srv, based on invalid /proc/mounts information:
Jan 31 00:00:00 server lvm[pid]: Unmounting invalid snapshot vg0-snap from /mnt/snap/srv.
Jan 31 00:00:00 server lvm[pid]: Unmounting invalid snapshot vg0-snap from /srv.
|Additional Information||[root@server ~]# rpm -q centos-release|
[root@server ~]# cat /proc/version
Linux version 3.10.0-1160.11.1.el7.x86_64 (email@example.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Fri Dec 18 16:34:56 UTC 2020
|Tags||No tags attached.|