2017-10-21 22:58 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0012986CentOS-7kernelpublic2017-03-27 08:53
Reporterdennisxrow 
PrioritynormalSeverityminorReproducibilityalways
StatusnewResolutionopen 
PlatformOSCentOSOS Version
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0012986: Stacked overlayfs: No such device or address
DescriptionWhen overlayfs is mounted inside a overlayfs, I get
a "no such device or address" error when I try to open a file. The error occured after updating the kernel from Kernel 3.10.0-327.36.3 to 3.10.0-514.6.1.
Steps To Reproduce# mkdir upper lower work merged
# mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/
# cd merged
# mkdir upper2 lower2 work2 merged2
# mount -t overlay overlay -olowerdir=lower2,upperdir=upper2,workdir=work2 merged2/
# cd merged2
# echo hello > test.txt
bash: test.txt: No such device or address
# ls
test.txt
# cat test.txt
bash: test.txt: No such device or address
Additional InformationIt works with Kernel 3.10.0-327

I found a patch that might have something to do with our issue: https://patchwork.kernel.org/patch/9447985/
Tagsfile system
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0028892

kabe (reporter)

I could reproduce this.
Note that overlayfs is still in TECH PREVIEW state, but RH is actively fixing bugs.

Pinning down the problematic part is hard and I haven't suceeded,
but I found a following upstream kernel-4.7 commit:

commit 76bc8e2843b66f8205026365966b49ec6da39ae7
Author: Miklos Szeredi <mszeredi.redhat.com>
Date: Fri Jul 29 12:05:24 2016 +0200

    ovl: disallow overlayfs as upperdir

    This does not work and does not make sense. So instead of fixing it
    (probably not hard) just disallow.

This commit disallows anything overlayfs as upper=, so the
reproducing example disintegrates into

    # mount -t overlay overlay -olowerdir=merged/lower2,upperdir=merged/upper2,workdir=merged/work2 merged/merged2/
    overlayfs: filesystem on 'merged/upper2' not supported as upperdir

So, using overlayfs as upperdir will be unsupported in future kernel.
RHE7's fs/overlayfs/ seems to track kernel-4.x rather than 3.x series, so
RH may or may not 'fix' this by disallowing overlayfs as upper.

~0028893

kabe (reporter)

Using non-overlayfs as upper2 and work2 works as expected:

sudo make test2
[sudo] password for kabe:
rm -fr upper lower work merged
mkdir upper lower work merged
mount -t overlay overlay -olowerdir=lower,upperdir=upper,workdir=work merged/

mkdir upper2 work2
mkdir merged/lower2 merged/merged2
mount -t overlay overlay -olowerdir=merged/lower2,upperdir=upper2,workdir=work2 merged/merged2/

echo hello > merged/merged2/test.txt
ls -li merged/merged2
total 4
536920 -rw-r--r--. 1 root root 6 Mar 21 14:26 test.txt


I know there are some corner usage cases when you can't prepare a
non-overlayfs directory to mount from.

~0028940

kabe (reporter)

Release notes for RHEL-7.3 says:

>> OverlayFS is only supported for use as a Docker graph driver.
>> Its use can only be supported for container COW content,
>> not for persistent storage. Any persistent storage must be placed
>> on non-OverlayFS volumes to be supported.
      ~~~~~~~~~~~~~
>> Only default Docker configuration can be used; that is,
>> one level of overlay, one lowerdir, and both lower and upper levels
   ~~~~~~~~~~~~~~~~~~~~
>> are on the same file system.

I don't think they will extend the support for layered overlayfs
since kernel upstream decided to not.

~0028942

dennisxrow (reporter)

Hi kabe,
thank you very much for all the information!
I already thought that they won't support stacked overlay any further but I couldn't find any proof for that, but the commit you provided (https://github.com/torvalds/linux/commit/76bc8e2843b66f8205026365966b49ec6da39ae7) clearly confirms that.
I think we will have to look for another solution.
+Notes

-Issue History
Date Modified Username Field Change
2017-03-17 16:06 dennisxrow New Issue
2017-03-17 16:06 dennisxrow Tag Attached: file system
2017-03-21 05:33 kabe Note Added: 0028892
2017-03-21 05:40 kabe Note Added: 0028893
2017-03-27 08:38 kabe Note Added: 0028940
2017-03-27 08:53 dennisxrow Note Added: 0028942
+Issue History