View Issue Details

IDProjectCategoryView StatusLast Update
0017742CentOS-8kernelpublic2020-09-22 22:58
ReporterGordonW 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version8.1.1911 
Target VersionFixed in Version 
Summary0017742: NFS Reports Stale File Handle after kernel-4.18.0-193.6.3.el8_2.x86_64
DescriptionNFS Reports Stale File Handle after kernel-4.18.0-193.6.3.el8_2.x86_64 but no seen when reboot to
kernel-4.18.0-147.8.1.el8_1.x86_64.

rsync reports:

[root@CentOS-8 ~]# ./Backup_rsync_Stale_Handle_Test
building file list ... done
./
Fedora_26-disk001.vmdk
rsync: write failed on "/mnt/HomeNAS/Linux_Backups/CentOS8_Test/Fedora_26-disk001.vmdk": Stale file handle (116)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
[root@CentOS-8 ~]#

cp reports:

[root@CentOS-8 ~]# cp /root/rsync_test/Fedora_26-disk001.vmdk /mnt/HomeNAS/Linux_Backups/CentOS8_Test/Fedora_26-disk001.vmdk
cp: failed to close '/mnt/HomeNAS/Linux_Backups/CentOS8_Test/Fedora_26-disk001.vmdk': Stale file handle
[root@CentOS-8 ~]#


Steps To ReproduceCreated test file:

Actually made a copy of a vm_disk:

[root@CentOS-8 ~]# ls -lsah ./rsync_test/
total 12G
   0 drwxr-xr-x. 2 root root 36 Sep 20 11:31 .
4.0K dr-xr-x---. 16 root root 4.0K Sep 20 12:11 ..
 12G -rw-------. 1 root root 12G Sep 20 11:32 Fedora_26-disk001.vmdk
[root@CentOS-8 ~]#

use cp command or this test rsync script

[root@CentOS-8 ~]# cat ./Backup_rsync_Stale_Handle_Test
#!/bin/bash
#
Dest="/mnt/HomeNAS/Linux_Backups/CentOS8_Test/"
#
/usr/bin/rsync -aurlv --no-links --stats --delete-before /root/rsync_test/ ${Dest}
[root@CentOS-8 ~]#




Additional InformationNFS Version 3 automounted off Synology NAS system:

[root@CentOS-8 ~]# mount |grep NAS

/etc/auto.synology on /mnt/HomeNAS type autofs (rw,relatime,fd=12,pgrp=1657,timeout=600,minproto=5,maxproto=5,indirect,pipe_ino=35241)

192.168.0.9:/volume1/Linux Backups on /mnt/HomeNAS/Linux_Backups type nfs (rw,relatime,vers=3,rsize=8192,wsize=8192,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.9,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.0.9)
[root@CentOS-8 ~]#


Problem not seem at kernel-4.18.0-147.8.1.el8_1.x86_64.
Problem also seen at kernel-4.18.0-193.19.1.el8_2.x86_64


nfs-utils-1:2.3.3-26.el8.x86_64

Regards,
Gordon
TagsNo tags attached.

Activities

GordonW

GordonW

2020-09-22 22:58

reporter   ~0037724

Performed some additional testing:

Built test system based off 8.2004 distribution, updated to latest kernel, hand mounted NFS server using defaults, ran the following test loop, which completed without any errors:

[root@CentOS-8 ~]# for i in {1..2048}
> do
> dd if=/dev/urandom bs=1M of=/mnt/HomeNAS/Linux_Backups/CentOS8_Test/size_${i} count=${i} 2>&1 |grep -v record
> rm -f /mnt/HomeNAS/Linux_Backups/CentOS8_Test/size_${i}
> done

Ran the same test on my original system and got failures at random file sizes.

153092096 bytes (153 MB, 146 MiB) copied, 2.31904 s, 66.0 MB/s
154140672 bytes (154 MB, 147 MiB) copied, 2.32882 s, 66.2 MB/s
155189248 bytes (155 MB, 148 MiB) copied, 2.30956 s, 67.2 MB/s
156237824 bytes (156 MB, 149 MiB) copied, 2.32989 s, 67.1 MB/s
dd: closing output file '/mnt/HomeNAS/Linux_Backups/CentOS8_Test/size_150': Stale file handle
158334976 bytes (158 MB, 151 MiB) copied, 2.37034 s, 66.8 MB/s
159383552 bytes (159 MB, 152 MiB) copied, 2.36798 s, 67.3 MB/s
160432128 bytes (160 MB, 153 MiB) copied, 2.41636 s, 66.4 MB/s
dd: closing output file '/mnt/HomeNAS/Linux_Backups/CentOS8_Test/size_154': Stale file handle
162529280 bytes (163 MB, 155 MiB) copied, 2.47603 s, 65.6 MB/s
163577856 bytes (164 MB, 156 MiB) copied, 2.44029 s, 67.0 MB/s
164626432 bytes (165 MB, 157 MiB) copied, 2.44239 s, 67.4 MB/s
165675008 bytes (166 MB, 158 MiB) copied, 2.46913 s, 67.1 MB/s
166723584 bytes (167 MB, 159 MiB) copied, 2.48472 s, 67.1 MB/s
167772160 bytes (168 MB, 160 MiB) copied, 2.50203 s, 67.1 MB/s
168820736 bytes (169 MB, 161 MiB) copied, 2.52402 s, 66.9 MB/s
169869312 bytes (170 MB, 162 MiB) copied, 2.52645 s, 67.2 MB/s
170917888 bytes (171 MB, 163 MiB) copied, 2.53589 s, 67.4 MB/s
171966464 bytes (172 MB, 164 MiB) copied, 2.56675 s, 67.0 MB/s
173015040 bytes (173 MB, 165 MiB) copied, 2.56817 s, 67.4 MB/s
174063616 bytes (174 MB, 166 MiB) copied, 2.60168 s, 66.9 MB/s
175112192 bytes (175 MB, 167 MiB) copied, 2.609 s, 67.1 MB/s
176160768 bytes (176 MB, 168 MiB) copied, 2.63066 s, 67.0 MB/s
177209344 bytes (177 MB, 169 MiB) copied, 2.64043 s, 67.1 MB/s
dd: closing output file '/mnt/HomeNAS/Linux_Backups/CentOS8_Test/size_170': Stale file handle
179306496 bytes (179 MB, 171 MiB) copied, 2.68551 s, 66.8 MB/s
180355072 bytes (180 MB, 172 MiB) copied, 2.69241 s, 67.0 MB/s
181403648 bytes (181 MB, 173 MiB) copied, 2.70395 s, 67.1 MB/s
182452224 bytes (182 MB, 174 MiB) copied, 2.75667 s, 66.2 MB/s
183500800 bytes (184 MB, 175 MiB) copied, 2.72837 s, 67.3 MB/s
184549376 bytes (185 MB, 176 MiB) copied, 2.80426 s, 65.8 MB/s
185597952 bytes (186 MB, 177 MiB) copied, 2.77782 s, 66.8 MB/s
186646528 bytes (187 MB, 178 MiB) copied, 2.78408 s, 67.0 MB/s
187695104 bytes (188 MB, 179 MiB) copied, 2.80151 s, 67.0 MB/s
188743680 bytes (189 MB, 180 MiB) copied, 2.80926 s, 67.2 MB/s
189792256 bytes (190 MB, 181 MiB) copied, 2.87398 s, 66.0 MB/s
190840832 bytes (191 MB, 182 MiB) copied, 2.86521 s, 66.6 MB/s
dd: closing output file '/mnt/HomeNAS/Linux_Backups/CentOS8_Test/size_183': Stale file handle
192937984 bytes (193 MB, 184 MiB) copied, 2.90914 s, 66.3 MB/s

Stopped automounter and hand mounted NFS filesystem, again using the default mount option and the problem was not repeatable.

Modified automounter file to remove non-default values and restarted autofs, ran test and observed no errors.

automaount file:
Linux_Backups -rw,credentials=/etc/synology_credentials.txt 192.168.0.9:/volume1/Linux Backups

After a couple of iterations removing variables from the automount file I discovered that removing "wsize=8192" produces a working NFS mount

Linux_Backups -rw,credentials=/etc/synology_credentials.txt,rsize=8192 192.168.0.9:/volume1/Linux Backups


[root@CentOS-8 ~]# mount|grep Backup
192.168.0.9:/volume1/Linux Backups on /mnt/HomeNAS/Linux_Backups type nfs (rw,relatime,vers=3,rsize=8192,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.9,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.0.9)
[root@CentOS-8 ~]#


So it looks like setting a wsize=8192 seems to be a good way to replicate the problem.

Issue History

Date Modified Username Field Change
2020-09-20 18:59 GordonW New Issue
2020-09-22 22:58 GordonW Note Added: 0037724