2017-04-29 13:33 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0012950CentOS-7nfs-utilspublic2017-03-21 00:46
Reportercentu478 
PrioritynormalSeverityminorReproducibilityalways
StatusnewResolutionopen 
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0012950: problem with the management of the ID map cache
DescriptionCentOS 7.3 NFS server, CentOS 7.3 NFS client. NFS V4. NFS mount works fine. ID mapping works fine. However, if file system is unmounted and then mounted again after 10 minutes, UID and GID shows as 4294967294. Repeating the process, the ID mapping is correct. Repeating the process again, the ID mapping is wrong -- it alternates between correct and wrong.
This happens with kernel 3.10.0-514.6.2.el7.x86_64 and
nfs-utils-1.3.0-0.33.el7_3.x86_64.rpm
Replacing /usr/sbin/nfsidmap with the version from
nfs-utils-1.3.0-0.21.el7.x86_64.rpm
solves the problem.
Steps To Reproduceserver name: server
file system on server: /export/foo
/export/foo on server includes directory /export/foo/bar
/export/foo/bar uid and gid map to the same normal user/group on both client and server

As root, run the "nfsprob" script attached. Output will have correct user/group in first iteration, then 4294967294, then correct again, etc.
TagsNo tags attached.
abrt_hash
URL
Attached Files
  • ? file icon nfsprob (401 bytes) 2017-03-13 00:18 -
    #!/bin/csh -f
    
    #  server name:  server
    #  file system on server:  /export/foo
    #  /export/foo on server includes directory /export/foo/bar
    #  /export/foo/bar uid and gid map to the same normal user/group
                       on both client and server
    
    while (1)
       mount -t nfs4 server:/export/foo /mnt
       echo -n `date +"%d %H:%M:%S"` " "
       ls -ld /mnt/bar
       sleep 300
       umount /mnt
       sleep 310
    end
    
    ? file icon nfsprob (401 bytes) 2017-03-13 00:18 +
  • patch file icon 0001-KEYS-request_key-should-reget-expired-keys-rather-th.patch (4,757 bytes) 2017-03-15 04:31 -
    From 9618e2c8d956d3374feb0b3b4e64bd99c9b6237f Mon Sep 17 00:00:00 2001
    From: "T.kabe" <kabe@>
    Date: Wed, 15 Mar 2017 12:57:49 +0900
    Subject: [PATCH] KEYS: request_key() should reget expired keys rather than
     give EKEYEXPIRED
    
    [upstream commit 0b0a84154eff56913e91df29de5c3a03a0029e38]
    Author: David Howells <dhowells.rredhat.com>
    Date:   Mon Dec 1 22:52:53 2014 +0000
    
    KEYS: request_key() should reget expired keys rather than give EKEYEXPIRED
    
    Since the keyring facility can be viewed as a cache (at least in some
    applications), the local expiration time on the key should probably be viewed
    as a 'needs updating after this time' property rather than an absolute 'anyone
    now wanting to use this object is out of luck' property.
    
    Since request_key() is the main interface for the usage of keys, this should
    update or replace an expired key rather than issuing EKEYEXPIRED if the local
    expiration has been reached (ie. it should refresh the cache).
    
    For absolute conditions where refreshing the cache probably doesn't help, the
    key can be negatively instantiated using KEYCTL_REJECT_KEY with EKEYEXPIRED
    given as the error to issue.  This will still cause request_key() to return
    EKEYEXPIRED as that was explicitly set.
    
    In the future, if the key type has an update op available, we might want to
    upcall with the expired key and allow the upcall to update it.  We would pass
    a different operation name (the first column in /etc/request-key.conf) to the
    request-key program.
    
    request_key() returning EKEYEXPIRED is causing an NFS problem which Chuck
    Lever describes thusly:
    
        After about 10 minutes, my NFSv4 functional tests fail because the
        ownership of the test files goes to "-2". Looking at /proc/keys
        shows that the id_resolv keys that map to my test user ID have
        expired. The ownership problem persists until the expired keys are
        purged from the keyring, and fresh keys are obtained.
    
        I bisected the problem to 3.13 commit b2a4df200d57 ("KEYS: Expand
        the capacity of a keyring"). This commit inadvertantly changes the
        API contract of the internal function keyring_search_aux().
    
        The root cause appears to be that b2a4df200d57 made "no state check"
        the default behavior. "No state check" means the keyring search
        iterator function skips checking the key's expiry timeout, and
        returns expired keys.  request_key_and_link() depends on getting
        an -EAGAIN result code to know when to perform an upcall to refresh
        an expired key.
    
    This patch can be tested directly by:
    
        keyctl request2 user debug:fred a @s
        keyctl timeout %user:debug:fred 3
        sleep 4
        keyctl request2 user debug:fred a @s
    
    Without the patch, the last command gives error EKEYEXPIRED, but with the
    command it gives a new key.
    
    Reported-by: Carl Hetherington <cth.carlh.net>
    Reported-by: Chuck Lever <chuck.lever.oracle.com>
    Signed-off-by: David Howells <dhowells.redhat.com>
    Tested-by: Chuck Lever <chuck.lever.oracle.com>
    ---
     security/keys/internal.h    | 1 +
     security/keys/keyring.c     | 3 ++-
     security/keys/request_key.c | 3 ++-
     3 files changed, 5 insertions(+), 2 deletions(-)
    
    diff --git a/security/keys/internal.h b/security/keys/internal.h
    index 80b2aac..6d5c72e 100644
    --- a/security/keys/internal.h
    +++ b/security/keys/internal.h
    @@ -121,6 +121,7 @@ struct keyring_search_context {
     #define KEYRING_SEARCH_NO_UPDATE_TIME	0x0008	/* Don't update times */
     #define KEYRING_SEARCH_NO_CHECK_PERM	0x0010	/* Don't check permissions */
     #define KEYRING_SEARCH_DETECT_TOO_DEEP	0x0020	/* Give an error on excessive depth */
    +#define KEYRING_SEARCH_SKIP_EXPIRED	0x0040	/* Ignore expired keys (intention to replace) */
     
     	int (*iterator)(const void *object, void *iterator_data);
     
    diff --git a/security/keys/keyring.c b/security/keys/keyring.c
    index 04d0d7c..13496c9 100644
    --- a/security/keys/keyring.c
    +++ b/security/keys/keyring.c
    @@ -526,7 +526,8 @@ static int keyring_search_iterator(const void *object, void *iterator_data)
     		}
     
     		if (key->expiry && ctx->now.tv_sec >= key->expiry) {
    -			ctx->result = ERR_PTR(-EKEYEXPIRED);
    +			if (!(ctx->flags & KEYRING_SEARCH_SKIP_EXPIRED))
    +				ctx->result = ERR_PTR(-EKEYEXPIRED);
     			kleave(" = %d [expire]", ctx->skipped_ret);
     			goto skipped;
     		}
    diff --git a/security/keys/request_key.c b/security/keys/request_key.c
    index 67e413c..7054b36 100644
    --- a/security/keys/request_key.c
    +++ b/security/keys/request_key.c
    @@ -519,7 +519,8 @@ struct key *request_key_and_link(struct key_type *type,
     		.cred			= current_cred(),
     		.match			= type->match,
     		.match_data		= description,
    -		.flags			= KEYRING_SEARCH_LOOKUP_DIRECT,
    +		.flags			= (KEYRING_SEARCH_DO_STATE_CHECK |
    +					   KEYRING_SEARCH_SKIP_EXPIRED),
     	};
     	struct key *key;
     	key_ref_t key_ref;
    -- 
    1.8.3.1
    
    

-Relationships
+Relationships

-Notes

~0028856

kabe (reporter)

I've succeeded in reproducing this.

- On NFS server, "echo Y > /sys/module/nfsd/parameters/nfs4_disable_idmapping" #enable non-default id-mapping
- Edit /etc/idmapd.conf, set Domain=<domain> to same string in NFS server and client

- nfs client: mount -o nfsvers=4 server:/export/foo /mnt
- nfs client: wait until kernel keys shown in
  grep id_ /proc/keys
  go to "expd" (expired) state
  (default 600secs, tunable in /etc/request-key.d/id_resolver.conf: nfsidmap -t 600)
- nfs client: umount /mnt; mount -o nfsvers=4 server:/export/foo /mnt
- nfs client: ls -l /mnt

Actual result:
ls -l /mnt
-rw-r--r--. 1 4294967294 4294967294 4 3? 14 16:34 bar-1001
-rw-r--r--. 1 4294967294 4294967294 4 3? 14 16:35 baz-501

"id_resolver" kernel keys, by default, is
created with 600 seconds lifetime, and after that, becomes
"expd" state, then deleted after 300 seconds.
If you umount/re-mount the NFS share during this 300 second window,
symptom (uid=(uid_t)-2) appears.

This is regression, or proper bugfix of,
  nfsidmap not setting key timeouts
  https://bugzilla.redhat.com/show_bug.cgi?id=1161222
Using old nfs-utils-1.3.0-0.21, nfsidmap creates
nonexpiring "permanent" (perm) kernel keys, which masks the problem.
(You won't be able to change the mapping until reboot, though)

For unknown reason, the expired keys aren't refreshed properly.


Workaround:
On nfs client, invoke "systemctl start nfs" (yes, nfs server)
For unknown reason, this makes "id_legacy" keys linger for 600 seconds,
taking over the data of "id_resolver" keys.

~0028857

kabe (reporter)

Correction:
- #enable non-default id-mapping
  On NFS server, "echo N > /sys/module/nfsd/parameters/nfs4_disable_idmapping"

~0028858

kabe (reporter)

I've found an kernel.org upstream commit which fixes this.
https://patchwork.kernel.org/patch/5336901/
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=0b0a84154eff56913e91df29de5c3a03a0029e38

After applying this patch, the id_resolver kernel key is properly refreshed
and thus NFSv4 id-mapping works as expected, across umount/mount.
Bonus: id-mapping is properly updated after 600seconds timeout
if you changed the mapping.

Note that user still can't write to the file despite uid# seems same;
this is limitation of NFS RPC.

This is the patch revised for CentOS kernel-3.10.0-514:
https://bugs.centos.org/file_download.php?file_id=19315&type=bug

I am going to file kernel-bug on Red Hat Bugzilla, but don't expect much
since enabling id-mapping seems a non-standard usage.
(RH docs implicity says to use LDAP to sync uid#)

~0028859

kabe (reporter)

Posted:
https://bugzilla.redhat.com/show_bug.cgi?id=1432313

~0028860

tru (administrator)

thanks for your work, really appreciated :)
RH has ack'ed and we can expect a fix from them.

~0028891

kabe (reporter)

RHBZ says it will be fixed in kernel-3.10.0-616.el7 .
+Notes

-Issue History
Date Modified Username Field Change
2017-03-13 00:18 centu478 New Issue
2017-03-13 00:18 centu478 File Added: nfsprob
2017-03-15 01:41 kabe Note Added: 0028856
2017-03-15 03:28 kabe Note Added: 0028857
2017-03-15 04:31 kabe File Added: 0001-KEYS-request_key-should-reget-expired-keys-rather-th.patch
2017-03-15 04:46 kabe Note Added: 0028858
2017-03-15 05:26 kabe Note Added: 0028859
2017-03-15 09:01 tru Note Added: 0028860
2017-03-21 00:46 kabe Note Added: 0028891
+Issue History