View Issue Details

IDProjectCategoryView StatusLast Update
0012950CentOS-7nfs-utilspublic2017-06-02 01:11
Status newResolutionopen 
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0012950: problem with the management of the ID map cache
DescriptionCentOS 7.3 NFS server, CentOS 7.3 NFS client. NFS V4. NFS mount works fine. ID mapping works fine. However, if file system is unmounted and then mounted again after 10 minutes, UID and GID shows as 4294967294. Repeating the process, the ID mapping is correct. Repeating the process again, the ID mapping is wrong -- it alternates between correct and wrong.
This happens with kernel 3.10.0-514.6.2.el7.x86_64 and
Replacing /usr/sbin/nfsidmap with the version from
solves the problem.
Steps To Reproduceserver name: server
file system on server: /export/foo
/export/foo on server includes directory /export/foo/bar
/export/foo/bar uid and gid map to the same normal user/group on both client and server

As root, run the "nfsprob" script attached. Output will have correct user/group in first iteration, then 4294967294, then correct again, etc.
TagsNo tags attached.




2017-03-13 00:18


nfsprob (401 bytes)
#!/bin/csh -f

#  server name:  server
#  file system on server:  /export/foo
#  /export/foo on server includes directory /export/foo/bar
#  /export/foo/bar uid and gid map to the same normal user/group
                   on both client and server

while (1)
   mount -t nfs4 server:/export/foo /mnt
   echo -n `date +"%d %H:%M:%S"` " "
   ls -ld /mnt/bar
   sleep 300
   umount /mnt
   sleep 310
nfsprob (401 bytes)


2017-03-15 01:41

reporter   ~0028856

I've succeeded in reproducing this.

- On NFS server, "echo Y > /sys/module/nfsd/parameters/nfs4_disable_idmapping" #enable non-default id-mapping
- Edit /etc/idmapd.conf, set Domain=<domain> to same string in NFS server and client

- nfs client: mount -o nfsvers=4 server:/export/foo /mnt
- nfs client: wait until kernel keys shown in
  grep id_ /proc/keys
  go to "expd" (expired) state
  (default 600secs, tunable in /etc/request-key.d/id_resolver.conf: nfsidmap -t 600)
- nfs client: umount /mnt; mount -o nfsvers=4 server:/export/foo /mnt
- nfs client: ls -l /mnt

Actual result:
ls -l /mnt
-rw-r--r--. 1 4294967294 4294967294 4 3? 14 16:34 bar-1001
-rw-r--r--. 1 4294967294 4294967294 4 3? 14 16:35 baz-501

"id_resolver" kernel keys, by default, is
created with 600 seconds lifetime, and after that, becomes
"expd" state, then deleted after 300 seconds.
If you umount/re-mount the NFS share during this 300 second window,
symptom (uid=(uid_t)-2) appears.

This is regression, or proper bugfix of,
  nfsidmap not setting key timeouts
Using old nfs-utils-1.3.0-0.21, nfsidmap creates
nonexpiring "permanent" (perm) kernel keys, which masks the problem.
(You won't be able to change the mapping until reboot, though)

For unknown reason, the expired keys aren't refreshed properly.

On nfs client, invoke "systemctl start nfs" (yes, nfs server)
For unknown reason, this makes "id_legacy" keys linger for 600 seconds,
taking over the data of "id_resolver" keys.


2017-03-15 03:28

reporter   ~0028857

- #enable non-default id-mapping
  On NFS server, "echo N > /sys/module/nfsd/parameters/nfs4_disable_idmapping"


2017-03-15 04:31


0001-KEYS-request_key-should-reget-expired-keys-rather-th.patch (4,757 bytes)
From 9618e2c8d956d3374feb0b3b4e64bd99c9b6237f Mon Sep 17 00:00:00 2001
From: "T.kabe" <kabe@>
Date: Wed, 15 Mar 2017 12:57:49 +0900
Subject: [PATCH] KEYS: request_key() should reget expired keys rather than

[upstream commit 0b0a84154eff56913e91df29de5c3a03a0029e38]
Author: David Howells <>
Date:   Mon Dec 1 22:52:53 2014 +0000

KEYS: request_key() should reget expired keys rather than give EKEYEXPIRED

Since the keyring facility can be viewed as a cache (at least in some
applications), the local expiration time on the key should probably be viewed
as a 'needs updating after this time' property rather than an absolute 'anyone
now wanting to use this object is out of luck' property.

Since request_key() is the main interface for the usage of keys, this should
update or replace an expired key rather than issuing EKEYEXPIRED if the local
expiration has been reached (ie. it should refresh the cache).

For absolute conditions where refreshing the cache probably doesn't help, the
key can be negatively instantiated using KEYCTL_REJECT_KEY with EKEYEXPIRED
given as the error to issue.  This will still cause request_key() to return
EKEYEXPIRED as that was explicitly set.

In the future, if the key type has an update op available, we might want to
upcall with the expired key and allow the upcall to update it.  We would pass
a different operation name (the first column in /etc/request-key.conf) to the
request-key program.

request_key() returning EKEYEXPIRED is causing an NFS problem which Chuck
Lever describes thusly:

    After about 10 minutes, my NFSv4 functional tests fail because the
    ownership of the test files goes to "-2". Looking at /proc/keys
    shows that the id_resolv keys that map to my test user ID have
    expired. The ownership problem persists until the expired keys are
    purged from the keyring, and fresh keys are obtained.

    I bisected the problem to 3.13 commit b2a4df200d57 ("KEYS: Expand
    the capacity of a keyring"). This commit inadvertantly changes the
    API contract of the internal function keyring_search_aux().

    The root cause appears to be that b2a4df200d57 made "no state check"
    the default behavior. "No state check" means the keyring search
    iterator function skips checking the key's expiry timeout, and
    returns expired keys.  request_key_and_link() depends on getting
    an -EAGAIN result code to know when to perform an upcall to refresh
    an expired key.

This patch can be tested directly by:

    keyctl request2 user debug:fred a @s
    keyctl timeout %user:debug:fred 3
    sleep 4
    keyctl request2 user debug:fred a @s

Without the patch, the last command gives error EKEYEXPIRED, but with the
command it gives a new key.

Reported-by: Carl Hetherington <>
Reported-by: Chuck Lever <>
Signed-off-by: David Howells <>
Tested-by: Chuck Lever <>
 security/keys/internal.h    | 1 +
 security/keys/keyring.c     | 3 ++-
 security/keys/request_key.c | 3 ++-
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/security/keys/internal.h b/security/keys/internal.h
index 80b2aac..6d5c72e 100644
--- a/security/keys/internal.h
+++ b/security/keys/internal.h
@@ -121,6 +121,7 @@ struct keyring_search_context {
 #define KEYRING_SEARCH_NO_UPDATE_TIME	0x0008	/* Don't update times */
 #define KEYRING_SEARCH_NO_CHECK_PERM	0x0010	/* Don't check permissions */
 #define KEYRING_SEARCH_DETECT_TOO_DEEP	0x0020	/* Give an error on excessive depth */
+#define KEYRING_SEARCH_SKIP_EXPIRED	0x0040	/* Ignore expired keys (intention to replace) */
 	int (*iterator)(const void *object, void *iterator_data);
diff --git a/security/keys/keyring.c b/security/keys/keyring.c
index 04d0d7c..13496c9 100644
--- a/security/keys/keyring.c
+++ b/security/keys/keyring.c
@@ -526,7 +526,8 @@ static int keyring_search_iterator(const void *object, void *iterator_data)
 		if (key->expiry && ctx->now.tv_sec >= key->expiry) {
-			ctx->result = ERR_PTR(-EKEYEXPIRED);
+			if (!(ctx->flags & KEYRING_SEARCH_SKIP_EXPIRED))
+				ctx->result = ERR_PTR(-EKEYEXPIRED);
 			kleave(" = %d [expire]", ctx->skipped_ret);
 			goto skipped;
diff --git a/security/keys/request_key.c b/security/keys/request_key.c
index 67e413c..7054b36 100644
--- a/security/keys/request_key.c
+++ b/security/keys/request_key.c
@@ -519,7 +519,8 @@ struct key *request_key_and_link(struct key_type *type,
 		.cred			= current_cred(),
 		.match			= type->match,
 		.match_data		= description,
 	struct key *key;
 	key_ref_t key_ref;



2017-03-15 04:46

reporter   ~0028858

I've found an upstream commit which fixes this.

After applying this patch, the id_resolver kernel key is properly refreshed
and thus NFSv4 id-mapping works as expected, across umount/mount.
Bonus: id-mapping is properly updated after 600seconds timeout
if you changed the mapping.

Note that user still can't write to the file despite uid# seems same;
this is limitation of NFS RPC.

This is the patch revised for CentOS kernel-3.10.0-514:

I am going to file kernel-bug on Red Hat Bugzilla, but don't expect much
since enabling id-mapping seems a non-standard usage.
(RH docs implicity says to use LDAP to sync uid#)


2017-03-15 05:26

reporter   ~0028859



2017-03-15 09:01

administrator   ~0028860

thanks for your work, really appreciated :)
RH has ack'ed and we can expect a fix from them.


2017-03-21 00:46

reporter   ~0028891

RHBZ says it will be fixed in kernel-3.10.0-616.el7 .


2017-06-02 01:11

reporter   ~0029389

nfs-utils-1.3.0-0.33 causes key expiration problem and intermittent uid/gid 4...

is closed as CLOSED ERRATA.

Errata published as, mentions the
latest kernel-3.10.0-514.21.1 .

Replaying the problematic reproduction myself, the problem seems to be fixed.
id_resolv /proc/keys kernel keys are refreshed, not staying in "expd"(expired) state
when key timeout arrives.

We could close this bugreport.

Issue History

Date Modified Username Field Change
2017-03-13 00:18 centu478 New Issue
2017-03-13 00:18 centu478 File Added: nfsprob
2017-03-15 01:41 kabe Note Added: 0028856
2017-03-15 03:28 kabe Note Added: 0028857
2017-03-15 04:31 kabe File Added: 0001-KEYS-request_key-should-reget-expired-keys-rather-th.patch
2017-03-15 04:46 kabe Note Added: 0028858
2017-03-15 05:26 kabe Note Added: 0028859
2017-03-15 09:01 tru Note Added: 0028860
2017-03-21 00:46 kabe Note Added: 0028891
2017-06-02 01:11 kabe Note Added: 0029389