View Issue Details

IDProjectCategoryView StatusLast Update
0018343CentOS-8kernelpublic2021-11-08 10:37
Reporterbenroberts_gsa Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Platformx86_64OSCentOSOS Version8.3
Product Version8.3.2011 
Summary0018343: Expired/Revoked kernel keyring keys not garbage collected in 8.3, 8-stream kernels
DescriptionOn machines upgraded from CentOS 8.1 to 8.3 I'm seeing multiple users exceed their kernel keyring keys quota, and checking the output of `/proc/keys` for the affected users almost all keys are expired/revoked with a refcount of 1 or higher. Being revoked, these cannot be interacted with using keyctl. It appears that either garbage collection is not running, or these expired/revoked keys are not being cleaned up when GC does run.

Example list of keys when the host is in this situation.
```
$ cat /proc/keys
00757a86 IR-Q--- 2 expd 3f030000 15171 5000 keyring _ses: empty
00aebcec IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
00d874cc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
00f354b4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
01317f10 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
01fc9be6 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0274d916 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
02fa88a1 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0301f42c IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0307418a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0337fe9a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0363e414 I--Q--- 3 perm 1f3f0000 15171 65534 keyring _uid.15171: empty
03a58bbc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
03ded0ca IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
03e00b58 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
047825a4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
049eb199 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
04d34978 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
052f1971 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
065b1604 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
065df0d3 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
066f89d9 I--Q--- 1 perm 1f3f0000 15171 65534 keyring _uid_ses.15171: 1
06749a56 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty
0680f7be IR-Q--- 18 expd 3f030000 15171 5000 keyring _ses: empty
# ...
```

I have verified that downgrading to an 8.1 kernel (leaving the userspace on 8.3), the GC does run correctly and and expired keys do not hang around indefinitely.
I have also tested the latest 8-stream kernel, where GC does not clean up the expired keys also.

Kernel versions tested:
- 8.1 4.18.0-147.5.1.el8.x86_64 NOT affected
- 8.3 4.18.0-240.1.1.el8.x86_64 affected
- 8.3 4.18.0-240.22.1.el8.x86_64 affected
- 8-stream 4.18.0-348.el8.x86_64 affected

As a temporary workaround, I am increasing the maxkeys and maxbytes sysctl settings so that users do not run into quota issues during normal operation.
Steps To Reproduce- Boot system with 8.3 or 8.4 kernel
- SSH in to the host, logout, repeat a few times
- Observe from /proc/key-users that the number of keys in use is increasing
- Observe from /proc/keys that keys relating to the logged out SSH connections are revoked/expired
- Wait > 5 minutes for the GC routine to run
- Observe that expired/revoked keys have not been cleaned up
- On a system with sufficient activity, observe that the per-user quota limit will be reached and subsequently new keys cannot be created.
TagsNo tags attached.

Activities

toracat

toracat

2021-11-08 10:12

manager   ~0038720

Because Stream kernels are involved, please file this bug report upstream at bugzilla.redhat.org. Details are in:

https://wiki.centos.org/ReportBugs
benroberts_gsa

benroberts_gsa

2021-11-08 10:37

reporter   ~0038721

Reported upstream: https://bugzilla.redhat.com/show_bug.cgi?id=2021088

Issue History

Date Modified Username Field Change
2021-11-03 09:11 benroberts_gsa New Issue
2021-11-08 10:12 toracat Note Added: 0038720
2021-11-08 10:37 benroberts_gsa Note Added: 0038721