View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0018343 | CentOS-8 | kernel | public | 2021-11-03 09:11 | 2021-11-08 10:37 |
| Reporter | benroberts_gsa | Assigned To | |||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | new | Resolution | open | ||
| Platform | x86_64 | OS | CentOS | OS Version | 8.3 |
| Product Version | 8.3.2011 | ||||
| Summary | 0018343: Expired/Revoked kernel keyring keys not garbage collected in 8.3, 8-stream kernels | ||||
| Description | On machines upgraded from CentOS 8.1 to 8.3 I'm seeing multiple users exceed their kernel keyring keys quota, and checking the output of `/proc/keys` for the affected users almost all keys are expired/revoked with a refcount of 1 or higher. Being revoked, these cannot be interacted with using keyctl. It appears that either garbage collection is not running, or these expired/revoked keys are not being cleaned up when GC does run. Example list of keys when the host is in this situation. ``` $ cat /proc/keys 00757a86 IR-Q--- 2 expd 3f030000 15171 5000 keyring _ses: empty 00aebcec IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 00d874cc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 00f354b4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 01317f10 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 01fc9be6 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0274d916 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 02fa88a1 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0301f42c IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0307418a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0337fe9a IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0363e414 I--Q--- 3 perm 1f3f0000 15171 65534 keyring _uid.15171: empty 03a58bbc IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 03ded0ca IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 03e00b58 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 047825a4 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 049eb199 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 04d34978 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 052f1971 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 065b1604 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 065df0d3 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 066f89d9 I--Q--- 1 perm 1f3f0000 15171 65534 keyring _uid_ses.15171: 1 06749a56 IR-Q--- 1 expd 3f030000 15171 5000 keyring _ses: empty 0680f7be IR-Q--- 18 expd 3f030000 15171 5000 keyring _ses: empty # ... ``` I have verified that downgrading to an 8.1 kernel (leaving the userspace on 8.3), the GC does run correctly and and expired keys do not hang around indefinitely. I have also tested the latest 8-stream kernel, where GC does not clean up the expired keys also. Kernel versions tested: - 8.1 4.18.0-147.5.1.el8.x86_64 NOT affected - 8.3 4.18.0-240.1.1.el8.x86_64 affected - 8.3 4.18.0-240.22.1.el8.x86_64 affected - 8-stream 4.18.0-348.el8.x86_64 affected As a temporary workaround, I am increasing the maxkeys and maxbytes sysctl settings so that users do not run into quota issues during normal operation. | ||||
| Steps To Reproduce | - Boot system with 8.3 or 8.4 kernel - SSH in to the host, logout, repeat a few times - Observe from /proc/key-users that the number of keys in use is increasing - Observe from /proc/keys that keys relating to the logged out SSH connections are revoked/expired - Wait > 5 minutes for the GC routine to run - Observe that expired/revoked keys have not been cleaned up - On a system with sufficient activity, observe that the per-user quota limit will be reached and subsequently new keys cannot be created. | ||||
| Tags | No tags attached. | ||||
 |
Because Stream kernels are involved, please file this bug report upstream at bugzilla.redhat.org. Details are in: https://wiki.centos.org/ReportBugs |
 |
Reported upstream: https://bugzilla.redhat.com/show_bug.cgi?id=2021088 |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2021-11-03 09:11 | benroberts_gsa | New Issue | |
| 2021-11-08 10:12 | toracat | Note Added: 0038720 | |
| 2021-11-08 10:37 | benroberts_gsa | Note Added: 0038721 |
