2018-01-23 17:43 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013683CentOS-7systemdpublic2018-01-01 18:40
Reportermorty 
PriorityhighSeveritycrashReproducibilityrandom
StatusnewResolutionopen 
PlatformPowerEdge C6220 IIOSCentOSOS Version7.3
Product Version 
Target VersionFixed in Version 
Summary0013683: systemctl stops working after low memory conditions
DescriptionWe're sometimes seeing cases where systemd stops working in all kinds of different ways after a system runs out of memory. When the OOM condition clears, systemd is still running but broken. systemctl doesn't work. It won't even harvest defunct PIDs with PPID==1. For example:

[root@intprod21 ~]# systemctl
Failed to list units: Activation of org.freedesktop.systemd1 timed out
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|tail
sshd 57937 1 57936 Z [sshd] <defunct>
nrpe 57981 1 3153 Z [nrpe] <defunct>
nrpe 58250 1 3153 Z [nrpe] <defunct>
sshd 58298 1 58297 Z [sshd] <defunct>
sshd 58381 1 58380 Z [sshd] <defunct>
nrpe 58421 1 3153 Z [nrpe] <defunct>
sshd 58663 1 58662 Z [sshd] <defunct>
sshd 59028 1 59027 Z [sshd] <defunct>
nrpe 59118 1 3153 Z [nrpe] <defunct>
nrpe 59154 1 3153 Z [nrpe] <defunct>
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|wc -l
16162
[root@intprod21 ~]# ps fp 1
   PID TTY STAT TIME COMMAND
     1 ? Ss 22:41 /usr/lib/systemd/systemd --switched-root --system --
[root@intprod21 ~]#

This has happened on several of our systems. It always follows an "out of memory" condition. We get the system back for other purposes, but systemd doesn't recover. Rebooting clears it.

Unfortunately, I don't (yet?) have a reliable way to reproduce this.
TagsNo tags attached.
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes
There are no notes attached to this issue.
+Notes

-Issue History
Date Modified Username Field Change
2017-08-17 19:46 morty New Issue
+Issue History