| Description | We're sometimes seeing cases where systemd stops working in all kinds of different ways after a system runs out of memory. When the OOM condition clears, systemd is still running but broken. systemctl doesn't work. It won't even harvest defunct PIDs with PPID==1. For example:
[root@intprod21 ~]# systemctl
Failed to list units: Activation of org.freedesktop.systemd1 timed out
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|tail
sshd 57937 1 57936 Z [sshd] <defunct>
nrpe 57981 1 3153 Z [nrpe] <defunct>
nrpe 58250 1 3153 Z [nrpe] <defunct>
sshd 58298 1 58297 Z [sshd] <defunct>
sshd 58381 1 58380 Z [sshd] <defunct>
nrpe 58421 1 3153 Z [nrpe] <defunct>
sshd 58663 1 58662 Z [sshd] <defunct>
sshd 59028 1 59027 Z [sshd] <defunct>
nrpe 59118 1 3153 Z [nrpe] <defunct>
nrpe 59154 1 3153 Z [nrpe] <defunct>
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|wc -l
16162
[root@intprod21 ~]# ps fp 1
PID TTY STAT TIME COMMAND
1 ? Ss 22:41 /usr/lib/systemd/systemd --switched-root --system --
[root@intprod21 ~]#
This has happened on several of our systems. It always follows an "out of memory" condition. We get the system back for other purposes, but systemd doesn't recover. Rebooting clears it.
Unfortunately, I don't (yet?) have a reliable way to reproduce this. |
|---|
