View Issue Details

IDProjectCategoryView StatusLast Update
0013683CentOS-7systemdpublic2018-01-25 15:42
Reportermorty 
PriorityhighSeveritycrashReproducibilityrandom
Status newResolutionopen 
PlatformPowerEdge C6220 IIOSCentOSOS Version7.3
Product Version 
Target VersionFixed in Version 
Summary0013683: systemctl stops working after low memory conditions
DescriptionWe're sometimes seeing cases where systemd stops working in all kinds of different ways after a system runs out of memory. When the OOM condition clears, systemd is still running but broken. systemctl doesn't work. It won't even harvest defunct PIDs with PPID==1. For example:

[root@intprod21 ~]# systemctl
Failed to list units: Activation of org.freedesktop.systemd1 timed out
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|tail
sshd 57937 1 57936 Z [sshd] <defunct>
nrpe 57981 1 3153 Z [nrpe] <defunct>
nrpe 58250 1 3153 Z [nrpe] <defunct>
sshd 58298 1 58297 Z [sshd] <defunct>
sshd 58381 1 58380 Z [sshd] <defunct>
nrpe 58421 1 3153 Z [nrpe] <defunct>
sshd 58663 1 58662 Z [sshd] <defunct>
sshd 59028 1 59027 Z [sshd] <defunct>
nrpe 59118 1 3153 Z [nrpe] <defunct>
nrpe 59154 1 3153 Z [nrpe] <defunct>
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|wc -l
16162
[root@intprod21 ~]# ps fp 1
   PID TTY STAT TIME COMMAND
     1 ? Ss 22:41 /usr/lib/systemd/systemd --switched-root --system --
[root@intprod21 ~]#

This has happened on several of our systems. It always follows an "out of memory" condition. We get the system back for other purposes, but systemd doesn't recover. Rebooting clears it.

Unfortunately, I don't (yet?) have a reliable way to reproduce this.
TagsNo tags attached.
abrt_hash
URL

Activities

xavinux

xavinux

2018-01-25 15:36

reporter   ~0031038

Please, post the CentOS version and kernel you are using in your Servers:

uname -r
3.10.0-693.11.6.el7.x86_64

cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

Issue History

Date Modified Username Field Change
2017-08-17 19:46 morty New Issue
2018-01-25 15:36 xavinux Note Added: 0031038