View Issue Details

IDProjectCategoryView StatusLast Update
0013683CentOS-7systemdpublic2020-04-21 06:22
Reportermorty 
PriorityhighSeveritycrashReproducibilityrandom
Status newResolutionopen 
PlatformPowerEdge C6220 IIOSCentOSOS Version7.3
Product Version 
Target VersionFixed in Version 
Summary0013683: systemctl stops working after low memory conditions
DescriptionWe're sometimes seeing cases where systemd stops working in all kinds of different ways after a system runs out of memory. When the OOM condition clears, systemd is still running but broken. systemctl doesn't work. It won't even harvest defunct PIDs with PPID==1. For example:

[root@intprod21 ~]# systemctl
Failed to list units: Activation of org.freedesktop.systemd1 timed out
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|tail
sshd 57937 1 57936 Z [sshd] <defunct>
nrpe 57981 1 3153 Z [nrpe] <defunct>
nrpe 58250 1 3153 Z [nrpe] <defunct>
sshd 58298 1 58297 Z [sshd] <defunct>
sshd 58381 1 58380 Z [sshd] <defunct>
nrpe 58421 1 3153 Z [nrpe] <defunct>
sshd 58663 1 58662 Z [sshd] <defunct>
sshd 59028 1 59027 Z [sshd] <defunct>
nrpe 59118 1 3153 Z [nrpe] <defunct>
nrpe 59154 1 3153 Z [nrpe] <defunct>
[root@intprod21 ~]# ps faxo user,pid,ppid,pgid,state,command|awk '$3==1 && $5=="Z"'|wc -l
16162
[root@intprod21 ~]# ps fp 1
   PID TTY STAT TIME COMMAND
     1 ? Ss 22:41 /usr/lib/systemd/systemd --switched-root --system --
[root@intprod21 ~]#

This has happened on several of our systems. It always follows an "out of memory" condition. We get the system back for other purposes, but systemd doesn't recover. Rebooting clears it.

Unfortunately, I don't (yet?) have a reliable way to reproduce this.
TagsNo tags attached.
abrt_hash
URL

Activities

xavinux

xavinux

2018-01-25 15:36

reporter   ~0031038

Please, post the CentOS version and kernel you are using in your Servers:

uname -r
3.10.0-693.11.6.el7.x86_64

cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
satty773

satty773

2020-03-06 10:42

reporter   ~0036469

any updates on this issue?
rajkumar.vinoo

rajkumar.vinoo

2020-04-21 06:22

reporter   ~0036741

We are also facing this issue on multiple hosts.. It would be great to have a workaround/fix for this..

Issue History

Date Modified Username Field Change
2017-08-17 19:46 morty New Issue
2018-01-25 15:36 xavinux Note Added: 0031038
2020-03-06 10:42 satty773 Note Added: 0036469
2020-04-21 06:22 rajkumar.vinoo Note Added: 0036741