View Issue Details

IDProjectCategoryView StatusLast Update
0014278CentOS-7systemdpublic2020-08-17 07:38
ReporterMarkusKoo Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Summary0014278: systemd update causes /run tmpfs to fill up with sessions
DescriptionAfter a systemd update on october 21. Our servers /run/systemd/sessions keep filling up with files and it hogs the whole /run filesystem

# du -hs /run/systemd/sessions
976M /run/systemd/sessions

Systemd updated on 21. of october and after that session files are not cleaned from /run/systemd/sessions. This causes the system to become unstable, if run partition disk space runs out

Boot fixes the problem

This is also confirmed by my friend who administeres RHEL 7 environment.
TagsNo tags attached.
abrt_hash
URL

Activities

TrevorH

TrevorH

2017-12-15 17:26

manager   ~0030754

It would appear that updates to systemd now require a reboot to avoid this sort of problem and others related to it. The problem appears to be a known one upstream and newer systemd versions have been fixed - it's to do with losing its dbus connection.
pmozdz

pmozdz

2018-03-07 06:55

reporter   ~0031374

Hello , I experienced the same issue , the latest package in Centos repo is still "systemd-219-42"
rbro

rbro

2018-03-16 08:22

reporter   ~0031449

I am running into the same issue. /run/systemd/sessions has been filling up with files since October 22, and tonight it maxed out the disk space of /run. Is there any way to fix this without rebooting?
TrevorH

TrevorH

2018-03-16 08:25

manager   ~0031450

No. You should treat systemd updates as requiring a reboot for now until upstream fix the bug that causes it to become disconnected from dbus.
rbro

rbro

2018-03-16 08:44

reporter   ~0031451

Thanks, will the problem happen again on the next update of systemd even if I reboot now? Is there a bug report ticket for upstream? Seems like a critical issue as all my servers are slowly maxing out disk space on /run.
MarkusKoo

MarkusKoo

2018-03-16 15:55

reporter   ~0031453

@rbro Have not experienced this anymore on any of our servers after a reboot.
ethan_xu

ethan_xu

2020-08-17 07:24

reporter   ~0037563

systemd version the issue has been seen with

219.42.el7_4.4

Used distribution

centos 7.4

Expected behaviour you didn't see

system-logind and dbus should not occupy one cpu nearly 100% and there should be no watchdog behavior in system-logind(log as "systemd-logind.service watchdog timeout (limit 1min)!" )

Unexpected behaviour you saw

huge closing sessions almost 30w+ appeared in the /run/systemd/sessions directory

Steps to reproduce the problem
the issue machines have the huge multi-sessions connections from remote login via ssh and crond tasks in localhost with 5min cycles. and I could not find the stable approach to reproduce the issue until now:(

I searched the issue with google and centos bugzilla, then found that the issue is a know issue as below link:

https://bugs.centos.org/view.php?id=14278

the issue has been resolved and needs to update the systemd version to solve the issue, but I could not update the systemd to the newest version because the fixup patches not related to the issue are so much, must consider compatibility and avoid bringing new issue for centos 7.4.
We have so much online machine using systemd 219.42.el7_4.4, update to newest is difficult for us.

The best way for us is to backport the fixup patches to the 219.42.el7_4.4, So my question is could you please help me:

1. provide us the fixup patches only for solving the above issue, we can backport the patches to our issue version?
2. provide us the stable approach to reproduce this issue?

btw, we think the issue is not same with #1961
we had checked the above patches which have been merged into 219.42.el7_4.4 version

Thanks for taking the time~
ManuelWolfshant

ManuelWolfshant

2020-08-17 07:37

manager   ~0037564

CentOS 7.4 is no longer supported for many years . The CentOS projects only supports, at any given time, only the very latest minor release of any major release. Even RedHat, which is the upstream of CentOS and offers the so called EUS ( Extended Update ) subscription does not support RHEL 7.4 for many years.
You are welcome to use the sources freely provided at git.centos.org to backport the fixes you need for your system or to hire someone to do that for you. The CentOS project does not have either the resources or the willingness to do your work for free..

As a side note, you really really really should fully update your systems, no matter how difficult you believe this to be. There are literally thousands of issues - including many security related one - that were corrected between the version you use and the one recommended for use.

Issue History

Date Modified Username Field Change
2017-12-15 17:23 MarkusKoo New Issue
2017-12-15 17:26 TrevorH Note Added: 0030754
2018-03-07 06:55 pmozdz Note Added: 0031374
2018-03-16 08:22 rbro Note Added: 0031449
2018-03-16 08:25 TrevorH Note Added: 0031450
2018-03-16 08:44 rbro Note Added: 0031451
2018-03-16 15:55 MarkusKoo Note Added: 0031453
2020-08-17 07:24 ethan_xu Note Added: 0037563
2020-08-17 07:37 ManuelWolfshant Note Added: 0037564
2020-08-17 07:38 ManuelWolfshant Status new => closed
2020-08-17 07:38 ManuelWolfshant Resolution open => fixed