View Issue Details

IDProjectCategoryView StatusLast Update
0017461CentOS-8systemdpublic2020-07-19 13:02
Reporterqguv 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformLXDOSCentOSOS Version8.2
Product Version 
Target VersionFixed in Version 
Summary0017461: systemd-239-30 package prevents some units from stopping
DescriptionThe postgresql unit installed in the postgresql:9.6 package on centos 8.2 fails to stop when using systemd-239-30 and running in an LXD container. This issue does not appear with systemd-239-18 and was introduced with the newest package release. We're having this issue with the Centos 8 LXD image.

Some other systemd services that we write and install in /etc/systemd/system/ also fail, but I mention postgres here because it's an easily reproducible case.

There were 256 patches in the package for systemd-239-18 and 360 in systemd-239-30, so I assume the issue was introduced in a patch between 257 and 360.

To reproduce: install postgres, initialize the database, start postgresql. After this, postgresql will not stop. The state of the systemd service is stuck in 'deactivating' (final-sigkill), even though all processes started by systemd (postmaster, postgresql-check-db-dir, and postgres itself) are dead according to systemd and according to `ps`.
Steps To ReproduceWe create the container with `lxc init images:centos/8 centos8box`.

[ now, inside the container ]

$ dnf install @postgresql:9.6
[...]

$ su postgres -c "/usr/bin/initdb -D /var/lib/pgsql/data -E UTF8"
[...]

$ sudo systemctl start postgresql
[...]

$ sudo systemcl stop postgresql
[ this hangs forever ]
Additional Information$ sudo systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled; vendor preset: disabled)
   Active: deactivating (final-sigkill) since Tue 2020-06-16 15:08:40 UTC; 10min ago
  Process: 519 ExecStart=/usr/bin/postmaster -D ${PGDATA} (code=exited, status=0/SUCCESS)
  Process: 517 ExecStartPre=/usr/libexec/postgresql-check-db-dir postgresql (code=exited, status=0/SUCCESS)
 Main PID: 519 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 202832)
   Memory: 2.8M
   CGroup: /system.slice/postgresql.serviceJun 16 15:08:33 xx systemd[1]: Starting PostgreSQL database server...
Jun 16 15:08:33 xx postmaster[519]: LOG: redirecting log output to logging collector process
Jun 16 15:08:33 xx postmaster[519]: HINT: Future log output will appear in directory "pg_log".
Jun 16 15:08:33 xx systemd[1]: Started PostgreSQL database server.
Jun 16 15:08:40 xx systemd[1]: Stopping PostgreSQL database server...
Jun 16 15:08:40 xx systemd[1]: postgresql.service: Killing process 520 (postmaster) with signal SIGKILL.

$ sudo ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 94636 10484 ? Ss 15:01 0:00 /usr/lib/systemd/systemd --system --deserialize 21
root 41 0.0 0.0 93432 11288 ? Ss 15:01 0:00 /usr/lib/systemd/systemd-journald
dbus 60 0.0 0.0 53072 4872 ? Ss 15:01 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 64 0.0 0.0 85444 7540 ? Ss 15:01 0:00 /usr/lib/systemd/systemd-logind
root 65 0.0 0.0 367084 16772 ? Ssl 15:01 0:00 /usr/sbin/NetworkManager --no-daemon
root 74 0.0 0.0 6520 1680 console Ss+ 15:01 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root 90 0.0 0.0 226728 3820 ? Ss 15:02 0:00 bash
root 170 0.0 0.0 22868 3148 ? Ss 15:02 0:00 /usr/sbin/crond -n
root 308 0.0 0.0 211524 4392 ? Ssl 15:03 0:00 /usr/sbin/rsyslogd -n
root 339 0.0 0.0 88232 7772 ? Ss 15:03 0:00 /usr/lib/systemd/systemd-udevd
root 559 0.0 0.0 264476 3804 ? R+ 15:20 0:00 ps aux
Tagssystemd

Activities

golmaal

golmaal

2020-07-09 22:56

reporter   ~0037349

First I run docker container on the host -
 
# docker run --privileged -dit --name pgdg centos:8.2.2004 /sbin/init

Then inside the container, I run the commands given below -

# yum -y install glibc-langpack-en glibc-locale-source
# localedef -i en_US -f UTF-8 en_US.UTF-8

# dnf install https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm

# dnf -qy module disable postgresql

# dnf install postgresql12-server

# /usr/pgsql-12/bin/postgresql-12-setup initdb

# systemctl enable --now postgresql-12

Postgresql starts at this point...

However, when I try to stop it, it just hangs -

# systemctl stop postgresql-12 (hangs)

systemd logs -

Jun 24 21:34:00 pgdg systemd[1]: Bus private-bus-connection: changing state UNSET → OPENING
Jun 24 21:34:00 pgdg systemd[1]: Bus private-bus-connection: changing state OPENING → AUTHENTICATING
Jun 24 21:34:00 pgdg systemd[1]: Accepted new private connection.
Jun 24 21:34:00 pgdg systemd[1]: Bus private-bus-connection: changing state AUTHENTICATING → RUNNING
Jun 24 21:34:00 pgdg systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=StopUnit cookie=1 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Trying to enqueue job postgresql-12.service/stop/replace
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Installed new job postgresql-12.service/stop as 110
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Enqueued job postgresql-12.service/stop as 110
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=o error-n
ame=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobNew cookie=2 reply_cookie=0 signature=uos error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member
=GetUnit cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=o error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Changed running -> stop-sigterm
Jun 24 21:34:00 pgdg systemd[1]: Stopping PostgreSQL 12 database server...
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=4 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=5 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/job/110 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=6 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=3 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=7 reply_cookie=3 signature=v error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: -.slice: Got notification message "STOPPING=1", ignoring.
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Got notification message from PID 166 (STOPPING=1)
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesk
top.DBus.Properties member=PropertiesChanged cookie=8 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=9 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Received SIGCHLD from PID 166 (postmaster).
Jun 24 21:34:00 pgdg systemd[1]: Child 166 (postmaster) died (code=exited, status=0/SUCCESS)
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Child 166 belongs to postgresql-12.service.
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Main process exited, code=exited, status=0/SUCCESS
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Killing process 195 (postmaster) with signal SIGKILL.
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Changed stop-sigterm -> final-sigkill
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=10 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/postgresql_2d12_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=11 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 24 21:34:00 pgdg systemd[1]: systemd-journald.service: Received EPOLLHUP on stored fd 42 (stored), closing.
Jun 24 21:34:00 pgdg systemd[1]: Received SIGCHLD from PID 195 (postmaster).
Jun 24 21:34:00 pgdg systemd[1]: Child 195 (postmaster) died (code=killed, status=9/KILL)
Jun 24 21:34:00 pgdg systemd[1]: postgresql-12.service: Child 195 belongs to postgresql-12.service.

Process list before trying to stop the postgresql-12 service -
postgres 166 0.0 0.0 287100 24900 ? Ss 21:32 0:00 /usr/pgsql-12/bin/postmaster -D /var/lib/pgsql/12/data/
postgres 195 0.0 0.0 141480 4928 ? Ss 21:32 0:00 postgres: logger
postgres 200 0.0 0.0 287100 6116 ? Ss 21:32 0:00 postgres: checkpointer
postgres 201 0.0 0.0 287100 6048 ? Ss 21:32 0:00 postgres: background writer
postgres 203 0.0 0.0 287100 10484 ? Ss 21:32 0:00 postgres: walwriter
postgres 204 0.0 0.0 287636 6892 ? Ss 21:32 0:00 postgres: autovacuum launcher
postgres 206 0.0 0.0 141480 4936 ? Ss 21:32 0:00 postgres: stats collector
postgres 207 0.0 0.0 287524 6696 ? Ss 21:32 0:00 postgres: logical replication launcher

All processes were killed, however, the systemctl stop postgresql-12 is still hung.
filovitch

filovitch

2020-07-19 13:02

reporter   ~0037381

I also have this problem with centos 8 lxd container or centos 8 proxmox container. The problem clearly come from systemd-239-30 but I really do not know how to track this problem down.

Issue History

Date Modified Username Field Change
2020-06-16 15:50 qguv New Issue
2020-06-16 15:50 qguv Tag Attached: systemd
2020-07-09 22:56 golmaal Note Added: 0037349
2020-07-19 13:02 filovitch Note Added: 0037381