View Issue Details

IDProjectCategoryView StatusLast Update
0016900CentOS-7systemdpublic2020-01-08 07:35
Reporterzyy 
PriorityhighSeveritymajorReproducibilityhave not tried
Status newResolutionopen 
Product Version7.7-1908 
Target VersionFixed in Version 
Summary0016900: systemd 219-67.el7_7.2 segment fault during reloading
DescriptionA lot of zombie processes were found in one of the VMs run docker containers.
[root@mail ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (AltArch)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (AltArch)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7:server"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@mail ~]# uname -r
4.18.0-80.7.2.el7.aarch64
[root@mail ~]#

Jan 07 11:31:08 mail.tecmint.lan systemd[1]: Reloading.
Jan 07 11:31:08 mail.tecmint.lan systemd[1]: Caught <SEGV>, dumped core as pid 22474.
Jan 07 11:31:08 mail.tecmint.lan systemd[1]: Freezing execution.
Jan 07 11:31:08 mail.tecmint.lan systemd[1]: Caught <SEGV>, dumped core as pid 22475.
Jan 07 11:31:08 mail.tecmint.lan systemd[1]: Freezing execution.

Obviously, systemd was frozen.

[root@mail ~]# gdb /core.22474
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 22474]
Reading symbols from /usr/lib/systemd/systemd...Reading symbols from /usr/lib/debug/usr/lib/systemd/systemd.debug...done.
done.
Missing separate debuginfo for
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/75/04c0d706c74c58e446c7e614e21bd19f0c0aff
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/lib/systemd/systemd --system --deserialize 18'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000ffff91de5488 in kill () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.5-4.el7.aarch64 glibc-2.17-292.el7.aarch64 kmod-libs-20-25.el7.aarch64 libattr-2.4.46-13.el7.aarch64 libblkid-2.23.2-61.el7_7.1.aarch64 libcap-2.22-10.el7.aarch64 libcap-ng-0.7.5-4.el7.aarch64 libmount-2.23.2-61.el7_7.1.aarch64 libselinux-2.5-14.1.el7.aarch64 libuuid-2.23.2-61.el7_7.1.aarch64 pam-1.1.8-22.el7.aarch64 pcre-8.32-17.el7.aarch64 xz-libs-5.2.2-1.el7.aarch64 zlib-1.2.7-18.el7.aarch64
(gdb) bt
#0 0x0000ffff91de5488 in kill () from /lib64/libc.so.6
#1 0x0000aaaabf67747c in crash (sig=11) at src/core/main.c:172
#2 <signal handler called>
#3 0x0000ffff91e38ae0 in stpcpy () from /lib64/libc.so.6
#4 0x0000aaaabf6eb740 in stpcpy (__src=<optimized out>, __dest=<optimized out>) at /usr/include/bits/string3.h:111
#5 strjoin (x=x@entry=0xaaaabf75e820 "/sys/fs/cgroup/") at src/shared/util.c:5342
#6 0x0000aaaabf6ce318 in join_path (controller=<optimized out>, path=<optimized out>, suffix=<optimized out>, fs=0xfffff8c68530) at src/shared/cgroup-util.c:458
#7 0x0000aaaabf6cefa4 in cg_enumerate_subgroups (controller=controller@entry=0xaaaabf749e28 "name=systemd",
    path=path@entry=0xaaaafc34f5d0 "/sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/s"..., _d=_d@entry=0xfffff8c68598) at src/shared/cgroup-util.c:97
#8 0x0000aaaabf6cf094 in cg_is_empty_recursive (controller=controller@entry=0xaaaabf749e28 "name=systemd",
    path=0xaaaafc34f5d0 "/sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/s"..., ignore_self=ignore_self@entry=true) at src/shared/cgroup-util.c:923
#9 0x0000aaaabf68e2a8 in cgroup_good (s=<optimized out>) at src/core/service.c:1329
#10 0x0000aaaabf68e308 in service_may_gc (u=0xaaaafc331b70) at src/core/service.c:2457
#11 0x0000aaaabf71c07c in unit_may_gc (u=0xaaaafc331b70) at src/core/unit.c:322
#12 0x0000aaaabf71c0dc in unit_add_to_gc_queue (u=0xaaaafc331b70) at src/core/unit.c:355
#13 0x0000aaaabf720a8c in unit_ref_unset (ref=0xaaaafc3ef5a8) at src/core/unit.c:3155
#14 0x0000aaaabf7222d8 in unit_free (u=0xaaaafc331b70) at src/core/unit.c:525
#15 0x0000aaaabf67a2d4 in manager_clear_jobs_and_units (m=m@entry=0xaaaafc290740) at src/core/manager.c:970
#16 0x0000aaaabf67e0bc in manager_reload (m=0xaaaafc290740) at src/core/manager.c:2804
#17 0x0000aaaabf674b30 in main (argc=4, argv=0xfffff8c691d8) at src/core/main.c:1799
(gdb) f 10
#10 0x0000aaaabf68e308 in service_may_gc (u=0xaaaafc331b70) at src/core/service.c:2457
2457 if (cgroup_good(s) > 0 ||
(gdb) p *s
$1 = {meta = {manager = 0xaaaafc290740, type = UNIT_SERVICE, load_state = UNIT_LOADED, merged_into = 0x0, id = 0xaaaafc3322f0 "rsyslog.service", instance = 0x0, names = 0xaaaafc332260,
    dependencies = {0xaaaafc37ff00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaaaafc37eef0, 0x0, 0x0, 0xaaaafc3eedd0, 0x0, 0xaaaafc3eed90, 0xaaaafc3ee5a0, 0x0, 0x0, 0xaaaafc377ac0, 0x0,
      0x0, 0x0, 0xaaaafc3bd800, 0xaaaafc37ef30}, requires_mounts_for = 0x0,
    description = 0xaaaafc3eefc0 "roup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sy"..., documentation = 0xaaaafc3ee620, fragment_path = 0xaaaafc2a4250 "/usr/lib/systemd/system/rsyslog.service", source_path = 0x0,
    dropin_paths = 0x0, fragment_mtime = 1577261508846770, source_mtime = 0, dropin_mtime = 0, job = 0x0, nop_job = 0x0, job_timeout = 0, job_timeout_action = EMERGENCY_ACTION_NONE,
    job_timeout_reboot_arg = 0x0, refs_by_target = 0xaaaafc3ef5a8, conditions = 0x0, asserts = 0x0, condition_timestamp = {realtime = 1576468742224899, monotonic = 6817259},
    assert_timestamp = {realtime = 1576468742224915, monotonic = 6817259}, inactive_exit_timestamp = {realtime = 1576468742226333, monotonic = 6818677}, active_enter_timestamp = {
      realtime = 1576468742303290, monotonic = 6895634}, active_exit_timestamp = {realtime = 0, monotonic = 0}, inactive_enter_timestamp = {realtime = 0, monotonic = 0}, slice = {
      source = 0x0, target = 0x0, refs_by_target_next = 0x0, refs_by_target_prev = 0x0}, units_by_type_next = 0xaaaafc331400, units_by_type_prev = 0xaaaafc37ef70,
    has_requires_mounts_for_next = 0x0, has_requires_mounts_for_prev = 0x0, load_queue_next = 0x0, load_queue_prev = 0x0, dbus_queue_next = 0x0, dbus_queue_prev = 0x0,
    cleanup_queue_next = 0x0, cleanup_queue_prev = 0x0, gc_queue_next = 0x0, gc_queue_prev = 0x0, cgroup_queue_next = 0x0, cgroup_queue_prev = 0x0, target_deps_queue_next = 0x0,
    target_deps_queue_prev = 0x0, pids = 0x0, sigchldgen = 0, gc_marker = 325258, deserialized_job = -1, load_error = 0, unit_file_state = _UNIT_FILE_STATE_INVALID,
    unit_file_preset = -1,
    cgroup_path = 0xaaaafc34f5d0 "/sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/s"..., cgroup_realized_mask = 0, cgroup_subtree_mask = 11, cgroup_members_mask = 0, on_failure_job_mode = JOB_REPLACE,
    stop_when_unneeded = false, default_dependencies = true, refuse_manual_start = false, refuse_manual_stop = false, allow_isolate = false, ignore_on_isolate = false,
    ignore_on_snapshot = false, condition_result = true, assert_result = true, transient = false, in_load_queue = false, in_dbus_queue = false, in_cleanup_queue = false,
    in_gc_queue = false, in_cgroup_queue = false, in_target_deps_queue = false, sent_dbus_new_signal = true, no_gc = false, in_audit = false, cgroup_realized = false,
    cgroup_members_mask_valid = true, cgroup_subtree_mask_valid = true}, type = SERVICE_NOTIFY, restart = SERVICE_RESTART_ON_FAILURE, restart_prevent_status = {status = 0x0,
    signal = 0x0}, restart_force_status = {status = 0x0, signal = 0x0}, success_status = {status = 0x0, signal = 0x0}, pid_file = 0x0, restart_usec = 100000,
  timeout_start_usec = 90000000, timeout_stop_usec = 90000000, watchdog_timestamp = {realtime = 0, monotonic = 0}, watchdog_usec = 0, watchdog_event_source = 0x0, exec_command = {0x0,
    0x0, 0x0, 0x0, 0x0, 0x0}, exec_context = {environment = 0x0, environment_files = 0x0, pass_environment = 0x0, rlimit = {0x0 <repeats 16 times>}, working_directory = 0x0,
    root_directory = 0x0, working_directory_missing_ok = false, umask = 18, oom_score_adjust = 0, nice = 0, ioprio = 16384, cpu_sched_policy = 0, cpu_sched_priority = 0, cpuset = 0x0,
    cpuset_ncpus = 0, std_input = EXEC_INPUT_NULL, std_output = EXEC_OUTPUT_NULL, std_error = EXEC_OUTPUT_INHERIT, timer_slack_nsec = 18446744073709551615, tty_path = 0x0,
    tty_reset = false, tty_vhangup = false, tty_vt_disallocate = false, ignore_sigpipe = true, user = 0x0, group = 0x0, supplementary_groups = 0x0, pam_name = 0x0, utmp_id = 0x0,
    selinux_context_ignore = false, selinux_context = 0x0, apparmor_profile_ignore = false, apparmor_profile = 0x0, smack_process_label_ignore = false, smack_process_label = 0x0,
    read_write_dirs = 0x0, read_only_dirs = 0x0, inaccessible_dirs = 0x0, mount_flags = 0, capability_bounding_set = 18446744073709551615, capability_ambient_set = 0,
    capabilities = 0x0, secure_bits = 0, syslog_priority = 30, syslog_identifier = 0x0, syslog_level_prefix = true, cpu_sched_reset_on_fork = false, non_blocking = false,
    private_tmp = false, private_network = false, private_devices = false, protect_system = PROTECT_SYSTEM_NO, protect_home = PROTECT_HOME_NO, no_new_privileges = false,
    same_pgrp = false, personality = 4294967295, syscall_filter = 0x0, syscall_archs = 0x0, syscall_errno = 0, syscall_whitelist = false, address_families = 0x0,
    address_families_whitelist = false, runtime_directory = 0x0, runtime_directory_mode = 493, oom_score_adjust_set = false, nice_set = false, ioprio_set = false, cpu_sched_set = false,
    no_new_privileges_set = false, bus_endpoint = 0x0}, kill_context = {kill_mode = KILL_CONTROL_GROUP, kill_signal = 15, send_sigkill = true, send_sighup = false}, cgroup_context = {
    cpu_accounting = true, blockio_accounting = false, memory_accounting = true, tasks_accounting = false, cpu_shares = 18446744073709551615, startup_cpu_shares = 18446744073709551615,
    cpu_quota_per_sec_usec = 18446744073709551615, blockio_weight = 18446744073709551615, startup_blockio_weight = 18446744073709551615, blockio_device_weights = 0x0,
    blockio_device_bandwidths = 0x0, memory_limit = 18446744073709551615, device_policy = CGROUP_AUTO, device_allow = 0x0, tasks_max = 18446744073709551615, delegate = false},
  state = SERVICE_DEAD, deserialized_state = SERVICE_DEAD, main_exec_status = {start_timestamp = {realtime = 1576468742226267, monotonic = 6818611}, exit_timestamp = {realtime = 0,
      monotonic = 0}, pid = 995, code = 0, status = 0}, control_command = 0x0, main_command = 0x0, control_command_id = _SERVICE_EXEC_COMMAND_INVALID, exec_runtime = 0x0, main_pid = 0,
  control_pid = 0, socket_fd = -1, socket_fd_selinux_context_net = false, bus_endpoint_fd = -1, permissions_start_only = false, root_directory_start_only = false,
  remain_after_exit = false, guess_main_pid = true, result = SERVICE_SUCCESS, reload_result = SERVICE_SUCCESS, main_pid_known = false, main_pid_alien = false, bus_name_good = false,
  forbid_restart = false, start_timeout_defined = false, bus_name = 0x0, status_text = 0x0, status_errno = 0, start_limit = {interval = 10000000, begin = 0, burst = 5, num = 0},
  start_limit_action = EMERGENCY_ACTION_NONE, emergency_action = EMERGENCY_ACTION_NONE, reboot_arg = 0x0, accept_socket = {source = 0x0, target = 0x0, refs_by_target_next = 0x0,
    refs_by_target_prev = 0x0}, timer_event_source = 0x0, pid_file_pathspec = 0x0, notify_access = NOTIFY_MAIN, notify_state = NOTIFY_UNKNOWN, fd_store = 0x0, n_fd_store = 0,
  n_fd_store_max = 0}
(gdb) x/1000s 0xaaaafc34f5d0
0xaaaafc34f5d0: "/sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/s"...
0xaaaafc34f698: "ystemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/c"...
0xaaaafc34f760: "group/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//s"...
0xaaaafc34f828: "ys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/sys"...
0xaaaafc34f8f0: "temd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgroup/systemd//sys/fs/cgr"...

After installing debuginfo rpm from http://debuginfo.centos.org/7/aarch64/systemd-debuginfo-219-67.el7_7.2.aarch64.rpm, backtrace shows that s->meta.cgroup_path was tainted. Addresses before cgroup_path 0xaaaafc34f5d0 seems fine, It must be some wrong logic using this pointer.

I've been lost in there after some search into the code, it seems recursive invocation of join_path from src/shared/cgroup-util.c can be the only truth. But I can't figure out how.

Seriously, I need your help. Show me some light please.

thanks in advance.
TagsNo tags attached.
abrt_hash
URL

Activities

zyy

zyy

2020-01-08 07:35

reporter  

core.22474.tar.bz2 (783,459 bytes)

Issue History

Date Modified Username Field Change
2020-01-08 07:35 zyy New Issue
2020-01-08 07:35 zyy File Added: core.22474.tar.bz2