2017-06-24 03:35 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013136CentOS-7kernelpublic2017-04-18 10:39
Reporteradip 
PrioritynormalSeveritymajorReproducibilityunable to reproduce
StatusclosedResolutionunable to reproduce 
Product Version 
Target VersionFixed in Version 
Summary0013136: [3.10.0-514.10.2.el7.x86_64] PANIC: ".1BUG: unable to handle kernel NULL pointer dereference at 0000000000000014"
DescriptionThe server is a Dell PowerEdge R730xd running as active node of an active/passive Pacemaker Cluster.

See attached some crash information.
TagsNo tags attached.
abrt_hash
URL
Attached Files
  • ? file icon crash (50,229 bytes) 2017-04-16 00:20 -
    PID: 0      TASK: ffff8804fa559f60  CPU: 3   COMMAND: "swapper/3"
     #0 [ffff88086de43aa8] machine_kexec at ffffffff81059bdb
     #1 [ffff88086de43b08] __crash_kexec at ffffffff81105382
     #2 [ffff88086de43bd8] crash_kexec at ffffffff81105470
     #3 [ffff88086de43bf0] oops_end at ffffffff8168f3c8
     #4 [ffff88086de43c18] no_context at ffffffff8167efd6
     #5 [ffff88086de43c68] __bad_area_nosemaphore at ffffffff8167f06c
     #6 [ffff88086de43cb0] bad_area_nosemaphore at ffffffff8167f1d6
     #7 [ffff88086de43cc0] __do_page_fault at ffffffff8169215e
     #8 [ffff88086de43d20] do_page_fault at ffffffff81692305
     #9 [ffff88086de43d50] page_fault at ffffffff8168e5c8
        [exception RIP: account_system_time+21]
        RIP: ffffffff810c8375  RSP: ffff88086de43e00  RFLAGS: 00010086
        RAX: 0000000000000000  RBX: ffff88086de56c40  RCX: 00000000000f4240
        RDX: 00000000000f4240  RSI: 0000000000010000  RDI: 0000000000000000
        RBP: ffff88086de43e28   R8: 0000000000000000   R9: 00000000000c1af5
        R10: 000000003b9aca00  R11: 0000000000000000  R12: 00000000000f4240
        R13: 0000000000000000  R14: 0000000000000000  R15: ffff88086de4f9d8
        ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    #10 [ffff88086de43e30] account_process_tick at ffffffff810c8682
    #11 [ffff88086de43e68] update_process_times at ffffffff8109932d
    #12 [ffff88086de43e90] tick_sched_handle at ffffffff810f3145
    #13 [ffff88086de43eb0] tick_sched_timer at ffffffff810f31c1
    #14 [ffff88086de43ed8] __hrtimer_run_queues at ffffffff810b4a32
    #15 [ffff88086de43f30] hrtimer_interrupt at ffffffff810b4fd0
    #16 [ffff88086de43f80] local_apic_timer_interrupt at ffffffff81050fd7
    #17 [ffff88086de43f98] smp_apic_timer_interrupt at ffffffff8169920f
    #18 [ffff88086de43fb0] apic_timer_interrupt at ffffffff8169775d
    --- <IRQ stack> ---
    #19 [ffff8804fa683da8] apic_timer_interrupt at ffffffff8169775d
        [exception RIP: cpuidle_enter_state+82]
        RIP: ffffffff81514492  RSP: ffff8804fa683e50  RFLAGS: 00000206
        RAX: 0002d1882599f91b  RBX: 000000000000f8a0  RCX: 0000000000000018
        RDX: 0000000225c17d03  RSI: ffff8804fa683fd8  RDI: 0002d1882599f91b
        RBP: ffff8804fa683e78   R8: 00000000000070f7   R9: 000000000000001c
        R10: 000000000000c32d  R11: 0000000000000000  R12: ffff8804fa683e20
        R13: ffff88086de4f8e0  R14: 0000000000000082  R15: 0002d18825999180
        ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
    #20 [ffff8804fa683e80] cpuidle_idle_call at ffffffff815145d9
    #21 [ffff8804fa683ec0] arch_cpu_idle at ffffffff810350ee
    #22 [ffff8804fa683ed0] cpu_startup_entry at ffffffff810e7e65
    #23 [ffff8804fa683f28] start_secondary at ffffffff8104f07a
       PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
    >     0      0   0  ffffffff819c1460  RU   0.0       0      0  [swapper/0]
    >     0      0   1  ffff8804fa55edd0  RU   0.0       0      0  [swapper/1]
    >     0      0   2  ffff88017a1dedd0  RU   0.0       0      0  [swapper/2]
    >     0      0   3  ffff8804fa559f60  RU   0.0       0      0  [swapper/3]
    >     0      0   4  ffff88017a1d8fb0  RU   0.0       0      0  [swapper/4]
    >     0      0   5  ffff8804fa55de20  RU   0.0       0      0  [swapper/5]
    >     0      0   6  ffff88017a1dde20  RU   0.0       0      0  [swapper/6]
    >     0      0   7  ffff8804fa55af10  RU   0.0       0      0  [swapper/7]
    >     0      0   8  ffff88017a1d9f60  RU   0.0       0      0  [swapper/8]
    >     0      0   9  ffff8804fa55ce70  RU   0.0       0      0  [swapper/9]
    >     0      0  10  ffff88017a1dce70  RU   0.0       0      0  [swapper/10]
    >     0      0  11  ffff8804fa55bec0  RU   0.0       0      0  [swapper/11]
          1      0   1  ffff8804fa558000  IN   0.0   44056   4424  systemd
          2      0  11  ffff8804fa558fb0  IN   0.0       0      0  [kthreadd]
          3      2   0  ffff88017a1a8000  IN   0.0       0      0  [ksoftirqd/0]
          5      2   0  ffff88017a1a9f60  IN   0.0       0      0  [kworker/0:0H]
          8      2   0  ffff88017a1ace70  IN   0.0       0      0  [migration/0]
          9      2   0  ffff88017a1ade20  IN   0.0       0      0  [rcu_bh]
         10      2  11  ffff88017a1aedd0  IN   0.0       0      0  [rcu_sched]
         11      2   0  ffff88017a1d8000  IN   0.0       0      0  [watchdog/0]
         12      2   1  ffff8804f9d15e20  IN   0.0       0      0  [watchdog/1]
         13      2   1  ffff8804f9d14e70  IN   0.0       0      0  [migration/1]
         14      2   1  ffff8804f9d13ec0  IN   0.0       0      0  [ksoftirqd/1]
         16      2   1  ffff8804f9d11f60  IN   0.0       0      0  [kworker/1:0H]
         18      2   2  ffff880179ab0000  IN   0.0       0      0  [watchdog/2]
         19      2   2  ffff880179ab0fb0  IN   0.0       0      0  [migration/2]
         20      2   2  ffff880179ab1f60  IN   0.0       0      0  [ksoftirqd/2]
         22      2   2  ffff880179ab3ec0  IN   0.0       0      0  [kworker/2:0H]
         23      2   3  ffff8804f9bc0000  IN   0.0       0      0  [watchdog/3]
         24      2   3  ffff8804f9bc0fb0  IN   0.0       0      0  [migration/3]
         25      2   3  ffff8804f9bc1f60  IN   0.0       0      0  [ksoftirqd/3]
         27      2   3  ffff8804f9bc3ec0  IN   0.0       0      0  [kworker/3:0H]
         28      2   4  ffff880179ab6dd0  IN   0.0       0      0  [watchdog/4]
         29      2   4  ffff880179ab5e20  IN   0.0       0      0  [migration/4]
         30      2   4  ffff880179ab4e70  IN   0.0       0      0  [ksoftirqd/4]
         32      2   4  ffff880179b18fb0  IN   0.0       0      0  [kworker/4:0H]
         33      2   5  ffff8804f9bc6dd0  IN   0.0       0      0  [watchdog/5]
         34      2   5  ffff8804f9bc5e20  IN   0.0       0      0  [migration/5]
         35      2   5  ffff8804f9bc4e70  IN   0.0       0      0  [ksoftirqd/5]
         37      2   5  ffff8804f9408fb0  IN   0.0       0      0  [kworker/5:0H]
         38      2   6  ffff880179b1edd0  IN   0.0       0      0  [watchdog/6]
         39      2   6  ffff880179b1de20  IN   0.0       0      0  [migration/6]
         40      2   6  ffff880179b1ce70  IN   0.0       0      0  [ksoftirqd/6]
         43      2   7  ffff8804f940edd0  IN   0.0       0      0  [watchdog/7]
         44      2   7  ffff8804f940de20  IN   0.0       0      0  [migration/7]
         45      2   7  ffff8804f940ce70  IN   0.0       0      0  [ksoftirqd/7]
         47      2   7  ffff8804f940af10  IN   0.0       0      0  [kworker/7:0H]
         48      2   8  ffff880179b19f60  IN   0.0       0      0  [watchdog/8]
         49      2   8  ffff880179b88000  IN   0.0       0      0  [migration/8]
         50      2   8  ffff880179b88fb0  IN   0.0       0      0  [ksoftirqd/8]
         52      2   8  ffff880179b8af10  IN   0.0       0      0  [kworker/8:0H]
         53      2   9  ffff8804f9409f60  IN   0.0       0      0  [watchdog/9]
         54      2   9  ffff8804f9458000  IN   0.0       0      0  [migration/9]
         55      2   9  ffff8804f9458fb0  IN   0.0       0      0  [ksoftirqd/9]
         57      2   9  ffff8804f945af10  IN   0.0       0      0  [kworker/9:0H]
         58      2  10  ffff880179b8edd0  IN   0.0       0      0  [watchdog/10]
         59      2  10  ffff880179b8de20  IN   0.0       0      0  [migration/10]
         60      2  10  ffff880179b8ce70  IN   0.0       0      0  [ksoftirqd/10]
         62      2  10  ffff880179bc8000  IN   0.0       0      0  [kworker/10:0H]
         63      2  11  ffff8804f945edd0  IN   0.0       0      0  [watchdog/11]
         64      2  11  ffff8804f945de20  IN   0.0       0      0  [migration/11]
         65      2  11  ffff8804f945ce70  IN   0.0       0      0  [ksoftirqd/11]
         67      2  11  ffff8804f94b8000  IN   0.0       0      0  [kworker/11:0H]
         70      2   8  ffff880179bcce70  IN   0.0       0      0  [kdevtmpfs]
         71      2   3  ffff880179bcbec0  IN   0.0       0      0  [netns]
         72      2   1  ffff880179bcaf10  IN   0.0       0      0  [khungtaskd]
         73      2   3  ffff88046fc68000  IN   0.0       0      0  [writeback]
         74      2   3  ffff88046fc68fb0  IN   0.0       0      0  [kintegrityd]
         75      2   3  ffff88046fc69f60  IN   0.0       0      0  [bioset]
         76      2   3  ffff88046fc6af10  IN   0.0       0      0  [kblockd]
         77      2   3  ffff88046fc6bec0  IN   0.0       0      0  [md]
         86      2   0  ffff88086eb7de20  IN   0.0       0      0  [kswapd0]
         87      2   3  ffff88086eb7edd0  IN   0.0       0      0  [kswapd1]
         88      2   3  ffff88046fc6edd0  IN   0.0       0      0  [ksmd]
         89      2   7  ffff88046fc6de20  IN   0.0       0      0  [khugepaged]
         90      2  11  ffff88046fc6ce70  IN   0.0       0      0  [fsnotify_mark]
         91      2   3  ffff88046cf08000  IN   0.0       0      0  [crypto]
         99      2   3  ffff88046cf30000  IN   0.0       0      0  [kthrotld]
        101      2   3  ffff88046cf31f60  IN   0.0       0      0  [kmpath_rdacd]
        104      2   6  ffff88046cf34e70  IN   0.0       0      0  [kpsmoused]
        105      2   6  ffff88046cf35e20  IN   0.0       0      0  [ipv6_addrconf]
        125      2   3  ffff88046c8d5e20  IN   0.0       0      0  [deferwq]
        179      2  10  ffff88046cbe8fb0  IN   0.0       0      0  [kauditd]
        365      2   5  ffff88086c352f10  IN   0.0       0      0  [scsi_eh_0]
        366      2   8  ffff88086c353ec0  IN   0.0       0      0  [scsi_tmf_0]
        367      2   7  ffff88086ce65e20  IN   0.0       0      0  [ata_sff]
        368      2   7  ffff88086c116dd0  IN   0.0       0      0  [scsi_eh_1]
        369      2   7  ffff88086c084e70  IN   0.0       0      0  [scsi_tmf_1]
        370      2   0  ffff88086c113ec0  IN   0.0       0      0  [usb-storage]
        379      2   5  ffff88086c114e70  IN   0.0       0      0  [ixgbe]
        381      2  11  ffff88086c7dde20  IN   0.0       0      0  [scsi_eh_2]
        382      2   0  ffff88086c7d9f60  IN   0.0       0      0  [scsi_tmf_2]
        383      2   2  ffff88086c7daf10  IN   0.0       0      0  [scsi_eh_3]
        384      2   8  ffff88086c115e20  IN   0.0       0      0  [scsi_tmf_3]
        385      2  11  ffff88086c111f60  IN   0.0       0      0  [scsi_eh_4]
        386      2   8  ffff88086c110fb0  IN   0.0       0      0  [scsi_tmf_4]
        387      2   1  ffff88086c7dbec0  IN   0.0       0      0  [scsi_eh_5]
        388      2   1  ffff88086c7d8fb0  IN   0.0       0      0  [scsi_tmf_5]
        393      2   2  ffff88086ce63ec0  IN   0.0       0      0  [scsi_eh_6]
        394      2   2  ffff88086ce62f10  IN   0.0       0      0  [scsi_tmf_6]
        396      2   2  ffff88086ce61f60  IN   0.0       0      0  [scsi_eh_7]
        397      2   9  ffff88086c081f60  IN   0.0       0      0  [scsi_tmf_7]
        398      2   9  ffff88086c085e20  IN   0.0       0      0  [scsi_eh_8]
        399      2   4  ffff88086c086dd0  IN   0.0       0      0  [scsi_tmf_8]
        401      2  11  ffff88086c082f10  IN   0.0       0      0  [scsi_eh_9]
        402      2   9  ffff88086a4a0000  IN   0.0       0      0  [scsi_tmf_9]
        403      2   9  ffff88086a4a0fb0  IN   0.0       0      0  [scsi_eh_10]
        405      2   7  ffff88086a4a1f60  IN   0.0       0      0  [scsi_tmf_10]
        406      2   4  ffff88086a4a2f10  IN   0.0       0      0  [scsi_eh_11]
        407      2   4  ffff88086a4a3ec0  IN   0.0       0      0  [scsi_tmf_11]
        541      2   2  ffff88086ce60fb0  IN   0.0       0      0  [kdmflush]
        542      2   2  ffff88086ce60000  IN   0.0       0      0  [bioset]
        551      2   2  ffff88086a4a6dd0  IN   0.0       0      0  [kdmflush]
        552      2   2  ffff88086a4a5e20  IN   0.0       0      0  [bioset]
        567      2   2  ffff88086a6b8000  IN   0.0       0      0  [xfsalloc]
        568      2   2  ffff88086a6b8fb0  IN   0.0       0      0  [xfs_mru_cache]
        569      2   2  ffff88086a6b9f60  IN   0.0       0      0  [xfs-buf/dm-0]
        570      2   2  ffff88086a6baf10  IN   0.0       0      0  [xfs-data/dm-0]
        571      2   2  ffff88086a6bbec0  IN   0.0       0      0  [xfs-conv/dm-0]
        572      2   2  ffff88086a6bce70  IN   0.0       0      0  [xfs-cil/dm-0]
        573      2   2  ffff88086a6bde20  IN   0.0       0      0  [xfs-reclaim/dm-]
        574      2   2  ffff88086a6bedd0  IN   0.0       0      0  [xfs-log/dm-0]
        575      2   2  ffff88086a6c8000  IN   0.0       0      0  [xfs-eofblocks/d]
        576      2   5  ffff88086a6c8fb0  IN   0.0       0      0  [xfsaild/dm-0]
        633      2   0  ffff88046b198000  IN   0.0       0      0  [kworker/0:1H]
        647      2  11  ffff88086af86dd0  IN   0.0       0      0  [kworker/11:1H]
        655      1   1  ffff88046c0cce70  IN   0.0  126644   1484  systemd-journal
        677      2   2  ffff88086af83ec0  IN   0.0       0      0  [rpciod]
        679      1   3  ffff88046c8d4e70  IN   0.0   43528   1816  systemd-udevd
       1511      2   5  ffff88086ce86dd0  IN   0.0       0      0  [edac-poller]
       2087      2   2  ffff88046cf0bec0  IN   0.0       0      0  [dm_bufio_cache]
       2089      2   2  ffff88046cf0af10  IN   0.0       0      0  [kdmflush]
       2090      2   2  ffff88046cf0ce70  IN   0.0       0      0  [bioset]
       2092      2   9  ffff88046cf0de20  IN   0.0       0      0  [kdmflush]
       2093      2   2  ffff88046cf08fb0  IN   0.0       0      0  [bioset]
       2095      2   2  ffff88086b68af10  IN   0.0       0      0  [kdmflush]
       2096      2   2  ffff88086b68ce70  IN   0.0       0      0  [bioset]
       2105      2   2  ffff88086b68edd0  IN   0.0       0      0  [kdmflush]
       2106      2   2  ffff88086b689f60  IN   0.0       0      0  [bioset]
       2107      2   2  ffff88086b68de20  IN   0.0       0      0  [kcopyd]
       2108      2   2  ffff88086b68bec0  IN   0.0       0      0  [bioset]
       2109      2   2  ffff88086b688fb0  IN   0.0       0      0  [dm-cache]
       2110      2   9  ffff88086b688000  IN   0.0       0      0  [bioset]
       2111      2   6  ffff88046cf09f60  IN   0.0       0      0  [kworker/6:1H]
       2123      2   4  ffff88046c088000  IN   0.0       0      0  [kworker/4:1H]
       2129      2   2  ffff88086eb7bec0  IN   0.0       0      0  [xfs-buf/sdd1]
       2130      2   2  ffff88086eb79f60  IN   0.0       0      0  [xfs-data/sdd1]
       2131      2   2  ffff88086eb7af10  IN   0.0       0      0  [xfs-conv/sdd1]
       2132      2   2  ffff8804f9738000  IN   0.0       0      0  [xfs-cil/sdd1]
       2133      2   2  ffff8804f973de20  IN   0.0       0      0  [xfs-buf/sda1]
       2134      2   2  ffff8804f9738fb0  IN   0.0       0      0  [xfs-reclaim/sdd]
       2136      2   1  ffff8804f973ce70  IN   0.0       0      0  [xfs-data/sda1]
       2137      2   2  ffff8804f9736dd0  IN   0.0       0      0  [xfs-log/sdd1]
       2138      2   2  ffff8804f9733ec0  IN   0.0       0      0  [xfs-conv/sda1]
       2139      2   2  ffff88046b694e70  IN   0.0       0      0  [xfs-eofblocks/s]
       2140      2   3  ffff88046b690000  IN   0.0       0      0  [xfs-cil/sda1]
       2141      2   3  ffff88046b693ec0  IN   0.0       0      0  [xfs-reclaim/sda]
       2142      2   2  ffff88046b690fb0  IN   0.0       0      0  [xfs-log/sda1]
       2143      2   2  ffff88046b692f10  IN   0.0       0      0  [xfs-eofblocks/s]
       2144      2   5  ffff88087f768fb0  IN   0.0       0      0  [xfsaild/sda1]
       2150      2   1  ffff88087f76bec0  IN   0.0       0      0  [xfsaild/sdd1]
       2167      1   1  ffff8800364a4e70  IN   0.0   55416   1836  auditd
       2171      1   5  ffff88046e3cde20  IN   0.0   55416   1836  auditd
       2188      1   0  ffff88087f769f60  IN   0.0   24420   1676  dbus-daemon
       2193      1   0  ffff88046e3cbec0  IN   0.1  637348  38052  pcsd
       2194      1   6  ffff88046e3c8fb0  IN   0.0   24192   1692  systemd-logind
       2196      1   0  ffff88046e3cce70  IN   0.0  128488   2960  smartd
       2209      1   0  ffff88087fe92f10  IN   0.0   12132   3832  haveged
       2212      1   8  ffff88087fe90fb0  IN   0.0  527516  14188  polkitd
       2219      1  11  ffff88046c2a4e70  IN   0.0  201212   1372  gssproxy
       2228      1   6  ffff88086af80000  IN   0.0  527516  14188  gmain
       2229      1   5  ffff88046c2a1f60  IN   0.0  201212   1372  gssproxy
       2230      1   7  ffff88046c2a0fb0  IN   0.0  201212   1372  gssproxy
       2231      1   0  ffff88046c2a6dd0  IN   0.0  201212   1372  gssproxy
       2232      1   5  ffff88046c2a5e20  IN   0.0  201212   1372  gssproxy
       2233      1   5  ffff88046c2a2f10  IN   0.0  201212   1372  gssproxy
       2235      1  10  ffff88046b68bec0  IN   0.0  527516  14188  gdbus
       2239      1   3  ffff88046e3c8000  IN   0.0  527516  14188  JS GC Helper
       2240      1   6  ffff88046e3c9f60  IN   0.0  527516  14188  JS Sour~ Thread
       2242      1   3  ffff88046c08edd0  IN   0.0  126224   1572  crond
       2245      1   7  ffff88046e3cedd0  IN   0.0  527516  14188  runaway-killer-
       2260      1   0  ffff88086af81f60  IN   0.0   44644   2368  ntpd
       2389      2   2  ffff8804f9739f60  IN   0.0       0      0  [bond0]
       2403      2   2  ffff8800364a2f10  IN   0.0       0      0  [bond1]
       2465      1   9  ffff88046b68ce70  IN   0.1  637348  38052  pcsd
       2466      1   8  ffff88046b68de20  IN   0.1  637348  38052  pcsd
       2584      2   2  ffff8800364a1f60  IN   0.0       0      0  [bond2]
       2876      1   5  ffff88046c3b5e20  IN   0.1  553152  18748  tuned
       2879      1   4  ffff88087f76edd0  IN   0.0   44452   4008  tgtd
       2918      1   8  ffff88046c08af10  IN   0.0   82524   1344  sshd
       2987      1   5  ffff88086a6cce70  IN   0.0  110036    844  agetty
       3001      1   0  ffff88046b691f60  IN   0.0   22800    744  ipmievd
       3153      1   1  ffff88046b68edd0  IN   0.0  191692  13884  corosync
       3157      1   4  ffff88046c3b4e70  IN   0.0  191692  13884  corosync
       3172      1   3  ffff88046c241f60  IN   0.1  553152  18748  gmain
       3180      1   3  ffff88046c240fb0  IN   0.1  553152  18748  tuned
       3183      1   4  ffff88046c245e20  IN   0.1  553152  18748  tuned
       3188      1   9  ffff880469fa0fb0  IN   0.1  553152  18748  tuned
       3288      1   8  ffff88086defce70  IN   0.0  155704   9472  munin-node
       3839      1   5  ffff88087f851f60  IN   0.0   89036   2172  master
       3882   3839   1  ffff88086879de20  IN   0.0   89208   4052  qmgr
       4124      1   9  ffff88046e3fde20  IN   0.0  132632   8196  pacemakerd
       4250   4124  10  ffff8808673e0fb0  IN   0.0  137084  12724  cib
       4253   4124   2  ffff8808673e1f60  IN   0.0  136004   7904  stonithd
       4254   4124   3  ffff8808673e6dd0  IN   0.0  105188   4988  lrmd
       4255   4124   3  ffff88086a4a4e70  IN   0.0  126920   7508  attrd
       4256   4124   0  ffff88086df70fb0  IN   0.0  117040   4628  pengine
       4258   4124  11  ffff88086df76dd0  IN   0.0  145332   9260  crmd
       4335      2   3  ffff8808673e5e20  IN   0.0       0      0  [kworker/3:1H]
       4340      2   1  ffff8804f9731f60  IN   0.0       0      0  [kworker/1:1H]
       4589      1   0  ffff88086b641f60  IN   0.1 1070688  52756  dsm_sa_datamgrd
       4624      2   8  ffff88046c2a0000  IN   0.0       0      0  [kworker/8:1H]
       4625      2  10  ffff88046c2a3ec0  IN   0.0       0      0  [kworker/10:1H]
       4741      2   9  ffff8804f94bce70  IN   0.0       0      0  [kworker/9:1H]
       5002      1   5  ffff88087f853ec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5003      1   0  ffff88085e2ebec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5004      1  11  ffff88046c913ec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5008      1   2  ffff88085e2e8fb0  IN   0.1  637348  38052  pcsd
       5326      1   5  ffff88086ce2de20  IN   0.0   64948   1444  rpcbind
       5407      2   9  ffff88087fe94e70  IN   0.0       0      0  [bioset]
       5412      2   2  ffff88087fe95e20  IN   0.0       0      0  [drbd-reissue]
       5438      2   9  ffff88087fe93ec0  IN   0.0       0      0  [drbd1_submit]
       5443      2   9  ffff88087fe90000  IN   0.0       0      0  [drbd0_submit]
       5452      2   0  ffff88046c244e70  IN   0.0       0      0  [drbd_w_r1_DRBDL]
       5455      2   1  ffff88046c242f10  IN   0.0       0      0  [drbd_w_r0_sdb]
       5470      2   0  ffff88046c240000  IN   0.0       0      0  [drbd_r_r1_DRBDL]
       5475      2   1  ffff88046c246dd0  IN   0.0       0      0  [drbd_r_r0_sdb]
       5616      1  11  ffff880869f86dd0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5617      1   4  ffff880869f80000  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5618      1   4  ffff880869f81f60  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5619      1   0  ffff880869f83ec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5620      1   2  ffff880869f84e70  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5622      1   2  ffff88085e2eedd0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5623      1   4  ffff88085c72edd0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5643      1   9  ffff88085c728fb0  IN   0.0  300504   4060  dsm_sa_eventmgr
       5644      1   0  ffff88046c914e70  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5645      1   6  ffff88086ce29f60  IN   0.0  300504   4060  dsm_sa_eventmgr
       5646      1   0  ffff88086ce2ce70  IN   0.0  300504   4060  dsm_sa_eventmgr
       5670      1  10  ffff88046c911f60  IN   0.0  510900   9948  dsm_sa_snmpd
       5671      1   4  ffff88085c72de20  IN   0.0  510900   9948  dsm_sa_snmpd
       5672      1   9  ffff88086ce28fb0  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5673      1  11  ffff88003645ce70  IN   0.0  510900   9948  dsm_sa_snmpd
       5674      1   5  ffff88003645de20  IN   0.0  510900   9948  dsm_sa_snmpd
       5677   5003   0  ffff88046c915e20  IN   0.1  725452  18484  dsm_sa_datamgrd
       5678      1   0  ffff88046c912f10  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5681      1   0  ffff88085e2e8000  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5695      1   8  ffff88046b1d8000  IN   0.0  510900   9948  dsm_sa_snmpd
       5696      1   1  ffff88046b1d8fb0  IN   0.0  510900   9948  dsm_sa_snmpd
       5772      1   0  ffff88086ce28000  IN   0.0  137908    880  dsm_om_connsvcd
       5773      1   1  ffff88087f76ce70  IN   0.0  632408   4548  dsm_om_shrsvcd
       5774   5772   2  ffff880469619f60  IN   0.5 4165768 186360  dsm_om_connsvcd
       5775      1   1  ffff88086ce2bec0  IN   0.0  632408   4548  dsm_om_shrsvcd
       5776   5772   6  ffff88046c910000  IN   0.5 4165768 186360  dsm_om_connsvcd
       5783      1  10  ffff88046a658000  IN   0.0  632408   4548  dsm_om_shrsvcd
       5786      1   2  ffff88046a65de20  IN   0.0  632408   4548  dsm_om_shrsvcd
       5787      1  10  ffff88086c341f60  IN   0.0  632408   4548  dsm_om_shrsvcd
       5788      1   6  ffff88046a65edd0  IN   0.0  632408   4548  dsm_om_shrsvcd
       5789      1   2  ffff88046a65ce70  IN   0.0  632408   4548  dsm_om_shrsvcd
       5790      1   2  ffff88046a65af10  IN   0.0  632408   4548  dsm_om_shrsvcd
       5876   5772  10  ffff88086af26dd0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5877   5772   6  ffff88086af23ec0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5878   5772   0  ffff88086af25e20  IN   0.5 4165768 186360  dsm_om_connsvcd
       5879   5772   3  ffff88086af20000  IN   0.5 4165768 186360  dsm_om_connsvcd
       5880   5772   2  ffff88086af24e70  IN   0.5 4165768 186360  dsm_om_connsvcd
       5881   5772   7  ffff88086af21f60  IN   0.5 4165768 186360  dsm_om_connsvcd
       5882   5772   8  ffff88086af22f10  IN   0.5 4165768 186360  dsm_om_connsvcd
       5883   5772   1  ffff88085c729f60  IN   0.5 4165768 186360  dsm_om_connsvcd
       5885   5772   9  ffff88085c72bec0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5886   5772   4  ffff880869f85e20  IN   0.5 4165768 186360  dsm_om_connsvcd
       5908   5772   7  ffff88046b1dce70  IN   0.5 4165768 186360  dsm_om_connsvcd
       5909   5772   0  ffff88046b1dedd0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5910   5772   8  ffff88046b1d9f60  IN   0.5 4165768 186360  dsm_om_connsvcd
       5911   5772   3  ffff88046b1dde20  IN   0.5 4165768 186360  dsm_om_connsvcd
       5912   5772   9  ffff88046b1dbec0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5913   5772   5  ffff88046b1daf10  IN   0.5 4165768 186360  dsm_om_connsvcd
       5914   5772   9  ffff88046c03edd0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5915   5772  11  ffff88046c03ce70  IN   0.5 4165768 186360  dsm_om_connsvcd
       5916   5772   5  ffff88046c03bec0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5918   5772   9  ffff88046c03af10  IN   0.5 4165768 186360  dsm_om_connsvcd
       5921      1   5  ffff88046e3f8000  IN   0.1 1070688  52756  dsm_sa_datamgrd
       5922   5772  10  ffff8800367b4e70  IN   0.5 4165768 186360  dsm_om_connsvcd
       5923   5772   2  ffff8800367b3ec0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5925   5772  11  ffff8800367b0fb0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5926   5772   6  ffff8800367b6dd0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5927   5772   1  ffff880866ec6dd0  IN   0.5 4165768 186360  dsm_om_connsvcd
       5940   5772   8  ffff88046a659f60  IN   0.5 4165768 186360  dsm_om_connsvcd
       5944      2   7  ffff880866ec4e70  IN   0.0       0      0  [kworker/7:1H]
       6069   5772   1  ffff88086a6caf10  IN   0.5 4165768 186360  dsm_om_connsvcd
       6274   5772   3  ffff88046c0cbec0  IN   0.5 4165768 186360  dsm_om_connsvcd
       6707      2   2  ffff88087f76af10  IN   0.0       0      0  [xfs-buf/drbd1]
       6708      2   2  ffff88003645edd0  IN   0.0       0      0  [xfs-data/drbd1]
       6709      2   2  ffff88003645bec0  IN   0.0       0      0  [xfs-conv/drbd1]
       6710      2   2  ffff880036459f60  IN   0.0       0      0  [xfs-cil/drbd1]
       6711      2   2  ffff880036458fb0  IN   0.0       0      0  [xfs-reclaim/drb]
       6712      2   4  ffff88087f84bec0  IN   0.0       0      0  [xfs-log/drbd1]
       6713      2   4  ffff88087f84edd0  IN   0.0       0      0  [xfs-eofblocks/d]
       6714      2   3  ffff88087f849f60  IN   0.0       0      0  [xfsaild/drbd1]
       6857      1   5  ffff88046b688fb0  IN   0.0   42484   1868  rpc.statd
       6872      1  10  ffff88046b68af10  IN   0.0   17148    628  rpc.idmapd
       6873      1   5  ffff88086879af10  IN   0.0   41024   2252  rpc.mountd
       6885      2   5  ffff88087f84de20  IN   0.0       0      0  [nfsd4_callbacks]
       6886      2   5  ffff88087f848000  IN   0.0       0      0  [lockd]
       6894      2   2  ffff88087f848fb0  IN   0.0       0      0  [nfsd]
       6895      2   6  ffff88087f84af10  IN   0.0       0      0  [nfsd]
       6896      2   5  ffff88087f84ce70  IN   0.0       0      0  [nfsd]
       6897      2   0  ffff88086b7d1f60  IN   0.0       0      0  [nfsd]
       6898      2   0  ffff88086b7d3ec0  IN   0.0       0      0  [nfsd]
       6899      2  11  ffff88086b7d0fb0  IN   0.0       0      0  [nfsd]
       6900      2   5  ffff88086b7d2f10  IN   0.0       0      0  [nfsd]
       6901      2   1  ffff88086b7d5e20  IN   0.0       0      0  [nfsd]
       6902      2   1  ffff88086b7d0000  IN   0.0       0      0  [nfsd]
       6903      2   3  ffff88086b7d4e70  IN   0.0       0      0  [nfsd]
       6904      2  10  ffff88086b7d6dd0  IN   0.0       0      0  [nfsd]
       6905      2   9  ffff88086defaf10  IN   0.0       0      0  [nfsd]
       6906      2   6  ffff88086defde20  IN   0.0       0      0  [nfsd]
       6907      2   6  ffff88086defbec0  IN   0.0       0      0  [nfsd]
       6908      2  11  ffff88086def8000  IN   0.0       0      0  [nfsd]
       6909      2   6  ffff88086def8fb0  IN   0.0       0      0  [nfsd]
       6910      2   2  ffff88086defedd0  IN   0.0       0      0  [nfsd]
       6911      2   0  ffff88086def9f60  IN   0.0       0      0  [nfsd]
       6912      2   2  ffff88086c083ec0  IN   0.0       0      0  [nfsd]
       6913      2  11  ffff88086c080000  IN   0.0       0      0  [nfsd]
       6914      2   5  ffff88086c112f10  IN   0.0       0      0  [nfsd]
       6915      2  10  ffff88086c110000  IN   0.0       0      0  [nfsd]
       6916      2   8  ffff8804f973edd0  IN   0.0       0      0  [nfsd]
       6917      2   2  ffff8804f973af10  IN   0.0       0      0  [nfsd]
       6918      2   1  ffff88086df74e70  IN   0.0       0      0  [nfsd]
       6919      2   0  ffff88086df73ec0  IN   0.0       0      0  [nfsd]
       6920      2   7  ffff88086df72f10  IN   0.0       0      0  [nfsd]
       6921      2   8  ffff88086df70000  IN   0.0       0      0  [nfsd]
       6922      2   7  ffff88086df71f60  IN   0.0       0      0  [nfsd]
       6923      2   3  ffff88086df75e20  IN   0.0       0      0  [nfsd]
       6924      2   4  ffff88087f856dd0  IN   0.0       0      0  [nfsd]
       6925      2  10  ffff88087f852f10  IN   0.0       0      0  [nfsd]
       6926      2  11  ffff88087f850fb0  IN   0.0       0      0  [nfsd]
       6927      2  11  ffff88087f855e20  IN   0.0       0      0  [nfsd]
       6928      2   4  ffff88087f850000  IN   0.0       0      0  [nfsd]
       6929      2   3  ffff88087f854e70  IN   0.0       0      0  [nfsd]
       6931      2   4  ffff8808673e2f10  IN   0.0       0      0  [nfsd]
       6932      2  11  ffff8808673e0000  IN   0.0       0      0  [nfsd]
       6933      2   5  ffff88086b643ec0  IN   0.0       0      0  [nfsd]
       6934      2   1  ffff88086b640fb0  IN   0.0       0      0  [nfsd]
       6935      2  11  ffff88086b644e70  IN   0.0       0      0  [nfsd]
       6936      2   0  ffff88086b645e20  IN   0.0       0      0  [nfsd]
       6937      2   7  ffff88086b642f10  IN   0.0       0      0  [nfsd]
       6938      2   8  ffff88086b646dd0  IN   0.0       0      0  [nfsd]
       6939      2   2  ffff88086b640000  IN   0.0       0      0  [nfsd]
       6940      2   2  ffff88085c72af10  IN   0.0       0      0  [nfsd]
       6941      2   0  ffff88085c728000  IN   0.0       0      0  [nfsd]
       6942      2   5  ffff88083a2a0000  IN   0.0       0      0  [nfsd]
       6943      2   6  ffff88083a2a0fb0  IN   0.0       0      0  [nfsd]
       6944      2   0  ffff88083a2a1f60  IN   0.0       0      0  [nfsd]
       6945      2   0  ffff88083a2a2f10  IN   0.0       0      0  [nfsd]
       6946      2   1  ffff88083a2a3ec0  IN   0.0       0      0  [nfsd]
       6947      2   1  ffff88083a2a4e70  IN   0.0       0      0  [nfsd]
       6948      2   0  ffff88083a2a5e20  IN   0.0       0      0  [nfsd]
       6949      2   0  ffff88083a2a6dd0  IN   0.0       0      0  [nfsd]
       6950      2   8  ffff8808486d8000  IN   0.0       0      0  [nfsd]
       6951      2  10  ffff8808486d8fb0  IN   0.0       0      0  [nfsd]
       6952      2   0  ffff8808486d9f60  IN   0.0       0      0  [nfsd]
       6953      2  11  ffff8808486daf10  IN   0.0       0      0  [nfsd]
       6954      2   1  ffff8808486dbec0  IN   0.0       0      0  [nfsd]
       6955      2   8  ffff8808486dce70  IN   0.0       0      0  [nfsd]
       6956      2  10  ffff8808486dde20  IN   0.0       0      0  [nfsd]
       6957      2   8  ffff8808486dedd0  IN   0.0       0      0  [nfsd]
       6958      2   0  ffff88085b250000  IN   0.0       0      0  [nfsd]
       7209   5772   7  ffff88046c0c9f60  IN   0.5 4165768 186360  dsm_om_connsvcd
       7210   5772   2  ffff88046c0c8000  IN   0.5 4165768 186360  dsm_om_connsvcd
       7211   5772   7  ffff88046c0caf10  IN   0.5 4165768 186360  dsm_om_connsvcd
       7212   5772  10  ffff88046c0cedd0  IN   0.5 4165768 186360  dsm_om_connsvcd
       7836      2   0  ffff880834341f60  IN   0.0       0      0  [drbd_a_r1_DRBDL]
       7837      2   2  ffff880834343ec0  IN   0.0       0      0  [drbd_as_r1_DRBD]
       7838      2   1  ffff880834345e20  IN   0.0       0      0  [drbd_a_r0_sdb]
       7839      2   2  ffff880834340000  IN   0.0       0      0  [drbd_as_r0_sdb]
       8131      2   6  ffff88046c243ec0  IN   0.0       0      0  [kworker/6:2H]
      19465      2  10  ffff880179bc9f60  IN   0.0       0      0  [kworker/10:2]
      21159      2   0  ffff8800364a6dd0  IN   0.0       0      0  [kworker/0:2]
      29408      2  11  ffff880868798fb0  IN   0.0       0      0  [kworker/11:2]
      33141      2   1  ffff8804f9d12f10  IN   0.0       0      0  [kworker/u386:0]
      59848      2   9  ffff880866ec2f10  IN   0.0       0      0  [kworker/9:0]
      67321      2   2  ffff88017a1abec0  IN   0.0       0      0  [kworker/2:1]
      67376      2   8  ffff88046cbeedd0  IN   0.0       0      0  [kworker/u385:2]
      67444      2   2  ffff8803c8f31f60  IN   0.0       0      0  [kworker/2:0]
      75178      2   1  ffff88086ce64e70  IN   0.0       0      0  [kworker/1:1]
      78254      2   5  ffff88086a44edd0  IN   0.0       0      0  [kworker/u386:1]
      81689      2   2  ffff88046b19de20  IN   0.0       0      0  [kworker/u385:1]
      82613      2   5  ffff88083123de20  IN   0.0       0      0  [kworker/5:1H]
      86454      2   1  ffff88046b19af10  IN   0.0       0      0  [kworker/u384:0]
      90260      2   5  ffff88086af20fb0  IN   0.0       0      0  [kworker/5:0]
      90327      2   7  ffff8804f94b9f60  IN   0.0       0      0  [kworker/7:0]
      96058      2   6  ffff88046cbeaf10  IN   0.0       0      0  [kworker/6:0]
      97270      2   4  ffff8803d3e2bec0  IN   0.0       0      0  [kworker/4:0]
      109557      2   9  ffff880831238000  IN   0.0       0      0  [kworker/9:2]
      112303      1   8  ffff88086a44af10  IN   0.0  224196  13140  snmpd
      113057      2   6  ffff8803e5af3ec0  IN   0.0       0      0  [kworker/6:1]
      113148      2   8  ffff8803e5af2f10  IN   0.0       0      0  [kworker/8:2]
      113161      2  10  ffff88046cbede20  IN   0.0       0      0  [kworker/10:0]
      114025   3839   3  ffff8803c8f32f10  IN   0.0   89140   4016  pickup
      120604      2   4  ffff88046c8d3ec0  IN   0.0       0      0  [kworker/4:2]
      121449      1   4  ffff880866ec0fb0  IN   0.0  419172   4852  rsyslogd
      121451      1  11  ffff88046a658fb0  IN   0.0  419172   4852  in:imjournal
      121452      1   9  ffff88046a65bec0  IN   0.0  419172   4852  rs:main Q:Reg
      125164      2   1  ffff880868a6edd0  IN   0.0       0      0  [kworker/u384:2]
      126017      2   8  ffff88046cf0edd0  IN   0.0       0      0  [kworker/8:1]
      128263      2   6  ffff8803c8f30000  IN   0.0       0      0  [kworker/6:2]
      128296      2   8  ffff8803c8f33ec0  IN   0.0       0      0  [kworker/8:0]
      129574      2   1  ffff880866ec5e20  IN   0.0       0      0  [kworker/1:0]
      131642      1   2  ffff8803dc384e70  IN   0.1  637348  38052  ruby-timer-thr
      132408      2   7  ffff88085c72ce70  IN   0.0       0      0  [kworker/7:2]
      141714      2   3  ffff88086a44ce70  IN   0.0       0      0  [kworker/3:0]
      155195      2   3  ffff88086c7dce70  IN   0.0       0      0  [kworker/3:1]
      155224      2   5  ffff88086c7d8000  IN   0.0       0      0  [kworker/5:2]
      155284      2  11  ffff8804f940bec0  IN   0.0       0      0  [kworker/11:1]
      157720      2   2  ffff880463f3af10  IN   0.0       0      0  [kworker/2:1H]
      170418      2   0  ffff8803dc383ec0  IN   0.0       0      0  [kworker/0:1]
      183496      2   1  ffff88085b253ec0  IN   0.0       0      0  [kworker/u384:1]
    PID: 0      TASK: ffff8804fa559f60  CPU: 3   COMMAND: "swapper/3"
    struct task_struct {
      state = 0, 
      stack = 0xffff8804fa680000, 
      usage = {
        counter = 2
      }, 
      flags = 270540864, 
      ptrace = 0, 
      wake_entry = {
        next = 0x0
      }, 
      on_cpu = 1, 
      last_wakee = 0xffff88087f849f60, 
      wakee_flips = 2, 
      wakee_flip_decay_ts = 5087999619, 
      wake_cpu = 3, 
      on_rq = 0, 
      prio = 120, 
      static_prio = 120, 
      normal_prio = 120, 
      rt_priority = 0, 
      sched_class = 0xffffffff816bb520 <idle_sched_class>, 
      se = {
        load = {
          weight = 1024, 
          inv_weight = 4194304
        }, 
        run_node = {
          __rb_parent_color = 1, 
          rb_right = 0x0, 
          rb_left = 0x0
        }, 
        group_node = {
          next = 0xffff8804fa559ff0, 
          prev = 0xffff8804fa559ff0
        }, 
        on_rq = 0, 
        exec_start = 285006861, 
        sum_exec_runtime = 0, 
        vruntime = 0, 
        prev_sum_exec_runtime = 0, 
        nr_migrations = 0, 
        parent = 0x0, 
        cfs_rq = 0xffff88086de56cc0, 
        my_q = 0x0, 
        avg = {
          runnable_avg_sum = 19159, 
          runnable_avg_period = 19207, 
          last_runnable_update = 192115309, 
          decay_count = 184, 
          load_avg_contrib = 1021
        }, 
        {
          statistics = 0xffff8804fa55ae30, 
          __UNIQUE_ID_rh_kabi_hide10 = {
            rh_reserved1 = 18446612153694006832
          }, 
          {<No data fields>}
        }, 
        rh_reserved2 = 0, 
        rh_reserved3 = 0, 
        rh_reserved4 = 0
      }, 
      rt = {
        run_list = {
          next = 0xffff8804fa55a088, 
          prev = 0xffff8804fa55a088
        }, 
        timeout = 0, 
        watchdog_stamp = 0, 
        time_slice = 100, 
        back = 0x0, 
        parent = 0x0, 
        rt_rq = 0xffff88086de56dd0, 
        my_q = 0x0
      }, 
      sched_task_group = 0xffffffff81e3d480 <root_task_group>, 
      preempt_notifiers = {
        first = 0x0
      }, 
      fpu_counter = 120 'x', 
      btrace_seq = 0, 
      policy = 0, 
      nr_cpus_allowed = 1, 
      cpus_allowed = {
        bits = {8, 0, 0, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615}
      }, 
      sched_info = {
        pcount = 0, 
        run_delay = 0, 
        last_arrival = 0, 
        last_queued = 0
      }, 
      tasks = {
        next = 0xffff8804fa5593e0, 
        prev = 0xffffffff819c1890 <init_task+1072>
      }, 
      pushable_tasks = {
        prio = 140, 
        prio_list = {
          next = 0xffff8804fa55a3a8, 
          prev = 0xffff8804fa55a3a8
        }, 
        node_list = {
          next = 0xffff8804fa55a3b8, 
          prev = 0xffff8804fa55a3b8
        }
      }, 
      mm = 0x0, 
      active_mm = 0xffff88046cbd4b00, 
      rss_stat = {
        events = 0, 
        count = {0, 0, 0}
      }, 
      exit_state = 0, 
      exit_code = 0, 
      exit_signal = 0, 
      pdeath_signal = 0, 
      jobctl = 0, 
      personality = 0, 
      did_exec = 0, 
      in_execve = 0, 
      in_iowait = 0, 
      no_new_privs = 0, 
      sched_reset_on_fork = 0, 
      sched_contributes_to_load = 1, 
      pid = 0, 
      tgid = 0, 
      stack_canary = 12140910610751055461, 
      real_parent = 0xffff8804fa558000, 
      parent = 0xffffffff819c1460 <init_task>, 
      children = {
        next = 0xffff8804fa55a428, 
        prev = 0xffff8804fa55a428
      }, 
      sibling = {
        next = 0xffff8804fa55a438, 
        prev = 0xffff8804fa55a438
      }, 
      group_leader = 0xffff8804fa559f60, 
      ptraced = {
        next = 0xffff8804fa5584f0, 
        prev = 0xffff8804fa5584f0
      }, 
      ptrace_entry = {
        next = 0xffff8804fa558500, 
        prev = 0xffff8804fa558500
      }, 
      pids = {{
          node = {
            next = 0x0, 
            pprev = 0x0
          }, 
          pid = 0xffffffff819f88c0 <init_struct_pid>
        }, {
          node = {
            next = 0x0, 
            pprev = 0x0
          }, 
          pid = 0xffffffff819f88c0 <init_struct_pid>
        }, {
          node = {
            next = 0x0, 
            pprev = 0x0
          }, 
          pid = 0xffffffff819f88c0 <init_struct_pid>
        }}, 
      thread_group = {
        next = 0xffff8804fa55a4b8, 
        prev = 0xffff8804fa55a4b8
      }, 
      thread_node = {
        next = 0xffff8804fa593a90, 
        prev = 0xffff8804fa593a90
      }, 
      vfork_done = 0x0, 
      set_child_tid = 0x0, 
      clear_child_tid = 0x0, 
      utime = 0, 
      stime = 4515000000, 
      utimescaled = 0, 
      stimescaled = 4515000000, 
      gtime = 0, 
      prev_cputime = {
        utime = 0, 
        stime = 0
      }, 
      vtime_seqlock = {
        seqcount = {
          sequence = 4
        }, 
        lock = {
          {
            rlock = {
              raw_lock = {
                {
                  head_tail = 0, 
                  tickets = {
                    head = 0, 
                    tail = 0
                  }
                }
              }
            }
          }
        }
      }, 
      vtime_snap = 4294667549, 
      vtime_snap_whence = VTIME_SYS, 
      nvcsw = 0, 
      nivcsw = 34917496, 
      start_time = {
        tv_sec = 0, 
        tv_nsec = 180000000
      }, 
      real_start_time = {
        tv_sec = 0, 
        tv_nsec = 180000000
      }, 
      min_flt = 0, 
      maj_flt = 0, 
      cputime_expires = {
        utime = 0, 
        stime = 0, 
        sum_exec_runtime = 0
      }, 
      cpu_timers = {{
          next = 0xffff8804fa55a598, 
          prev = 0xffff8804fa55a598
        }, {
          next = 0xffff8804fa55a5a8, 
          prev = 0xffff8804fa55a5a8
        }, {
          next = 0xffff8804fa55a5b8, 
          prev = 0xffff8804fa55a5b8
        }}, 
      real_cred = 0xffff88017fc59d40, 
      cred = 0xffff88017fc59d40, 
      comm = "swapper/3\000\000\000\000\000\000", 
      link_count = 0, 
      total_link_count = 0, 
      sysvsem = {
        undo_list = 0x0
      }, 
      thread = {
        tls_array = {{
            {
              {
                a = 0, 
                b = 0
              }, 
              {
                limit0 = 0, 
                base0 = 0, 
                base1 = 0, 
                type = 0, 
                s = 0, 
                dpl = 0, 
                p = 0, 
                limit = 0, 
                avl = 0, 
                l = 0, 
                d = 0, 
                g = 0, 
                base2 = 0
              }
            }
          }, {
            {
              {
                a = 0, 
                b = 0
              }, 
              {
                limit0 = 0, 
                base0 = 0, 
                base1 = 0, 
                type = 0, 
                s = 0, 
                dpl = 0, 
                p = 0, 
                limit = 0, 
                avl = 0, 
                l = 0, 
                d = 0, 
                g = 0, 
                base2 = 0
              }
            }
          }, {
            {
              {
                a = 0, 
                b = 0
              }, 
              {
                limit0 = 0, 
                base0 = 0, 
                base1 = 0, 
                type = 0, 
                s = 0, 
                dpl = 0, 
                p = 0, 
                limit = 0, 
                avl = 0, 
                l = 0, 
                d = 0, 
                g = 0, 
                base2 = 0
              }
            }
          }}, 
        sp0 = 18446612153695223808, 
        sp = 18446612153695223384, 
        usersp = 0, 
        es = 0, 
        ds = 0, 
        fsindex = 0, 
        gsindex = 0, 
        fs = 0, 
        gs = 0, 
        ptrace_bps = {0x0, 0x0, 0x0, 0x0}, 
        debugreg6 = 0, 
        ptrace_dr7 = 0, 
        cr2 = 20, 
        trap_nr = 14, 
        error_code = 0, 
        fpu = {
          last_cpu = 3, 
          has_fpu = 1, 
          state = 0xffff8804fa582a40
        }, 
        io_bitmap_ptr = 0x0, 
        iopl = 0, 
        io_bitmap_max = 0
      }, 
      fs = 0xffff88017a133e40, 
      files = 0xffff8804fa588780, 
      nsproxy = 0xffffffff819fa120 <init_nsproxy>, 
      signal = 0xffff8804fa593a80, 
      sighand = 0xffff88017a196300, 
      blocked = {
        sig = {0}
      }, 
      real_blocked = {
        sig = {0}
      }, 
      saved_sigmask = {
        sig = {0}
      }, 
      pending = {
        list = {
          next = 0xffff8804fa55a6f0, 
          prev = 0xffff8804fa55a6f0
        }, 
        signal = {
          sig = {0}
        }
      }, 
      sas_ss_sp = 0, 
      sas_ss_size = 0, 
      notifier = 0x0, 
      notifier_data = 0x0, 
      notifier_mask = 0x0, 
      task_works = 0x0, 
      audit_context = 0x0, 
      loginuid = {
        val = 4294967295
      }, 
      sessionid = 4294967295, 
      seccomp = {
        mode = 0, 
        filter = 0x0
      }, 
      parent_exec_id = 0, 
      self_exec_id = 0, 
      alloc_lock = {
        {
          rlock = {
            raw_lock = {
              {
                head_tail = 0, 
                tickets = {
                  head = 0, 
                  tail = 0
                }
              }
            }
          }
        }
      }, 
      pi_lock = {
        raw_lock = {
          {
            head_tail = 131074, 
            tickets = {
              head = 2, 
              tail = 2
            }
          }
        }
      }, 
      pi_waiters = {
        rb_node = 0x0
      }, 
      pi_waiters_leftmost = 0x0, 
      pi_blocked_on = 0x0, 
      journal_info = 0x0, 
      bio_list = 0x0, 
      plug = 0x0, 
      reclaim_state = 0x0, 
      backing_dev_info = 0x0, 
      io_context = 0x0, 
      ptrace_message = 0, 
      last_siginfo = 0x0, 
      ioac = {
        rchar = 0, 
        wchar = 0, 
        syscr = 0, 
        syscw = 0, 
        read_bytes = 0, 
        write_bytes = 0, 
        cancelled_write_bytes = 0
      }, 
      acct_rss_mem1 = 0, 
      acct_vm_mem1 = 0, 
      acct_timexpd = 0, 
      mems_allowed = {
        bits = {3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
      }, 
      mems_allowed_seq = {
        sequence = 0
      }, 
      cpuset_mem_spread_rotor = -1, 
      cpuset_slab_spread_rotor = -1, 
      cgroups = 0xffffffff81e67080 <init_css_set>, 
      cg_list = {
        next = 0xffff8804fa55a8a8, 
        prev = 0xffff8804fa55a8a8
      }, 
      robust_list = 0x0, 
      compat_robust_list = 0x0, 
      pi_state_list = {
        next = 0xffff8804fa55a8c8, 
        prev = 0xffff8804fa55a8c8
      }, 
      pi_state_cache = 0x0, 
      perf_event_ctxp = {0x0, 0x0}, 
      perf_event_mutex = {
        count = {
          counter = 1
        }, 
        wait_lock = {
          {
            rlock = {
              raw_lock = {
                {
                  head_tail = 0, 
                  tickets = {
                    head = 0, 
                    tail = 0
                  }
                }
              }
            }
          }
        }, 
        wait_list = {
          next = 0xffff8804fa55a8f8, 
          prev = 0xffff8804fa55a8f8
        }, 
        owner = 0x0, 
        {
          osq = 0x0, 
          __UNIQUE_ID_rh_kabi_hide0 = {
            spin_mlock = 0x0
          }, 
          {<No data fields>}
        }
      }, 
      perf_event_list = {
        next = 0xffff8804fa55a918, 
        prev = 0xffff8804fa55a918
      }, 
      mempolicy = 0xffff88017b880318, 
      il_next = 1, 
      pref_node_fork = 0, 
      numa_scan_seq = 0, 
      numa_scan_period = 1000, 
      numa_scan_period_max = 0, 
      numa_preferred_nid = -1, 
      numa_migrate_retry = 0, 
      node_stamp = 0, 
      last_task_numa_placement = 0, 
      last_sum_exec_runtime = 0, 
      numa_work = {
        next = 0xffff8804fa55a968, 
        func = 0x0
      }, 
      numa_entry = {
        next = 0xffff8804fa55a978, 
        prev = 0xffff8804fa55a978
      }, 
      numa_group = 0x0, 
      numa_faults_memory = 0x0, 
      total_numa_faults = 0, 
      numa_faults_buffer_memory = 0x0, 
      numa_faults_cpu = 0x0, 
      numa_faults_buffer_cpu = 0x0, 
      numa_faults_locality = {0, 0}, 
      numa_pages_migrated = 0, 
      rcu = {
        next = 0x0, 
        func = 0x0
      }, 
      splice_pipe = 0x0, 
      task_frag = {
        page = 0x0, 
        offset = 0, 
        size = 0
      }, 
      delays = 0xffff88017a09e700, 
      nr_dirtied = 0, 
      nr_dirtied_pause = 32, 
      dirty_paused_when = 0, 
      timer_slack_ns = 50000, 
      default_timer_slack_ns = 50000, 
      curr_ret_stack = -1, 
      ret_stack = 0x0, 
      ftrace_timestamp = 0, 
      trace_overrun = {
        counter = 0
      }, 
      tracing_graph_pause = {
        counter = 0
      }, 
      trace = 0, 
      trace_recursion = 0, 
      memcg_batch = {
        do_batch = 0, 
        memcg = 0x0, 
        nr_pages = 0, 
        memsw_nr_pages = 0
      }, 
      memcg_kmem_skip_account = 0, 
      ptrace_bp_refcnt = {
        counter = 1
      }, 
      utask = 0x0, 
      {
        last_switch_count = 0, 
        __UNIQUE_ID_rh_kabi_hide11 = {
          rh_reserved1 = 0
        }, 
        {<No data fields>}
      }, 
      {
        atomic_flags = 0, 
        __UNIQUE_ID_rh_kabi_hide12 = {
          rh_reserved2 = 0
        }, 
        {<No data fields>}
      }, 
      rh_reserved3 = 0, 
      {
        mm_shmempages = 0, 
        __UNIQUE_ID_rh_kabi_hide13 = {
          rh_reserved4 = 0
        }, 
        {<No data fields>}
      }, 
      rh_reserved5 = 0, 
      rh_reserved6 = 0, 
      rh_reserved7 = 0, 
      rh_reserved8 = 0, 
      memcg_oom = {
        memcg = 0x0, 
        gfp_mask = 0, 
        order = 0, 
        may_oom = 0
      }, 
      tlb_ubc = {
        cpumask = {
          bits = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
        }, 
        flush_required = false, 
        writable = false
      }, 
      dl = {
        rb_node = {
          __rb_parent_color = 18446612153694006624, 
          rb_right = 0x0, 
          rb_left = 0x0
        }, 
        dl_runtime = 0, 
        dl_deadline = 0, 
        dl_period = 0, 
        dl_bw = 0, 
        runtime = 0, 
        deadline = 0, 
        flags = 0, 
        dl_throttled = 0, 
        dl_boosted = 0, 
        dl_yielded = 0, 
        dl_timer = {
          node = {
            node = {
              __rb_parent_color = 18446612153694006712, 
              rb_right = 0x0, 
              rb_left = 0x0
            }, 
            expires = {
              tv64 = 0
            }
          }, 
          _softexpires = {
            tv64 = 0
          }, 
          function = 0xffffffff810d7d40 <dl_task_timer>, 
          base = 0xffff88046e60f8e0, 
          state = 0, 
          start_pid = -1, 
          start_site = 0x0, 
          start_comm = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
        }
      }, 
      pushable_dl_tasks = {
        __rb_parent_color = 18446612153694006808, 
        rb_right = 0x0, 
        rb_left = 0x0
      }, 
      statistics = {
        wait_start = 0, 
        wait_max = 0, 
        wait_count = 0, 
        wait_sum = 0, 
        iowait_count = 0, 
        iowait_sum = 0, 
        sleep_start = 0, 
        sleep_max = 0, 
        sum_sleep_runtime = 0, 
        block_start = 0, 
        block_max = 0, 
        exec_max = 0, 
        slice_max = 0, 
        nr_migrations_cold = 0, 
        nr_failed_migrations_affine = 0, 
        nr_failed_migrations_running = 0, 
        nr_failed_migrations_hot = 0, 
        nr_forced_migrations = 0, 
        nr_wakeups = 0, 
        nr_wakeups_sync = 0, 
        nr_wakeups_migrate = 0, 
        nr_wakeups_local = 0, 
        nr_wakeups_remote = 0, 
        nr_wakeups_affine = 0, 
        nr_wakeups_affine_attempts = 0, 
        nr_wakeups_passive = 0, 
        nr_wakeups_idle = 0
      }
    }
    
    struct thread_info {
      task = 0xffff8804fa559f60, 
      exec_domain = 0xffffffff819eb6e0 <default_exec_domain>, 
      flags = 0, 
      status = 0, 
      cpu = 3, 
      preempt_count = 65536, 
      addr_limit = {
        seg = 18446744073709551615
      }, 
      restart_block = {
        fn = 0xffffffff8109dcf0 <do_no_restart_syscall>, 
        {
          futex = {
            uaddr = 0x0, 
            val = 0, 
            flags = 0, 
            bitset = 0, 
            time = 0, 
            uaddr2 = 0x0
          }, 
          nanosleep = {
            clockid = 0, 
            rmtp = 0x0, 
            compat_rmtp = 0x0, 
            expires = 0
          }, 
          poll = {
            ufds = 0x0, 
            nfds = 0, 
            has_timeout = 0, 
            tv_sec = 0, 
            tv_nsec = 0
          }
        }
      }, 
      sysenter_return = 0x0, 
      sig_on_uaccess_error = 0, 
      uaccess_err = 0
    }
    
    PID: 0      TASK: ffff8804fa559f60  CPU: 3   COMMAND: "swapper/3"
           MM               PGD          RSS    TOTAL_VM
           0                 0            0k       0k   
    
    ? file icon crash (50,229 bytes) 2017-04-16 00:20 +

-Relationships
+Relationships

-Notes

~0029075

adip (reporter)

KERNEL: /data/adi/crash/3.10.0-514.10.2.el7.x86_64/vmlinux
    DUMPFILE: /data/adi/crash/127.0.0.1-2017-04-15-02:53:06/vmcore [PARTIAL DUMP]
        CPUS: 12
        DATE: Sat Apr 15 02:52:55 2017
      UPTIME: 9 days, 04:22:12
LOAD AVERAGE: 0.08, 0.08, 0.05
       TASKS: 416
    NODENAME: xd3
     RELEASE: 3.10.0-514.10.2.el7.x86_64
     VERSION: #1 SMP Fri Mar 3 00:04:05 UTC 2017
     MACHINE: x86_64 (1700 Mhz)
      MEMORY: 31.9 GB
       PANIC: ".1BUG: unable to handle kernel NULL pointer dereference at 0000000000000014"
         PID: 0
     COMMAND: "swapper/3"
        TASK: ffff8804fa559f60 (1 of 12) [THREAD_INFO: ffff8804fa680000]
         CPU: 3
       STATE: TASK_RUNNING (PANIC)

~0029076

adip (reporter)

Reported upstream, see https://bugzilla.redhat.com/show_bug.cgi?id=1442593

~0029085

tigalch (manager)

Kernel got updated. What is the state of the above mentioned bugzilla issue (since kernel issues are usually private)?

~0029086

adip (reporter)

The bug was closed because of out-of-tree loading of drbd module. Quoting:

"A few things: This looks like a bug in which a task has disappeared. The dereference of p->flags in account_system_time() is what is causing the panic, ie) p is null.

A quick search for similar panics doesn't yield anything, so this looks like it is something specific to your system. I did notice that you've installed an out-of-tree driver which may be responsible for this problem. I cannot debug an issue like this where out-of-tree modules have been loaded as I have no idea about the stability of the code.

If you can reproduce the failure without the drbd, please reopen this bug."

The node is now running kernel 3.10.0-514.16.1.el7.x86_64.

~0029087

tigalch (manager)

So, can you help RH in that matter, or not?
Else we will close this issue as well, since the fix has to come from RH.

~0029088

adip (reporter)

Please close it, thanks.

~0029089

tigalch (manager)

Closing per reporters feedback
+Notes

-Issue History
Date Modified Username Field Change
2017-04-16 00:20 adip New Issue
2017-04-16 00:20 adip File Added: crash
2017-04-16 00:33 adip Note Added: 0029075
2017-04-16 00:36 adip Note Added: 0029076
2017-04-18 09:58 tigalch Note Added: 0029085
2017-04-18 10:34 adip Note Added: 0029086
2017-04-18 10:36 tigalch Note Added: 0029087
2017-04-18 10:37 adip Note Added: 0029088
2017-04-18 10:39 tigalch Status new => closed
2017-04-18 10:39 tigalch Resolution open => unable to reproduce
2017-04-18 10:39 tigalch Note Added: 0029089
+Issue History