View Issue Details

IDProjectCategoryView StatusLast Update
0013136CentOS-7kernelpublic2017-04-18 10:39
Reporteradip 
PrioritynormalSeveritymajorReproducibilityunable to reproduce
Status closedResolutionunable to reproduce 
Product Version 
Target VersionFixed in Version 
Summary0013136: [3.10.0-514.10.2.el7.x86_64] PANIC: ".1BUG: unable to handle kernel NULL pointer dereference at 0000000000000014"
DescriptionThe server is a Dell PowerEdge R730xd running as active node of an active/passive Pacemaker Cluster.

See attached some crash information.
TagsNo tags attached.
abrt_hash
URL

Activities

adip

adip

2017-04-16 00:20

reporter  

crash (50,229 bytes)
PID: 0      TASK: ffff8804fa559f60  CPU: 3   COMMAND: "swapper/3"
 #0 [ffff88086de43aa8] machine_kexec at ffffffff81059bdb
 #1 [ffff88086de43b08] __crash_kexec at ffffffff81105382
 #2 [ffff88086de43bd8] crash_kexec at ffffffff81105470
 #3 [ffff88086de43bf0] oops_end at ffffffff8168f3c8
 #4 [ffff88086de43c18] no_context at ffffffff8167efd6
 #5 [ffff88086de43c68] __bad_area_nosemaphore at ffffffff8167f06c
 #6 [ffff88086de43cb0] bad_area_nosemaphore at ffffffff8167f1d6
 #7 [ffff88086de43cc0] __do_page_fault at ffffffff8169215e
 #8 [ffff88086de43d20] do_page_fault at ffffffff81692305
 #9 [ffff88086de43d50] page_fault at ffffffff8168e5c8
    [exception RIP: account_system_time+21]
    RIP: ffffffff810c8375  RSP: ffff88086de43e00  RFLAGS: 00010086
    RAX: 0000000000000000  RBX: ffff88086de56c40  RCX: 00000000000f4240
    RDX: 00000000000f4240  RSI: 0000000000010000  RDI: 0000000000000000
    RBP: ffff88086de43e28   R8: 0000000000000000   R9: 00000000000c1af5
    R10: 000000003b9aca00  R11: 0000000000000000  R12: 00000000000f4240
    R13: 0000000000000000  R14: 0000000000000000  R15: ffff88086de4f9d8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff88086de43e30] account_process_tick at ffffffff810c8682
#11 [ffff88086de43e68] update_process_times at ffffffff8109932d
#12 [ffff88086de43e90] tick_sched_handle at ffffffff810f3145
#13 [ffff88086de43eb0] tick_sched_timer at ffffffff810f31c1
#14 [ffff88086de43ed8] __hrtimer_run_queues at ffffffff810b4a32
#15 [ffff88086de43f30] hrtimer_interrupt at ffffffff810b4fd0
#16 [ffff88086de43f80] local_apic_timer_interrupt at ffffffff81050fd7
#17 [ffff88086de43f98] smp_apic_timer_interrupt at ffffffff8169920f
#18 [ffff88086de43fb0] apic_timer_interrupt at ffffffff8169775d
--- <IRQ stack> ---
#19 [ffff8804fa683da8] apic_timer_interrupt at ffffffff8169775d
    [exception RIP: cpuidle_enter_state+82]
    RIP: ffffffff81514492  RSP: ffff8804fa683e50  RFLAGS: 00000206
    RAX: 0002d1882599f91b  RBX: 000000000000f8a0  RCX: 0000000000000018
    RDX: 0000000225c17d03  RSI: ffff8804fa683fd8  RDI: 0002d1882599f91b
    RBP: ffff8804fa683e78   R8: 00000000000070f7   R9: 000000000000001c
    R10: 000000000000c32d  R11: 0000000000000000  R12: ffff8804fa683e20
    R13: ffff88086de4f8e0  R14: 0000000000000082  R15: 0002d18825999180
    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
#20 [ffff8804fa683e80] cpuidle_idle_call at ffffffff815145d9
#21 [ffff8804fa683ec0] arch_cpu_idle at ffffffff810350ee
#22 [ffff8804fa683ed0] cpu_startup_entry at ffffffff810e7e65
#23 [ffff8804fa683f28] start_secondary at ffffffff8104f07a
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>     0      0   0  ffffffff819c1460  RU   0.0       0      0  [swapper/0]
>     0      0   1  ffff8804fa55edd0  RU   0.0       0      0  [swapper/1]
>     0      0   2  ffff88017a1dedd0  RU   0.0       0      0  [swapper/2]
>     0      0   3  ffff8804fa559f60  RU   0.0       0      0  [swapper/3]
>     0      0   4  ffff88017a1d8fb0  RU   0.0       0      0  [swapper/4]
>     0      0   5  ffff8804fa55de20  RU   0.0       0      0  [swapper/5]
>     0      0   6  ffff88017a1dde20  RU   0.0       0      0  [swapper/6]
>     0      0   7  ffff8804fa55af10  RU   0.0       0      0  [swapper/7]
>     0      0   8  ffff88017a1d9f60  RU   0.0       0      0  [swapper/8]
>     0      0   9  ffff8804fa55ce70  RU   0.0       0      0  [swapper/9]
>     0      0  10  ffff88017a1dce70  RU   0.0       0      0  [swapper/10]
>     0      0  11  ffff8804fa55bec0  RU   0.0       0      0  [swapper/11]
      1      0   1  ffff8804fa558000  IN   0.0   44056   4424  systemd
      2      0  11  ffff8804fa558fb0  IN   0.0       0      0  [kthreadd]
      3      2   0  ffff88017a1a8000  IN   0.0       0      0  [ksoftirqd/0]
      5      2   0  ffff88017a1a9f60  IN   0.0       0      0  [kworker/0:0H]
      8      2   0  ffff88017a1ace70  IN   0.0       0      0  [migration/0]
      9      2   0  ffff88017a1ade20  IN   0.0       0      0  [rcu_bh]
     10      2  11  ffff88017a1aedd0  IN   0.0       0      0  [rcu_sched]
     11      2   0  ffff88017a1d8000  IN   0.0       0      0  [watchdog/0]
     12      2   1  ffff8804f9d15e20  IN   0.0       0      0  [watchdog/1]
     13      2   1  ffff8804f9d14e70  IN   0.0       0      0  [migration/1]
     14      2   1  ffff8804f9d13ec0  IN   0.0       0      0  [ksoftirqd/1]
     16      2   1  ffff8804f9d11f60  IN   0.0       0      0  [kworker/1:0H]
     18      2   2  ffff880179ab0000  IN   0.0       0      0  [watchdog/2]
     19      2   2  ffff880179ab0fb0  IN   0.0       0      0  [migration/2]
     20      2   2  ffff880179ab1f60  IN   0.0       0      0  [ksoftirqd/2]
     22      2   2  ffff880179ab3ec0  IN   0.0       0      0  [kworker/2:0H]
     23      2   3  ffff8804f9bc0000  IN   0.0       0      0  [watchdog/3]
     24      2   3  ffff8804f9bc0fb0  IN   0.0       0      0  [migration/3]
     25      2   3  ffff8804f9bc1f60  IN   0.0       0      0  [ksoftirqd/3]
     27      2   3  ffff8804f9bc3ec0  IN   0.0       0      0  [kworker/3:0H]
     28      2   4  ffff880179ab6dd0  IN   0.0       0      0  [watchdog/4]
     29      2   4  ffff880179ab5e20  IN   0.0       0      0  [migration/4]
     30      2   4  ffff880179ab4e70  IN   0.0       0      0  [ksoftirqd/4]
     32      2   4  ffff880179b18fb0  IN   0.0       0      0  [kworker/4:0H]
     33      2   5  ffff8804f9bc6dd0  IN   0.0       0      0  [watchdog/5]
     34      2   5  ffff8804f9bc5e20  IN   0.0       0      0  [migration/5]
     35      2   5  ffff8804f9bc4e70  IN   0.0       0      0  [ksoftirqd/5]
     37      2   5  ffff8804f9408fb0  IN   0.0       0      0  [kworker/5:0H]
     38      2   6  ffff880179b1edd0  IN   0.0       0      0  [watchdog/6]
     39      2   6  ffff880179b1de20  IN   0.0       0      0  [migration/6]
     40      2   6  ffff880179b1ce70  IN   0.0       0      0  [ksoftirqd/6]
     43      2   7  ffff8804f940edd0  IN   0.0       0      0  [watchdog/7]
     44      2   7  ffff8804f940de20  IN   0.0       0      0  [migration/7]
     45      2   7  ffff8804f940ce70  IN   0.0       0      0  [ksoftirqd/7]
     47      2   7  ffff8804f940af10  IN   0.0       0      0  [kworker/7:0H]
     48      2   8  ffff880179b19f60  IN   0.0       0      0  [watchdog/8]
     49      2   8  ffff880179b88000  IN   0.0       0      0  [migration/8]
     50      2   8  ffff880179b88fb0  IN   0.0       0      0  [ksoftirqd/8]
     52      2   8  ffff880179b8af10  IN   0.0       0      0  [kworker/8:0H]
     53      2   9  ffff8804f9409f60  IN   0.0       0      0  [watchdog/9]
     54      2   9  ffff8804f9458000  IN   0.0       0      0  [migration/9]
     55      2   9  ffff8804f9458fb0  IN   0.0       0      0  [ksoftirqd/9]
     57      2   9  ffff8804f945af10  IN   0.0       0      0  [kworker/9:0H]
     58      2  10  ffff880179b8edd0  IN   0.0       0      0  [watchdog/10]
     59      2  10  ffff880179b8de20  IN   0.0       0      0  [migration/10]
     60      2  10  ffff880179b8ce70  IN   0.0       0      0  [ksoftirqd/10]
     62      2  10  ffff880179bc8000  IN   0.0       0      0  [kworker/10:0H]
     63      2  11  ffff8804f945edd0  IN   0.0       0      0  [watchdog/11]
     64      2  11  ffff8804f945de20  IN   0.0       0      0  [migration/11]
     65      2  11  ffff8804f945ce70  IN   0.0       0      0  [ksoftirqd/11]
     67      2  11  ffff8804f94b8000  IN   0.0       0      0  [kworker/11:0H]
     70      2   8  ffff880179bcce70  IN   0.0       0      0  [kdevtmpfs]
     71      2   3  ffff880179bcbec0  IN   0.0       0      0  [netns]
     72      2   1  ffff880179bcaf10  IN   0.0       0      0  [khungtaskd]
     73      2   3  ffff88046fc68000  IN   0.0       0      0  [writeback]
     74      2   3  ffff88046fc68fb0  IN   0.0       0      0  [kintegrityd]
     75      2   3  ffff88046fc69f60  IN   0.0       0      0  [bioset]
     76      2   3  ffff88046fc6af10  IN   0.0       0      0  [kblockd]
     77      2   3  ffff88046fc6bec0  IN   0.0       0      0  [md]
     86      2   0  ffff88086eb7de20  IN   0.0       0      0  [kswapd0]
     87      2   3  ffff88086eb7edd0  IN   0.0       0      0  [kswapd1]
     88      2   3  ffff88046fc6edd0  IN   0.0       0      0  [ksmd]
     89      2   7  ffff88046fc6de20  IN   0.0       0      0  [khugepaged]
     90      2  11  ffff88046fc6ce70  IN   0.0       0      0  [fsnotify_mark]
     91      2   3  ffff88046cf08000  IN   0.0       0      0  [crypto]
     99      2   3  ffff88046cf30000  IN   0.0       0      0  [kthrotld]
    101      2   3  ffff88046cf31f60  IN   0.0       0      0  [kmpath_rdacd]
    104      2   6  ffff88046cf34e70  IN   0.0       0      0  [kpsmoused]
    105      2   6  ffff88046cf35e20  IN   0.0       0      0  [ipv6_addrconf]
    125      2   3  ffff88046c8d5e20  IN   0.0       0      0  [deferwq]
    179      2  10  ffff88046cbe8fb0  IN   0.0       0      0  [kauditd]
    365      2   5  ffff88086c352f10  IN   0.0       0      0  [scsi_eh_0]
    366      2   8  ffff88086c353ec0  IN   0.0       0      0  [scsi_tmf_0]
    367      2   7  ffff88086ce65e20  IN   0.0       0      0  [ata_sff]
    368      2   7  ffff88086c116dd0  IN   0.0       0      0  [scsi_eh_1]
    369      2   7  ffff88086c084e70  IN   0.0       0      0  [scsi_tmf_1]
    370      2   0  ffff88086c113ec0  IN   0.0       0      0  [usb-storage]
    379      2   5  ffff88086c114e70  IN   0.0       0      0  [ixgbe]
    381      2  11  ffff88086c7dde20  IN   0.0       0      0  [scsi_eh_2]
    382      2   0  ffff88086c7d9f60  IN   0.0       0      0  [scsi_tmf_2]
    383      2   2  ffff88086c7daf10  IN   0.0       0      0  [scsi_eh_3]
    384      2   8  ffff88086c115e20  IN   0.0       0      0  [scsi_tmf_3]
    385      2  11  ffff88086c111f60  IN   0.0       0      0  [scsi_eh_4]
    386      2   8  ffff88086c110fb0  IN   0.0       0      0  [scsi_tmf_4]
    387      2   1  ffff88086c7dbec0  IN   0.0       0      0  [scsi_eh_5]
    388      2   1  ffff88086c7d8fb0  IN   0.0       0      0  [scsi_tmf_5]
    393      2   2  ffff88086ce63ec0  IN   0.0       0      0  [scsi_eh_6]
    394      2   2  ffff88086ce62f10  IN   0.0       0      0  [scsi_tmf_6]
    396      2   2  ffff88086ce61f60  IN   0.0       0      0  [scsi_eh_7]
    397      2   9  ffff88086c081f60  IN   0.0       0      0  [scsi_tmf_7]
    398      2   9  ffff88086c085e20  IN   0.0       0      0  [scsi_eh_8]
    399      2   4  ffff88086c086dd0  IN   0.0       0      0  [scsi_tmf_8]
    401      2  11  ffff88086c082f10  IN   0.0       0      0  [scsi_eh_9]
    402      2   9  ffff88086a4a0000  IN   0.0       0      0  [scsi_tmf_9]
    403      2   9  ffff88086a4a0fb0  IN   0.0       0      0  [scsi_eh_10]
    405      2   7  ffff88086a4a1f60  IN   0.0       0      0  [scsi_tmf_10]
    406      2   4  ffff88086a4a2f10  IN   0.0       0      0  [scsi_eh_11]
    407      2   4  ffff88086a4a3ec0  IN   0.0       0      0  [scsi_tmf_11]
    541      2   2  ffff88086ce60fb0  IN   0.0       0      0  [kdmflush]
    542      2   2  ffff88086ce60000  IN   0.0       0      0  [bioset]
    551      2   2  ffff88086a4a6dd0  IN   0.0       0      0  [kdmflush]
    552      2   2  ffff88086a4a5e20  IN   0.0       0      0  [bioset]
    567      2   2  ffff88086a6b8000  IN   0.0       0      0  [xfsalloc]
    568      2   2  ffff88086a6b8fb0  IN   0.0       0      0  [xfs_mru_cache]
    569      2   2  ffff88086a6b9f60  IN   0.0       0      0  [xfs-buf/dm-0]
    570      2   2  ffff88086a6baf10  IN   0.0       0      0  [xfs-data/dm-0]
    571      2   2  ffff88086a6bbec0  IN   0.0       0      0  [xfs-conv/dm-0]
    572      2   2  ffff88086a6bce70  IN   0.0       0      0  [xfs-cil/dm-0]
    573      2   2  ffff88086a6bde20  IN   0.0       0      0  [xfs-reclaim/dm-]
    574      2   2  ffff88086a6bedd0  IN   0.0       0      0  [xfs-log/dm-0]
    575      2   2  ffff88086a6c8000  IN   0.0       0      0  [xfs-eofblocks/d]
    576      2   5  ffff88086a6c8fb0  IN   0.0       0      0  [xfsaild/dm-0]
    633      2   0  ffff88046b198000  IN   0.0       0      0  [kworker/0:1H]
    647      2  11  ffff88086af86dd0  IN   0.0       0      0  [kworker/11:1H]
    655      1   1  ffff88046c0cce70  IN   0.0  126644   1484  systemd-journal
    677      2   2  ffff88086af83ec0  IN   0.0       0      0  [rpciod]
    679      1   3  ffff88046c8d4e70  IN   0.0   43528   1816  systemd-udevd
   1511      2   5  ffff88086ce86dd0  IN   0.0       0      0  [edac-poller]
   2087      2   2  ffff88046cf0bec0  IN   0.0       0      0  [dm_bufio_cache]
   2089      2   2  ffff88046cf0af10  IN   0.0       0      0  [kdmflush]
   2090      2   2  ffff88046cf0ce70  IN   0.0       0      0  [bioset]
   2092      2   9  ffff88046cf0de20  IN   0.0       0      0  [kdmflush]
   2093      2   2  ffff88046cf08fb0  IN   0.0       0      0  [bioset]
   2095      2   2  ffff88086b68af10  IN   0.0       0      0  [kdmflush]
   2096      2   2  ffff88086b68ce70  IN   0.0       0      0  [bioset]
   2105      2   2  ffff88086b68edd0  IN   0.0       0      0  [kdmflush]
   2106      2   2  ffff88086b689f60  IN   0.0       0      0  [bioset]
   2107      2   2  ffff88086b68de20  IN   0.0       0      0  [kcopyd]
   2108      2   2  ffff88086b68bec0  IN   0.0       0      0  [bioset]
   2109      2   2  ffff88086b688fb0  IN   0.0       0      0  [dm-cache]
   2110      2   9  ffff88086b688000  IN   0.0       0      0  [bioset]
   2111      2   6  ffff88046cf09f60  IN   0.0       0      0  [kworker/6:1H]
   2123      2   4  ffff88046c088000  IN   0.0       0      0  [kworker/4:1H]
   2129      2   2  ffff88086eb7bec0  IN   0.0       0      0  [xfs-buf/sdd1]
   2130      2   2  ffff88086eb79f60  IN   0.0       0      0  [xfs-data/sdd1]
   2131      2   2  ffff88086eb7af10  IN   0.0       0      0  [xfs-conv/sdd1]
   2132      2   2  ffff8804f9738000  IN   0.0       0      0  [xfs-cil/sdd1]
   2133      2   2  ffff8804f973de20  IN   0.0       0      0  [xfs-buf/sda1]
   2134      2   2  ffff8804f9738fb0  IN   0.0       0      0  [xfs-reclaim/sdd]
   2136      2   1  ffff8804f973ce70  IN   0.0       0      0  [xfs-data/sda1]
   2137      2   2  ffff8804f9736dd0  IN   0.0       0      0  [xfs-log/sdd1]
   2138      2   2  ffff8804f9733ec0  IN   0.0       0      0  [xfs-conv/sda1]
   2139      2   2  ffff88046b694e70  IN   0.0       0      0  [xfs-eofblocks/s]
   2140      2   3  ffff88046b690000  IN   0.0       0      0  [xfs-cil/sda1]
   2141      2   3  ffff88046b693ec0  IN   0.0       0      0  [xfs-reclaim/sda]
   2142      2   2  ffff88046b690fb0  IN   0.0       0      0  [xfs-log/sda1]
   2143      2   2  ffff88046b692f10  IN   0.0       0      0  [xfs-eofblocks/s]
   2144      2   5  ffff88087f768fb0  IN   0.0       0      0  [xfsaild/sda1]
   2150      2   1  ffff88087f76bec0  IN   0.0       0      0  [xfsaild/sdd1]
   2167      1   1  ffff8800364a4e70  IN   0.0   55416   1836  auditd
   2171      1   5  ffff88046e3cde20  IN   0.0   55416   1836  auditd
   2188      1   0  ffff88087f769f60  IN   0.0   24420   1676  dbus-daemon
   2193      1   0  ffff88046e3cbec0  IN   0.1  637348  38052  pcsd
   2194      1   6  ffff88046e3c8fb0  IN   0.0   24192   1692  systemd-logind
   2196      1   0  ffff88046e3cce70  IN   0.0  128488   2960  smartd
   2209      1   0  ffff88087fe92f10  IN   0.0   12132   3832  haveged
   2212      1   8  ffff88087fe90fb0  IN   0.0  527516  14188  polkitd
   2219      1  11  ffff88046c2a4e70  IN   0.0  201212   1372  gssproxy
   2228      1   6  ffff88086af80000  IN   0.0  527516  14188  gmain
   2229      1   5  ffff88046c2a1f60  IN   0.0  201212   1372  gssproxy
   2230      1   7  ffff88046c2a0fb0  IN   0.0  201212   1372  gssproxy
   2231      1   0  ffff88046c2a6dd0  IN   0.0  201212   1372  gssproxy
   2232      1   5  ffff88046c2a5e20  IN   0.0  201212   1372  gssproxy
   2233      1   5  ffff88046c2a2f10  IN   0.0  201212   1372  gssproxy
   2235      1  10  ffff88046b68bec0  IN   0.0  527516  14188  gdbus
   2239      1   3  ffff88046e3c8000  IN   0.0  527516  14188  JS GC Helper
   2240      1   6  ffff88046e3c9f60  IN   0.0  527516  14188  JS Sour~ Thread
   2242      1   3  ffff88046c08edd0  IN   0.0  126224   1572  crond
   2245      1   7  ffff88046e3cedd0  IN   0.0  527516  14188  runaway-killer-
   2260      1   0  ffff88086af81f60  IN   0.0   44644   2368  ntpd
   2389      2   2  ffff8804f9739f60  IN   0.0       0      0  [bond0]
   2403      2   2  ffff8800364a2f10  IN   0.0       0      0  [bond1]
   2465      1   9  ffff88046b68ce70  IN   0.1  637348  38052  pcsd
   2466      1   8  ffff88046b68de20  IN   0.1  637348  38052  pcsd
   2584      2   2  ffff8800364a1f60  IN   0.0       0      0  [bond2]
   2876      1   5  ffff88046c3b5e20  IN   0.1  553152  18748  tuned
   2879      1   4  ffff88087f76edd0  IN   0.0   44452   4008  tgtd
   2918      1   8  ffff88046c08af10  IN   0.0   82524   1344  sshd
   2987      1   5  ffff88086a6cce70  IN   0.0  110036    844  agetty
   3001      1   0  ffff88046b691f60  IN   0.0   22800    744  ipmievd
   3153      1   1  ffff88046b68edd0  IN   0.0  191692  13884  corosync
   3157      1   4  ffff88046c3b4e70  IN   0.0  191692  13884  corosync
   3172      1   3  ffff88046c241f60  IN   0.1  553152  18748  gmain
   3180      1   3  ffff88046c240fb0  IN   0.1  553152  18748  tuned
   3183      1   4  ffff88046c245e20  IN   0.1  553152  18748  tuned
   3188      1   9  ffff880469fa0fb0  IN   0.1  553152  18748  tuned
   3288      1   8  ffff88086defce70  IN   0.0  155704   9472  munin-node
   3839      1   5  ffff88087f851f60  IN   0.0   89036   2172  master
   3882   3839   1  ffff88086879de20  IN   0.0   89208   4052  qmgr
   4124      1   9  ffff88046e3fde20  IN   0.0  132632   8196  pacemakerd
   4250   4124  10  ffff8808673e0fb0  IN   0.0  137084  12724  cib
   4253   4124   2  ffff8808673e1f60  IN   0.0  136004   7904  stonithd
   4254   4124   3  ffff8808673e6dd0  IN   0.0  105188   4988  lrmd
   4255   4124   3  ffff88086a4a4e70  IN   0.0  126920   7508  attrd
   4256   4124   0  ffff88086df70fb0  IN   0.0  117040   4628  pengine
   4258   4124  11  ffff88086df76dd0  IN   0.0  145332   9260  crmd
   4335      2   3  ffff8808673e5e20  IN   0.0       0      0  [kworker/3:1H]
   4340      2   1  ffff8804f9731f60  IN   0.0       0      0  [kworker/1:1H]
   4589      1   0  ffff88086b641f60  IN   0.1 1070688  52756  dsm_sa_datamgrd
   4624      2   8  ffff88046c2a0000  IN   0.0       0      0  [kworker/8:1H]
   4625      2  10  ffff88046c2a3ec0  IN   0.0       0      0  [kworker/10:1H]
   4741      2   9  ffff8804f94bce70  IN   0.0       0      0  [kworker/9:1H]
   5002      1   5  ffff88087f853ec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5003      1   0  ffff88085e2ebec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5004      1  11  ffff88046c913ec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5008      1   2  ffff88085e2e8fb0  IN   0.1  637348  38052  pcsd
   5326      1   5  ffff88086ce2de20  IN   0.0   64948   1444  rpcbind
   5407      2   9  ffff88087fe94e70  IN   0.0       0      0  [bioset]
   5412      2   2  ffff88087fe95e20  IN   0.0       0      0  [drbd-reissue]
   5438      2   9  ffff88087fe93ec0  IN   0.0       0      0  [drbd1_submit]
   5443      2   9  ffff88087fe90000  IN   0.0       0      0  [drbd0_submit]
   5452      2   0  ffff88046c244e70  IN   0.0       0      0  [drbd_w_r1_DRBDL]
   5455      2   1  ffff88046c242f10  IN   0.0       0      0  [drbd_w_r0_sdb]
   5470      2   0  ffff88046c240000  IN   0.0       0      0  [drbd_r_r1_DRBDL]
   5475      2   1  ffff88046c246dd0  IN   0.0       0      0  [drbd_r_r0_sdb]
   5616      1  11  ffff880869f86dd0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5617      1   4  ffff880869f80000  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5618      1   4  ffff880869f81f60  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5619      1   0  ffff880869f83ec0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5620      1   2  ffff880869f84e70  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5622      1   2  ffff88085e2eedd0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5623      1   4  ffff88085c72edd0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5643      1   9  ffff88085c728fb0  IN   0.0  300504   4060  dsm_sa_eventmgr
   5644      1   0  ffff88046c914e70  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5645      1   6  ffff88086ce29f60  IN   0.0  300504   4060  dsm_sa_eventmgr
   5646      1   0  ffff88086ce2ce70  IN   0.0  300504   4060  dsm_sa_eventmgr
   5670      1  10  ffff88046c911f60  IN   0.0  510900   9948  dsm_sa_snmpd
   5671      1   4  ffff88085c72de20  IN   0.0  510900   9948  dsm_sa_snmpd
   5672      1   9  ffff88086ce28fb0  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5673      1  11  ffff88003645ce70  IN   0.0  510900   9948  dsm_sa_snmpd
   5674      1   5  ffff88003645de20  IN   0.0  510900   9948  dsm_sa_snmpd
   5677   5003   0  ffff88046c915e20  IN   0.1  725452  18484  dsm_sa_datamgrd
   5678      1   0  ffff88046c912f10  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5681      1   0  ffff88085e2e8000  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5695      1   8  ffff88046b1d8000  IN   0.0  510900   9948  dsm_sa_snmpd
   5696      1   1  ffff88046b1d8fb0  IN   0.0  510900   9948  dsm_sa_snmpd
   5772      1   0  ffff88086ce28000  IN   0.0  137908    880  dsm_om_connsvcd
   5773      1   1  ffff88087f76ce70  IN   0.0  632408   4548  dsm_om_shrsvcd
   5774   5772   2  ffff880469619f60  IN   0.5 4165768 186360  dsm_om_connsvcd
   5775      1   1  ffff88086ce2bec0  IN   0.0  632408   4548  dsm_om_shrsvcd
   5776   5772   6  ffff88046c910000  IN   0.5 4165768 186360  dsm_om_connsvcd
   5783      1  10  ffff88046a658000  IN   0.0  632408   4548  dsm_om_shrsvcd
   5786      1   2  ffff88046a65de20  IN   0.0  632408   4548  dsm_om_shrsvcd
   5787      1  10  ffff88086c341f60  IN   0.0  632408   4548  dsm_om_shrsvcd
   5788      1   6  ffff88046a65edd0  IN   0.0  632408   4548  dsm_om_shrsvcd
   5789      1   2  ffff88046a65ce70  IN   0.0  632408   4548  dsm_om_shrsvcd
   5790      1   2  ffff88046a65af10  IN   0.0  632408   4548  dsm_om_shrsvcd
   5876   5772  10  ffff88086af26dd0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5877   5772   6  ffff88086af23ec0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5878   5772   0  ffff88086af25e20  IN   0.5 4165768 186360  dsm_om_connsvcd
   5879   5772   3  ffff88086af20000  IN   0.5 4165768 186360  dsm_om_connsvcd
   5880   5772   2  ffff88086af24e70  IN   0.5 4165768 186360  dsm_om_connsvcd
   5881   5772   7  ffff88086af21f60  IN   0.5 4165768 186360  dsm_om_connsvcd
   5882   5772   8  ffff88086af22f10  IN   0.5 4165768 186360  dsm_om_connsvcd
   5883   5772   1  ffff88085c729f60  IN   0.5 4165768 186360  dsm_om_connsvcd
   5885   5772   9  ffff88085c72bec0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5886   5772   4  ffff880869f85e20  IN   0.5 4165768 186360  dsm_om_connsvcd
   5908   5772   7  ffff88046b1dce70  IN   0.5 4165768 186360  dsm_om_connsvcd
   5909   5772   0  ffff88046b1dedd0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5910   5772   8  ffff88046b1d9f60  IN   0.5 4165768 186360  dsm_om_connsvcd
   5911   5772   3  ffff88046b1dde20  IN   0.5 4165768 186360  dsm_om_connsvcd
   5912   5772   9  ffff88046b1dbec0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5913   5772   5  ffff88046b1daf10  IN   0.5 4165768 186360  dsm_om_connsvcd
   5914   5772   9  ffff88046c03edd0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5915   5772  11  ffff88046c03ce70  IN   0.5 4165768 186360  dsm_om_connsvcd
   5916   5772   5  ffff88046c03bec0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5918   5772   9  ffff88046c03af10  IN   0.5 4165768 186360  dsm_om_connsvcd
   5921      1   5  ffff88046e3f8000  IN   0.1 1070688  52756  dsm_sa_datamgrd
   5922   5772  10  ffff8800367b4e70  IN   0.5 4165768 186360  dsm_om_connsvcd
   5923   5772   2  ffff8800367b3ec0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5925   5772  11  ffff8800367b0fb0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5926   5772   6  ffff8800367b6dd0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5927   5772   1  ffff880866ec6dd0  IN   0.5 4165768 186360  dsm_om_connsvcd
   5940   5772   8  ffff88046a659f60  IN   0.5 4165768 186360  dsm_om_connsvcd
   5944      2   7  ffff880866ec4e70  IN   0.0       0      0  [kworker/7:1H]
   6069   5772   1  ffff88086a6caf10  IN   0.5 4165768 186360  dsm_om_connsvcd
   6274   5772   3  ffff88046c0cbec0  IN   0.5 4165768 186360  dsm_om_connsvcd
   6707      2   2  ffff88087f76af10  IN   0.0       0      0  [xfs-buf/drbd1]
   6708      2   2  ffff88003645edd0  IN   0.0       0      0  [xfs-data/drbd1]
   6709      2   2  ffff88003645bec0  IN   0.0       0      0  [xfs-conv/drbd1]
   6710      2   2  ffff880036459f60  IN   0.0       0      0  [xfs-cil/drbd1]
   6711      2   2  ffff880036458fb0  IN   0.0       0      0  [xfs-reclaim/drb]
   6712      2   4  ffff88087f84bec0  IN   0.0       0      0  [xfs-log/drbd1]
   6713      2   4  ffff88087f84edd0  IN   0.0       0      0  [xfs-eofblocks/d]
   6714      2   3  ffff88087f849f60  IN   0.0       0      0  [xfsaild/drbd1]
   6857      1   5  ffff88046b688fb0  IN   0.0   42484   1868  rpc.statd
   6872      1  10  ffff88046b68af10  IN   0.0   17148    628  rpc.idmapd
   6873      1   5  ffff88086879af10  IN   0.0   41024   2252  rpc.mountd
   6885      2   5  ffff88087f84de20  IN   0.0       0      0  [nfsd4_callbacks]
   6886      2   5  ffff88087f848000  IN   0.0       0      0  [lockd]
   6894      2   2  ffff88087f848fb0  IN   0.0       0      0  [nfsd]
   6895      2   6  ffff88087f84af10  IN   0.0       0      0  [nfsd]
   6896      2   5  ffff88087f84ce70  IN   0.0       0      0  [nfsd]
   6897      2   0  ffff88086b7d1f60  IN   0.0       0      0  [nfsd]
   6898      2   0  ffff88086b7d3ec0  IN   0.0       0      0  [nfsd]
   6899      2  11  ffff88086b7d0fb0  IN   0.0       0      0  [nfsd]
   6900      2   5  ffff88086b7d2f10  IN   0.0       0      0  [nfsd]
   6901      2   1  ffff88086b7d5e20  IN   0.0       0      0  [nfsd]
   6902      2   1  ffff88086b7d0000  IN   0.0       0      0  [nfsd]
   6903      2   3  ffff88086b7d4e70  IN   0.0       0      0  [nfsd]
   6904      2  10  ffff88086b7d6dd0  IN   0.0       0      0  [nfsd]
   6905      2   9  ffff88086defaf10  IN   0.0       0      0  [nfsd]
   6906      2   6  ffff88086defde20  IN   0.0       0      0  [nfsd]
   6907      2   6  ffff88086defbec0  IN   0.0       0      0  [nfsd]
   6908      2  11  ffff88086def8000  IN   0.0       0      0  [nfsd]
   6909      2   6  ffff88086def8fb0  IN   0.0       0      0  [nfsd]
   6910      2   2  ffff88086defedd0  IN   0.0       0      0  [nfsd]
   6911      2   0  ffff88086def9f60  IN   0.0       0      0  [nfsd]
   6912      2   2  ffff88086c083ec0  IN   0.0       0      0  [nfsd]
   6913      2  11  ffff88086c080000  IN   0.0       0      0  [nfsd]
   6914      2   5  ffff88086c112f10  IN   0.0       0      0  [nfsd]
   6915      2  10  ffff88086c110000  IN   0.0       0      0  [nfsd]
   6916      2   8  ffff8804f973edd0  IN   0.0       0      0  [nfsd]
   6917      2   2  ffff8804f973af10  IN   0.0       0      0  [nfsd]
   6918      2   1  ffff88086df74e70  IN   0.0       0      0  [nfsd]
   6919      2   0  ffff88086df73ec0  IN   0.0       0      0  [nfsd]
   6920      2   7  ffff88086df72f10  IN   0.0       0      0  [nfsd]
   6921      2   8  ffff88086df70000  IN   0.0       0      0  [nfsd]
   6922      2   7  ffff88086df71f60  IN   0.0       0      0  [nfsd]
   6923      2   3  ffff88086df75e20  IN   0.0       0      0  [nfsd]
   6924      2   4  ffff88087f856dd0  IN   0.0       0      0  [nfsd]
   6925      2  10  ffff88087f852f10  IN   0.0       0      0  [nfsd]
   6926      2  11  ffff88087f850fb0  IN   0.0       0      0  [nfsd]
   6927      2  11  ffff88087f855e20  IN   0.0       0      0  [nfsd]
   6928      2   4  ffff88087f850000  IN   0.0       0      0  [nfsd]
   6929      2   3  ffff88087f854e70  IN   0.0       0      0  [nfsd]
   6931      2   4  ffff8808673e2f10  IN   0.0       0      0  [nfsd]
   6932      2  11  ffff8808673e0000  IN   0.0       0      0  [nfsd]
   6933      2   5  ffff88086b643ec0  IN   0.0       0      0  [nfsd]
   6934      2   1  ffff88086b640fb0  IN   0.0       0      0  [nfsd]
   6935      2  11  ffff88086b644e70  IN   0.0       0      0  [nfsd]
   6936      2   0  ffff88086b645e20  IN   0.0       0      0  [nfsd]
   6937      2   7  ffff88086b642f10  IN   0.0       0      0  [nfsd]
   6938      2   8  ffff88086b646dd0  IN   0.0       0      0  [nfsd]
   6939      2   2  ffff88086b640000  IN   0.0       0      0  [nfsd]
   6940      2   2  ffff88085c72af10  IN   0.0       0      0  [nfsd]
   6941      2   0  ffff88085c728000  IN   0.0       0      0  [nfsd]
   6942      2   5  ffff88083a2a0000  IN   0.0       0      0  [nfsd]
   6943      2   6  ffff88083a2a0fb0  IN   0.0       0      0  [nfsd]
   6944      2   0  ffff88083a2a1f60  IN   0.0       0      0  [nfsd]
   6945      2   0  ffff88083a2a2f10  IN   0.0       0      0  [nfsd]
   6946      2   1  ffff88083a2a3ec0  IN   0.0       0      0  [nfsd]
   6947      2   1  ffff88083a2a4e70  IN   0.0       0      0  [nfsd]
   6948      2   0  ffff88083a2a5e20  IN   0.0       0      0  [nfsd]
   6949      2   0  ffff88083a2a6dd0  IN   0.0       0      0  [nfsd]
   6950      2   8  ffff8808486d8000  IN   0.0       0      0  [nfsd]
   6951      2  10  ffff8808486d8fb0  IN   0.0       0      0  [nfsd]
   6952      2   0  ffff8808486d9f60  IN   0.0       0      0  [nfsd]
   6953      2  11  ffff8808486daf10  IN   0.0       0      0  [nfsd]
   6954      2   1  ffff8808486dbec0  IN   0.0       0      0  [nfsd]
   6955      2   8  ffff8808486dce70  IN   0.0       0      0  [nfsd]
   6956      2  10  ffff8808486dde20  IN   0.0       0      0  [nfsd]
   6957      2   8  ffff8808486dedd0  IN   0.0       0      0  [nfsd]
   6958      2   0  ffff88085b250000  IN   0.0       0      0  [nfsd]
   7209   5772   7  ffff88046c0c9f60  IN   0.5 4165768 186360  dsm_om_connsvcd
   7210   5772   2  ffff88046c0c8000  IN   0.5 4165768 186360  dsm_om_connsvcd
   7211   5772   7  ffff88046c0caf10  IN   0.5 4165768 186360  dsm_om_connsvcd
   7212   5772  10  ffff88046c0cedd0  IN   0.5 4165768 186360  dsm_om_connsvcd
   7836      2   0  ffff880834341f60  IN   0.0       0      0  [drbd_a_r1_DRBDL]
   7837      2   2  ffff880834343ec0  IN   0.0       0      0  [drbd_as_r1_DRBD]
   7838      2   1  ffff880834345e20  IN   0.0       0      0  [drbd_a_r0_sdb]
   7839      2   2  ffff880834340000  IN   0.0       0      0  [drbd_as_r0_sdb]
   8131      2   6  ffff88046c243ec0  IN   0.0       0      0  [kworker/6:2H]
  19465      2  10  ffff880179bc9f60  IN   0.0       0      0  [kworker/10:2]
  21159      2   0  ffff8800364a6dd0  IN   0.0       0      0  [kworker/0:2]
  29408      2  11  ffff880868798fb0  IN   0.0       0      0  [kworker/11:2]
  33141      2   1  ffff8804f9d12f10  IN   0.0       0      0  [kworker/u386:0]
  59848      2   9  ffff880866ec2f10  IN   0.0       0      0  [kworker/9:0]
  67321      2   2  ffff88017a1abec0  IN   0.0       0      0  [kworker/2:1]
  67376      2   8  ffff88046cbeedd0  IN   0.0       0      0  [kworker/u385:2]
  67444      2   2  ffff8803c8f31f60  IN   0.0       0      0  [kworker/2:0]
  75178      2   1  ffff88086ce64e70  IN   0.0       0      0  [kworker/1:1]
  78254      2   5  ffff88086a44edd0  IN   0.0       0      0  [kworker/u386:1]
  81689      2   2  ffff88046b19de20  IN   0.0       0      0  [kworker/u385:1]
  82613      2   5  ffff88083123de20  IN   0.0       0      0  [kworker/5:1H]
  86454      2   1  ffff88046b19af10  IN   0.0       0      0  [kworker/u384:0]
  90260      2   5  ffff88086af20fb0  IN   0.0       0      0  [kworker/5:0]
  90327      2   7  ffff8804f94b9f60  IN   0.0       0      0  [kworker/7:0]
  96058      2   6  ffff88046cbeaf10  IN   0.0       0      0  [kworker/6:0]
  97270      2   4  ffff8803d3e2bec0  IN   0.0       0      0  [kworker/4:0]
  109557      2   9  ffff880831238000  IN   0.0       0      0  [kworker/9:2]
  112303      1   8  ffff88086a44af10  IN   0.0  224196  13140  snmpd
  113057      2   6  ffff8803e5af3ec0  IN   0.0       0      0  [kworker/6:1]
  113148      2   8  ffff8803e5af2f10  IN   0.0       0      0  [kworker/8:2]
  113161      2  10  ffff88046cbede20  IN   0.0       0      0  [kworker/10:0]
  114025   3839   3  ffff8803c8f32f10  IN   0.0   89140   4016  pickup
  120604      2   4  ffff88046c8d3ec0  IN   0.0       0      0  [kworker/4:2]
  121449      1   4  ffff880866ec0fb0  IN   0.0  419172   4852  rsyslogd
  121451      1  11  ffff88046a658fb0  IN   0.0  419172   4852  in:imjournal
  121452      1   9  ffff88046a65bec0  IN   0.0  419172   4852  rs:main Q:Reg
  125164      2   1  ffff880868a6edd0  IN   0.0       0      0  [kworker/u384:2]
  126017      2   8  ffff88046cf0edd0  IN   0.0       0      0  [kworker/8:1]
  128263      2   6  ffff8803c8f30000  IN   0.0       0      0  [kworker/6:2]
  128296      2   8  ffff8803c8f33ec0  IN   0.0       0      0  [kworker/8:0]
  129574      2   1  ffff880866ec5e20  IN   0.0       0      0  [kworker/1:0]
  131642      1   2  ffff8803dc384e70  IN   0.1  637348  38052  ruby-timer-thr
  132408      2   7  ffff88085c72ce70  IN   0.0       0      0  [kworker/7:2]
  141714      2   3  ffff88086a44ce70  IN   0.0       0      0  [kworker/3:0]
  155195      2   3  ffff88086c7dce70  IN   0.0       0      0  [kworker/3:1]
  155224      2   5  ffff88086c7d8000  IN   0.0       0      0  [kworker/5:2]
  155284      2  11  ffff8804f940bec0  IN   0.0       0      0  [kworker/11:1]
  157720      2   2  ffff880463f3af10  IN   0.0       0      0  [kworker/2:1H]
  170418      2   0  ffff8803dc383ec0  IN   0.0       0      0  [kworker/0:1]
  183496      2   1  ffff88085b253ec0  IN   0.0       0      0  [kworker/u384:1]
PID: 0      TASK: ffff8804fa559f60  CPU: 3   COMMAND: "swapper/3"
struct task_struct {
  state = 0, 
  stack = 0xffff8804fa680000, 
  usage = {
    counter = 2
  }, 
  flags = 270540864, 
  ptrace = 0, 
  wake_entry = {
    next = 0x0
  }, 
  on_cpu = 1, 
  last_wakee = 0xffff88087f849f60, 
  wakee_flips = 2, 
  wakee_flip_decay_ts = 5087999619, 
  wake_cpu = 3, 
  on_rq = 0, 
  prio = 120, 
  static_prio = 120, 
  normal_prio = 120, 
  rt_priority = 0, 
  sched_class = 0xffffffff816bb520 <idle_sched_class>, 
  se = {
    load = {
      weight = 1024, 
      inv_weight = 4194304
    }, 
    run_node = {
      __rb_parent_color = 1, 
      rb_right = 0x0, 
      rb_left = 0x0
    }, 
    group_node = {
      next = 0xffff8804fa559ff0, 
      prev = 0xffff8804fa559ff0
    }, 
    on_rq = 0, 
    exec_start = 285006861, 
    sum_exec_runtime = 0, 
    vruntime = 0, 
    prev_sum_exec_runtime = 0, 
    nr_migrations = 0, 
    parent = 0x0, 
    cfs_rq = 0xffff88086de56cc0, 
    my_q = 0x0, 
    avg = {
      runnable_avg_sum = 19159, 
      runnable_avg_period = 19207, 
      last_runnable_update = 192115309, 
      decay_count = 184, 
      load_avg_contrib = 1021
    }, 
    {
      statistics = 0xffff8804fa55ae30, 
      __UNIQUE_ID_rh_kabi_hide10 = {
        rh_reserved1 = 18446612153694006832
      }, 
      {<No data fields>}
    }, 
    rh_reserved2 = 0, 
    rh_reserved3 = 0, 
    rh_reserved4 = 0
  }, 
  rt = {
    run_list = {
      next = 0xffff8804fa55a088, 
      prev = 0xffff8804fa55a088
    }, 
    timeout = 0, 
    watchdog_stamp = 0, 
    time_slice = 100, 
    back = 0x0, 
    parent = 0x0, 
    rt_rq = 0xffff88086de56dd0, 
    my_q = 0x0
  }, 
  sched_task_group = 0xffffffff81e3d480 <root_task_group>, 
  preempt_notifiers = {
    first = 0x0
  }, 
  fpu_counter = 120 'x', 
  btrace_seq = 0, 
  policy = 0, 
  nr_cpus_allowed = 1, 
  cpus_allowed = {
    bits = {8, 0, 0, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615}
  }, 
  sched_info = {
    pcount = 0, 
    run_delay = 0, 
    last_arrival = 0, 
    last_queued = 0
  }, 
  tasks = {
    next = 0xffff8804fa5593e0, 
    prev = 0xffffffff819c1890 <init_task+1072>
  }, 
  pushable_tasks = {
    prio = 140, 
    prio_list = {
      next = 0xffff8804fa55a3a8, 
      prev = 0xffff8804fa55a3a8
    }, 
    node_list = {
      next = 0xffff8804fa55a3b8, 
      prev = 0xffff8804fa55a3b8
    }
  }, 
  mm = 0x0, 
  active_mm = 0xffff88046cbd4b00, 
  rss_stat = {
    events = 0, 
    count = {0, 0, 0}
  }, 
  exit_state = 0, 
  exit_code = 0, 
  exit_signal = 0, 
  pdeath_signal = 0, 
  jobctl = 0, 
  personality = 0, 
  did_exec = 0, 
  in_execve = 0, 
  in_iowait = 0, 
  no_new_privs = 0, 
  sched_reset_on_fork = 0, 
  sched_contributes_to_load = 1, 
  pid = 0, 
  tgid = 0, 
  stack_canary = 12140910610751055461, 
  real_parent = 0xffff8804fa558000, 
  parent = 0xffffffff819c1460 <init_task>, 
  children = {
    next = 0xffff8804fa55a428, 
    prev = 0xffff8804fa55a428
  }, 
  sibling = {
    next = 0xffff8804fa55a438, 
    prev = 0xffff8804fa55a438
  }, 
  group_leader = 0xffff8804fa559f60, 
  ptraced = {
    next = 0xffff8804fa5584f0, 
    prev = 0xffff8804fa5584f0
  }, 
  ptrace_entry = {
    next = 0xffff8804fa558500, 
    prev = 0xffff8804fa558500
  }, 
  pids = {{
      node = {
        next = 0x0, 
        pprev = 0x0
      }, 
      pid = 0xffffffff819f88c0 <init_struct_pid>
    }, {
      node = {
        next = 0x0, 
        pprev = 0x0
      }, 
      pid = 0xffffffff819f88c0 <init_struct_pid>
    }, {
      node = {
        next = 0x0, 
        pprev = 0x0
      }, 
      pid = 0xffffffff819f88c0 <init_struct_pid>
    }}, 
  thread_group = {
    next = 0xffff8804fa55a4b8, 
    prev = 0xffff8804fa55a4b8
  }, 
  thread_node = {
    next = 0xffff8804fa593a90, 
    prev = 0xffff8804fa593a90
  }, 
  vfork_done = 0x0, 
  set_child_tid = 0x0, 
  clear_child_tid = 0x0, 
  utime = 0, 
  stime = 4515000000, 
  utimescaled = 0, 
  stimescaled = 4515000000, 
  gtime = 0, 
  prev_cputime = {
    utime = 0, 
    stime = 0
  }, 
  vtime_seqlock = {
    seqcount = {
      sequence = 4
    }, 
    lock = {
      {
        rlock = {
          raw_lock = {
            {
              head_tail = 0, 
              tickets = {
                head = 0, 
                tail = 0
              }
            }
          }
        }
      }
    }
  }, 
  vtime_snap = 4294667549, 
  vtime_snap_whence = VTIME_SYS, 
  nvcsw = 0, 
  nivcsw = 34917496, 
  start_time = {
    tv_sec = 0, 
    tv_nsec = 180000000
  }, 
  real_start_time = {
    tv_sec = 0, 
    tv_nsec = 180000000
  }, 
  min_flt = 0, 
  maj_flt = 0, 
  cputime_expires = {
    utime = 0, 
    stime = 0, 
    sum_exec_runtime = 0
  }, 
  cpu_timers = {{
      next = 0xffff8804fa55a598, 
      prev = 0xffff8804fa55a598
    }, {
      next = 0xffff8804fa55a5a8, 
      prev = 0xffff8804fa55a5a8
    }, {
      next = 0xffff8804fa55a5b8, 
      prev = 0xffff8804fa55a5b8
    }}, 
  real_cred = 0xffff88017fc59d40, 
  cred = 0xffff88017fc59d40, 
  comm = "swapper/3\000\000\000\000\000\000", 
  link_count = 0, 
  total_link_count = 0, 
  sysvsem = {
    undo_list = 0x0
  }, 
  thread = {
    tls_array = {{
        {
          {
            a = 0, 
            b = 0
          }, 
          {
            limit0 = 0, 
            base0 = 0, 
            base1 = 0, 
            type = 0, 
            s = 0, 
            dpl = 0, 
            p = 0, 
            limit = 0, 
            avl = 0, 
            l = 0, 
            d = 0, 
            g = 0, 
            base2 = 0
          }
        }
      }, {
        {
          {
            a = 0, 
            b = 0
          }, 
          {
            limit0 = 0, 
            base0 = 0, 
            base1 = 0, 
            type = 0, 
            s = 0, 
            dpl = 0, 
            p = 0, 
            limit = 0, 
            avl = 0, 
            l = 0, 
            d = 0, 
            g = 0, 
            base2 = 0
          }
        }
      }, {
        {
          {
            a = 0, 
            b = 0
          }, 
          {
            limit0 = 0, 
            base0 = 0, 
            base1 = 0, 
            type = 0, 
            s = 0, 
            dpl = 0, 
            p = 0, 
            limit = 0, 
            avl = 0, 
            l = 0, 
            d = 0, 
            g = 0, 
            base2 = 0
          }
        }
      }}, 
    sp0 = 18446612153695223808, 
    sp = 18446612153695223384, 
    usersp = 0, 
    es = 0, 
    ds = 0, 
    fsindex = 0, 
    gsindex = 0, 
    fs = 0, 
    gs = 0, 
    ptrace_bps = {0x0, 0x0, 0x0, 0x0}, 
    debugreg6 = 0, 
    ptrace_dr7 = 0, 
    cr2 = 20, 
    trap_nr = 14, 
    error_code = 0, 
    fpu = {
      last_cpu = 3, 
      has_fpu = 1, 
      state = 0xffff8804fa582a40
    }, 
    io_bitmap_ptr = 0x0, 
    iopl = 0, 
    io_bitmap_max = 0
  }, 
  fs = 0xffff88017a133e40, 
  files = 0xffff8804fa588780, 
  nsproxy = 0xffffffff819fa120 <init_nsproxy>, 
  signal = 0xffff8804fa593a80, 
  sighand = 0xffff88017a196300, 
  blocked = {
    sig = {0}
  }, 
  real_blocked = {
    sig = {0}
  }, 
  saved_sigmask = {
    sig = {0}
  }, 
  pending = {
    list = {
      next = 0xffff8804fa55a6f0, 
      prev = 0xffff8804fa55a6f0
    }, 
    signal = {
      sig = {0}
    }
  }, 
  sas_ss_sp = 0, 
  sas_ss_size = 0, 
  notifier = 0x0, 
  notifier_data = 0x0, 
  notifier_mask = 0x0, 
  task_works = 0x0, 
  audit_context = 0x0, 
  loginuid = {
    val = 4294967295
  }, 
  sessionid = 4294967295, 
  seccomp = {
    mode = 0, 
    filter = 0x0
  }, 
  parent_exec_id = 0, 
  self_exec_id = 0, 
  alloc_lock = {
    {
      rlock = {
        raw_lock = {
          {
            head_tail = 0, 
            tickets = {
              head = 0, 
              tail = 0
            }
          }
        }
      }
    }
  }, 
  pi_lock = {
    raw_lock = {
      {
        head_tail = 131074, 
        tickets = {
          head = 2, 
          tail = 2
        }
      }
    }
  }, 
  pi_waiters = {
    rb_node = 0x0
  }, 
  pi_waiters_leftmost = 0x0, 
  pi_blocked_on = 0x0, 
  journal_info = 0x0, 
  bio_list = 0x0, 
  plug = 0x0, 
  reclaim_state = 0x0, 
  backing_dev_info = 0x0, 
  io_context = 0x0, 
  ptrace_message = 0, 
  last_siginfo = 0x0, 
  ioac = {
    rchar = 0, 
    wchar = 0, 
    syscr = 0, 
    syscw = 0, 
    read_bytes = 0, 
    write_bytes = 0, 
    cancelled_write_bytes = 0
  }, 
  acct_rss_mem1 = 0, 
  acct_vm_mem1 = 0, 
  acct_timexpd = 0, 
  mems_allowed = {
    bits = {3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
  }, 
  mems_allowed_seq = {
    sequence = 0
  }, 
  cpuset_mem_spread_rotor = -1, 
  cpuset_slab_spread_rotor = -1, 
  cgroups = 0xffffffff81e67080 <init_css_set>, 
  cg_list = {
    next = 0xffff8804fa55a8a8, 
    prev = 0xffff8804fa55a8a8
  }, 
  robust_list = 0x0, 
  compat_robust_list = 0x0, 
  pi_state_list = {
    next = 0xffff8804fa55a8c8, 
    prev = 0xffff8804fa55a8c8
  }, 
  pi_state_cache = 0x0, 
  perf_event_ctxp = {0x0, 0x0}, 
  perf_event_mutex = {
    count = {
      counter = 1
    }, 
    wait_lock = {
      {
        rlock = {
          raw_lock = {
            {
              head_tail = 0, 
              tickets = {
                head = 0, 
                tail = 0
              }
            }
          }
        }
      }
    }, 
    wait_list = {
      next = 0xffff8804fa55a8f8, 
      prev = 0xffff8804fa55a8f8
    }, 
    owner = 0x0, 
    {
      osq = 0x0, 
      __UNIQUE_ID_rh_kabi_hide0 = {
        spin_mlock = 0x0
      }, 
      {<No data fields>}
    }
  }, 
  perf_event_list = {
    next = 0xffff8804fa55a918, 
    prev = 0xffff8804fa55a918
  }, 
  mempolicy = 0xffff88017b880318, 
  il_next = 1, 
  pref_node_fork = 0, 
  numa_scan_seq = 0, 
  numa_scan_period = 1000, 
  numa_scan_period_max = 0, 
  numa_preferred_nid = -1, 
  numa_migrate_retry = 0, 
  node_stamp = 0, 
  last_task_numa_placement = 0, 
  last_sum_exec_runtime = 0, 
  numa_work = {
    next = 0xffff8804fa55a968, 
    func = 0x0
  }, 
  numa_entry = {
    next = 0xffff8804fa55a978, 
    prev = 0xffff8804fa55a978
  }, 
  numa_group = 0x0, 
  numa_faults_memory = 0x0, 
  total_numa_faults = 0, 
  numa_faults_buffer_memory = 0x0, 
  numa_faults_cpu = 0x0, 
  numa_faults_buffer_cpu = 0x0, 
  numa_faults_locality = {0, 0}, 
  numa_pages_migrated = 0, 
  rcu = {
    next = 0x0, 
    func = 0x0
  }, 
  splice_pipe = 0x0, 
  task_frag = {
    page = 0x0, 
    offset = 0, 
    size = 0
  }, 
  delays = 0xffff88017a09e700, 
  nr_dirtied = 0, 
  nr_dirtied_pause = 32, 
  dirty_paused_when = 0, 
  timer_slack_ns = 50000, 
  default_timer_slack_ns = 50000, 
  curr_ret_stack = -1, 
  ret_stack = 0x0, 
  ftrace_timestamp = 0, 
  trace_overrun = {
    counter = 0
  }, 
  tracing_graph_pause = {
    counter = 0
  }, 
  trace = 0, 
  trace_recursion = 0, 
  memcg_batch = {
    do_batch = 0, 
    memcg = 0x0, 
    nr_pages = 0, 
    memsw_nr_pages = 0
  }, 
  memcg_kmem_skip_account = 0, 
  ptrace_bp_refcnt = {
    counter = 1
  }, 
  utask = 0x0, 
  {
    last_switch_count = 0, 
    __UNIQUE_ID_rh_kabi_hide11 = {
      rh_reserved1 = 0
    }, 
    {<No data fields>}
  }, 
  {
    atomic_flags = 0, 
    __UNIQUE_ID_rh_kabi_hide12 = {
      rh_reserved2 = 0
    }, 
    {<No data fields>}
  }, 
  rh_reserved3 = 0, 
  {
    mm_shmempages = 0, 
    __UNIQUE_ID_rh_kabi_hide13 = {
      rh_reserved4 = 0
    }, 
    {<No data fields>}
  }, 
  rh_reserved5 = 0, 
  rh_reserved6 = 0, 
  rh_reserved7 = 0, 
  rh_reserved8 = 0, 
  memcg_oom = {
    memcg = 0x0, 
    gfp_mask = 0, 
    order = 0, 
    may_oom = 0
  }, 
  tlb_ubc = {
    cpumask = {
      bits = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
    }, 
    flush_required = false, 
    writable = false
  }, 
  dl = {
    rb_node = {
      __rb_parent_color = 18446612153694006624, 
      rb_right = 0x0, 
      rb_left = 0x0
    }, 
    dl_runtime = 0, 
    dl_deadline = 0, 
    dl_period = 0, 
    dl_bw = 0, 
    runtime = 0, 
    deadline = 0, 
    flags = 0, 
    dl_throttled = 0, 
    dl_boosted = 0, 
    dl_yielded = 0, 
    dl_timer = {
      node = {
        node = {
          __rb_parent_color = 18446612153694006712, 
          rb_right = 0x0, 
          rb_left = 0x0
        }, 
        expires = {
          tv64 = 0
        }
      }, 
      _softexpires = {
        tv64 = 0
      }, 
      function = 0xffffffff810d7d40 <dl_task_timer>, 
      base = 0xffff88046e60f8e0, 
      state = 0, 
      start_pid = -1, 
      start_site = 0x0, 
      start_comm = "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
    }
  }, 
  pushable_dl_tasks = {
    __rb_parent_color = 18446612153694006808, 
    rb_right = 0x0, 
    rb_left = 0x0
  }, 
  statistics = {
    wait_start = 0, 
    wait_max = 0, 
    wait_count = 0, 
    wait_sum = 0, 
    iowait_count = 0, 
    iowait_sum = 0, 
    sleep_start = 0, 
    sleep_max = 0, 
    sum_sleep_runtime = 0, 
    block_start = 0, 
    block_max = 0, 
    exec_max = 0, 
    slice_max = 0, 
    nr_migrations_cold = 0, 
    nr_failed_migrations_affine = 0, 
    nr_failed_migrations_running = 0, 
    nr_failed_migrations_hot = 0, 
    nr_forced_migrations = 0, 
    nr_wakeups = 0, 
    nr_wakeups_sync = 0, 
    nr_wakeups_migrate = 0, 
    nr_wakeups_local = 0, 
    nr_wakeups_remote = 0, 
    nr_wakeups_affine = 0, 
    nr_wakeups_affine_attempts = 0, 
    nr_wakeups_passive = 0, 
    nr_wakeups_idle = 0
  }
}

struct thread_info {
  task = 0xffff8804fa559f60, 
  exec_domain = 0xffffffff819eb6e0 <default_exec_domain>, 
  flags = 0, 
  status = 0, 
  cpu = 3, 
  preempt_count = 65536, 
  addr_limit = {
    seg = 18446744073709551615
  }, 
  restart_block = {
    fn = 0xffffffff8109dcf0 <do_no_restart_syscall>, 
    {
      futex = {
        uaddr = 0x0, 
        val = 0, 
        flags = 0, 
        bitset = 0, 
        time = 0, 
        uaddr2 = 0x0
      }, 
      nanosleep = {
        clockid = 0, 
        rmtp = 0x0, 
        compat_rmtp = 0x0, 
        expires = 0
      }, 
      poll = {
        ufds = 0x0, 
        nfds = 0, 
        has_timeout = 0, 
        tv_sec = 0, 
        tv_nsec = 0
      }
    }
  }, 
  sysenter_return = 0x0, 
  sig_on_uaccess_error = 0, 
  uaccess_err = 0
}

PID: 0      TASK: ffff8804fa559f60  CPU: 3   COMMAND: "swapper/3"
       MM               PGD          RSS    TOTAL_VM
       0                 0            0k       0k   
crash (50,229 bytes)
adip

adip

2017-04-16 00:33

reporter   ~0029075

KERNEL: /data/adi/crash/3.10.0-514.10.2.el7.x86_64/vmlinux
    DUMPFILE: /data/adi/crash/127.0.0.1-2017-04-15-02:53:06/vmcore [PARTIAL DUMP]
        CPUS: 12
        DATE: Sat Apr 15 02:52:55 2017
      UPTIME: 9 days, 04:22:12
LOAD AVERAGE: 0.08, 0.08, 0.05
       TASKS: 416
    NODENAME: xd3
     RELEASE: 3.10.0-514.10.2.el7.x86_64
     VERSION: #1 SMP Fri Mar 3 00:04:05 UTC 2017
     MACHINE: x86_64 (1700 Mhz)
      MEMORY: 31.9 GB
       PANIC: ".1BUG: unable to handle kernel NULL pointer dereference at 0000000000000014"
         PID: 0
     COMMAND: "swapper/3"
        TASK: ffff8804fa559f60 (1 of 12) [THREAD_INFO: ffff8804fa680000]
         CPU: 3
       STATE: TASK_RUNNING (PANIC)
adip

adip

2017-04-16 00:36

reporter   ~0029076

Reported upstream, see https://bugzilla.redhat.com/show_bug.cgi?id=1442593
tigalch

tigalch

2017-04-18 09:58

manager   ~0029085

Kernel got updated. What is the state of the above mentioned bugzilla issue (since kernel issues are usually private)?
adip

adip

2017-04-18 10:34

reporter   ~0029086

The bug was closed because of out-of-tree loading of drbd module. Quoting:

"A few things: This looks like a bug in which a task has disappeared. The dereference of p->flags in account_system_time() is what is causing the panic, ie) p is null.

A quick search for similar panics doesn't yield anything, so this looks like it is something specific to your system. I did notice that you've installed an out-of-tree driver which may be responsible for this problem. I cannot debug an issue like this where out-of-tree modules have been loaded as I have no idea about the stability of the code.

If you can reproduce the failure without the drbd, please reopen this bug."

The node is now running kernel 3.10.0-514.16.1.el7.x86_64.
tigalch

tigalch

2017-04-18 10:36

manager   ~0029087

So, can you help RH in that matter, or not?
Else we will close this issue as well, since the fix has to come from RH.
adip

adip

2017-04-18 10:37

reporter   ~0029088

Please close it, thanks.
tigalch

tigalch

2017-04-18 10:39

manager   ~0029089

Closing per reporters feedback

Issue History

Date Modified Username Field Change
2017-04-16 00:20 adip New Issue
2017-04-16 00:20 adip File Added: crash
2017-04-16 00:33 adip Note Added: 0029075
2017-04-16 00:36 adip Note Added: 0029076
2017-04-18 09:58 tigalch Note Added: 0029085
2017-04-18 10:34 adip Note Added: 0029086
2017-04-18 10:36 tigalch Note Added: 0029087
2017-04-18 10:37 adip Note Added: 0029088
2017-04-18 10:39 tigalch Status new => closed
2017-04-18 10:39 tigalch Resolution open => unable to reproduce
2017-04-18 10:39 tigalch Note Added: 0029089