View Issue Details

IDProjectCategoryView StatusLast Update
0006317CentOS-6kernelpublic2013-03-14 06:46
Reportersivic 
PriorityurgentSeveritycrashReproducibilityrandom
Status newResolutionopen 
Product Version6.4 
Target VersionFixed in Version 
Summary0006317: Random kernel panic on kernel 2.6.32-358.0.1.el6.x86_64 and 2.6.32-279.22.1.el6.x86_64
DescriptionOn CentOS 6.3 and 6.4 kernel panics after apparently random amount of time ranging from 24 hours to two weeks.

Crash dump for 2.6.32-358.0.1.el6.x86_64

      KERNEL: /usr/lib/debug/lib/modules/2.6.32-358.0.1.el6.x86_64/vmlinux
    DUMPFILE: vmcore [PARTIAL DUMP]
        CPUS: 4
        DATE: Wed Mar 13 18:11:41 2013
      UPTIME: 3 days, 23:55:36
LOAD AVERAGE: 0.00, 0.00, 0.00
       TASKS: 287
    NODENAME: FS-01
     RELEASE: 2.6.32-358.0.1.el6.x86_64
     VERSION: #1 SMP Wed Feb 27 06:06:45 UTC 2013
     MACHINE: x86_64 (2660 Mhz)
      MEMORY: 4 GB
       PANIC: "Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3"
         PID: 29503
     COMMAND: "freeswitch"
        TASK: ffff880136782040 [THREAD_INFO: ffff880013008000]
         CPU: 3
       STATE: TASK_INTERRUPTIBLE (PANIC)

crash> bt
PID: 29503 TASK: ffff880136782040 CPU: 3 COMMAND: "freeswitch"
 #0 [ffff880028387b30] machine_kexec at ffffffff81035b7b
 #1 [ffff880028387b90] crash_kexec at ffffffff810c0dd2
 #2 [ffff880028387c60] panic at ffffffff8150d0af
 #3 [ffff880028387ce0] watchdog_overflow_callback at ffffffff810e11dd
 #4 [ffff880028387d00] __perf_event_overflow at ffffffff81116fe0
 #5 [ffff880028387da0] perf_event_overflow at ffffffff81117604
 #6 [ffff880028387db0] intel_pmu_handle_irq at ffffffff810207e2
 #7 [ffff880028387e90] perf_event_nmi_handler at ffffffff815117f9
 #8 [ffff880028387ea0] notifier_call_chain at ffffffff815132b5
 #9 [ffff880028387ee0] atomic_notifier_call_chain at ffffffff8151331a
#10 [ffff880028387ef0] notify_die at ffffffff8109cc1e
#11 [ffff880028387f20] do_nmi at ffffffff81510f7b
#12 [ffff880028387f50] nmi at ffffffff81510840
    [exception RIP: _spin_lock+30]
    RIP: ffffffff815100ae RSP: ffff880013009ab8 RFLAGS: 00000093
    RAX: 000000000000105a RBX: ffff880028396700 RCX: 000000000000003c
    RDX: 0000000000001058 RSI: ffff880028296700 RDI: ffff880028296700
    RBP: ffff880013009ab8 R8: ffff88013dc65200 R9: 0000000000000000
    R10: 0000000000000000 R11: 00000000000e7ef0 R12: ffff880028296700
    R13: 0000000000000003 R14: 0000000000000001 R15: ffff880028296700
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#13 [ffff880013009ab8] _spin_lock at ffffffff815100ae
#14 [ffff880013009ac0] double_lock_balance at ffffffff81055bda
#15 [ffff880013009ae0] pull_rt_task at ffffffff81064124
#16 [ffff880013009b30] pre_schedule_rt at ffffffff810642d5
#17 [ffff880013009b40] schedule at ffffffff8150d4f2
#18 [ffff880013009c00] futex_wait_queue_me at ffffffff810a9af9
#19 [ffff880013009c40] futex_wait at ffffffff810aac08
#20 [ffff880013009dc0] do_futex at ffffffff810ac371
#21 [ffff880013009ef0] sys_futex at ffffffff810ace2b
#22 [ffff880013009f80] system_call_fastpath at ffffffff8100b072
    RIP: 00007f66f9b8743c RSP: 00007f66ae5a33b0 RFLAGS: 00000282
    RAX: 00000000000000ca RBX: ffffffff8100b072 RCX: 00000000fffffff8
    RDX: 00000000747f507a RSI: 0000000000000080 RDI: 0000000001d79254
    RBP: 00007f66bc48cea8 R8: 0000000000000000 R9: 000000003a3fa82e
    R10: 0000000000000000 R11: 0000000000000246 R12: 00007f66fa606df0
    R13: 0000000000000320 R14: 00007f66fa606de0 R15: 00007f66fa606de0
    ORIG_RAX: 00000000000000ca CS: 0033 SS: 002b

Crash dump for 2.6.32-279.22.1.el6.x86_64

      KERNEL: /usr/lib/debug/lib/modules/2.6.32-279.22.1.el6.x86_64/vmlinux
    DUMPFILE: vmcore [PARTIAL DUMP]
        CPUS: 4
        DATE: Fri Mar 8 17:35:52 2013
      UPTIME: 1 days, 08:05:58
LOAD AVERAGE: 0.01, 0.05, 0.02
       TASKS: 305
    NODENAME: FS-01
     RELEASE: 2.6.32-279.22.1.el6.x86_64
     VERSION: #1 SMP Wed Feb 6 03:10:46 UTC 2013
     MACHINE: x86_64 (2659 Mhz)
      MEMORY: 4 GB
       PANIC: "Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3"
         PID: 23875
     COMMAND: "freeswitch"
        TASK: ffff8800373ad540 [THREAD_INFO: ffff88003720e000]
         CPU: 3
       STATE: TASK_INTERRUPTIBLE (PANIC)
       
crash> bt
PID: 23875 TASK: ffff8800373ad540 CPU: 3 COMMAND: "freeswitch"
 #0 [ffff880028387b00] machine_kexec at ffffffff81031f7b
 #1 [ffff880028387b60] crash_kexec at ffffffff810b8e72
 #2 [ffff880028387c30] panic at ffffffff814e990a
 #3 [ffff880028387cb0] watchdog_overflow_callback at ffffffff810d881d
 #4 [ffff880028387cd0] __perf_event_overflow at ffffffff8110a63d
 #5 [ffff880028387d70] perf_event_overflow at ffffffff8110abf4
 #6 [ffff880028387d80] intel_pmu_handle_irq at ffffffff8101e9e6
 #7 [ffff880028387e90] perf_event_nmi_handler at ffffffff814edfb9
 #8 [ffff880028387ea0] notifier_call_chain at ffffffff814efb05
 #9 [ffff880028387ee0] atomic_notifier_call_chain at ffffffff814efb6a
#10 [ffff880028387ef0] notify_die at ffffffff81096bfe
#11 [ffff880028387f20] do_nmi at ffffffff814ed783
#12 [ffff880028387f50] nmi at ffffffff814ed090
    [exception RIP: _spin_lock+30]
    RIP: ffffffff814ec8fe RSP: ffff88003720fab8 RFLAGS: 00000093
    RAX: 000000000000711b RBX: ffff880028396680 RCX: 000000000000003c
    RDX: 0000000000007119 RSI: ffff880028216680 RDI: ffff880028216680
    RBP: ffff88003720fab8 R8: ffff88013cc64a00 R9: 0000000000000000
    R10: 0000000000000000 R11: dead000000200200 R12: ffff880028216680
    R13: 0000000000000003 R14: 0000000000000000 R15: ffff880028216680
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#13 [ffff88003720fab8] _spin_lock at ffffffff814ec8fe
#14 [ffff88003720fac0] double_lock_balance at ffffffff8105234a
#15 [ffff88003720fae0] pull_rt_task at ffffffff81060844
#16 [ffff88003720fb30] pre_schedule_rt at ffffffff810609f5
#17 [ffff88003720fb40] schedule at ffffffff814e9d42
#18 [ffff88003720fc00] futex_wait_queue_me at ffffffff810a24e9
#19 [ffff88003720fc40] futex_wait at ffffffff810a35f8
#20 [ffff88003720fdc0] do_futex at ffffffff810a4d61
#21 [ffff88003720fef0] sys_futex at ffffffff810a581b
#22 [ffff88003720ff80] system_call_fastpath at ffffffff8100b072
    RIP: 00007ff9f58f943c RSP: 00007ff99a633fa0 RFLAGS: 00000246
    RAX: 00000000000000ca RBX: ffffffff8100b072 RCX: 0000000000000000
    RDX: 000000003aa53fa8 RSI: 0000000000000080 RDI: 0000000001341354
    RBP: 00007ff98ca19548 R8: 0000000000000000 R9: 000000001d529fd2
    R10: 0000000000000000 R11: 0000000000000246 R12: 00007ff9f6378df0
    R13: 0000000000000320 R14: 00007ff9f6378de0 R15: 00007ff9f6378de0
    ORIG_RAX: 00000000000000ca CS: 0033 SS: 002b
TagsNo tags attached.

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2013-03-14 06:46 sivic New Issue