View Issue Details

IDProjectCategoryView StatusLast Update
0016722CentOS-7crashpublic2019-11-13 08:37
Reporterxqjcool@163.com 
PriorityhighSeverityblockReproducibilityhave not tried
Status newResolutionopen 
Product Version7.5.1804 
Target VersionFixed in Version 
Summary0016722: Kernel panic - not syncing: Hard LOCKUP
Descriptionwhen 'run_timer_softirq'(own tvec_base->lock) is running in softirq, page_fault interrupt it. finally call 'mod_timer' which want to own tvec_base->lock. That caused hard lock.
Steps To Reproducedidn't do the reproduce
TagsHard LOCKUP
abrt_hash
URL

Activities

xqjcool@163.com

xqjcool@163.com

2019-11-12 08:25

reporter  

111crash.txt (4,504 bytes)
PID: 1336   TASK: ffff8800364cbec0  CPU: 1   COMMAND: "rmmod"
 #0 [ffff88081f285e48] crash_nmi_callback at ffffffff8104d342
 #1 [ffff88081f285e58] nmi_handle at ffffffff816901b7
 #2 [ffff88081f285eb0] do_nmi at ffffffff816903c3
 #3 [ffff88081f285ef0] end_repeat_nmi at ffffffff8168f5d3
    [exception RIP: _raw_spin_lock_irqsave+71]
    RIP: ffffffff8168eb47  RSP: ffff88081f2839a8  RFLAGS: 00000002
    RAX: 000000000000463d  RBX: ffff880174a50000  RCX: 0000000000000ca4
    RDX: 0000000000000ca2  RSI: 0000000000000ca4  RDI: ffff880174a50000
    RBP: ffff88081f2839a8   R8: 0000000000000086   R9: 0000000000000000
    R10: 0000000000000000  R11: ffff88081f2839fe  R12: ffff880174a50000
    R13: ffff8800364822f0  R14: ffff88081f2839e8  R15: 0000000000000e20
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #4 [ffff88081f2839a8] _raw_spin_lock_irqsave at ffffffff8168eb47
 #5 [ffff88081f2839b0] lock_timer_base at ffffffff8109700b
 #6 [ffff88081f2839e0] mod_timer at ffffffff81099234
 #7 [ffff88081f283a28] add_timer at ffffffff810993e8
 #8 [ffff88081f283a38] fbcon_add_cursor_timer at ffffffff81381a69
 #9 [ffff88081f283a60] fbcon_cursor at ffffffff81384c2a
#10 [ffff88081f283ab0] hide_cursor at ffffffff813f70b8
#11 [ffff88081f283ac8] vt_console_print at ffffffff813f8ae8
#12 [ffff88081f283b30] call_console_drivers.constprop.17 at ffffffff81087011
#13 [ffff88081f283b58] console_unlock at ffffffff810888b8
#14 [ffff88081f283b98] vprintk_emit at ffffffff81088df4
#15 [ffff88081f283c08] vprintk_default at ffffffff81089169
#16 [ffff88081f283c18] printk at ffffffff816806db
#17 [ffff88081f283c78] no_context at ffffffff8167fbbe
#18 [ffff88081f283cc8] __bad_area_nosemaphore at ffffffff8167fd2c
#19 [ffff88081f283d10] bad_area_nosemaphore at ffffffff8167fe96
#20 [ffff88081f283d20] __do_page_fault at ffffffff81692e4e
#21 [ffff88081f283d80] do_page_fault at ffffffff81692ff5
#22 [ffff88081f283db0] page_fault at ffffffff8168f208
    [exception RIP: run_timer_softirq+248]
    RIP: ffffffff81098a68  RSP: ffff88081f283e68  RFLAGS: 00010086
    RAX: ffff880174a50ee8  RBX: ffff880174a50000  RCX: ffffffffa889b260
    RDX: ffff880174a50ec0  RSI: ffff88081f283e90  RDI: ffff880174a50000
    RBP: ffff88081f283ed0   R8: 0000f005671608c0   R9: ffff88081f283d98
    R10: 0000000000000000  R11: ffff88081f283da0  R12: 00000000000000ec
    R13: 0000000000000001  R14: ffffffff819b70c8  R15: 0000000000000101
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#23 [ffff88081f283e60] run_timer_softirq at ffffffff810989b3
#24 [ffff88081f283ed8] __do_softirq at ffffffff8108f63f
#25 [ffff88081f283f48] call_softirq at ffffffff8169929c
#26 [ffff88081f283f60] do_softirq at ffffffff8102d365
#27 [ffff88081f283f80] irq_exit at ffffffff8108f9d5
#28 [ffff88081f283f98] smp_apic_timer_interrupt at ffffffff81699f15
#29 [ffff88081f283fb0] apic_timer_interrupt at ffffffff8169845d
--- <IRQ stack> ---
#30 [ffff8800d0e17d68] apic_timer_interrupt at ffffffff8169845d
    [exception RIP: free_pages_prepare+246]
    RIP: ffffffff811893f6  RSP: ffff8800d0e17e18  RFLAGS: 00000282
    RAX: 001fffff00000001  RBX: ffffffff81189cab  RCX: fff00000ffffffff
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: ffffea00034737c0
    RBP: ffff8800d0e17e20   R8: fff00000fe000000   R9: ffff88081f1b4340
    R10: 0000000000000038  R11: 0000000000000001  R12: 0000000000000002
    R13: ffffea0003318d80  R14: ffffea0003318d80  R15: 0000000000000282
    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
#31 [ffff8800d0e17e28] free_hot_cold_page at ffffffff81189ee4
#32 [ffff8800d0e17e60] __free_pages at ffffffff8118a095
#33 [ffff8800d0e17e70] __vunmap at ffffffff811c01a3
#34 [ffff8800d0e17e98] vfree at ffffffff811c02a6
#35 [ffff8800d0e17eb0] module_free at ffffffff810fdf71
#36 [ffff8800d0e17ec0] free_module at ffffffff810fe20d
#37 [ffff8800d0e17ef8] sys_delete_module at ffffffff810fe414
#38 [ffff8800d0e17f80] system_call_fastpath at ffffffff81697809
    RIP: 00007f7430810c57  RSP: 00007ffc0a2be808  RFLAGS: 00010246
    RAX: 00000000000000b0  RBX: ffffffff81697809  RCX: 0000000000000000
    RDX: 00007f7430881040  RSI: 0000000000000800  RDI: 00000000020a9338
    RBP: 0000000000000000   R8: 00007f7430ad4060   R9: 00007f7430881040
    R10: 00007ffc0a2be560  R11: 0000000000000206  R12: 0000000000000000
    R13: 00007ffc0a2bef64  R14: 00000000020a92d0  R15: 00000000b11753f2
    ORIG_RAX: 00000000000000b0  CS: 0033  SS: 002b
111crash.txt (4,504 bytes)
xqjcool@163.com

xqjcool@163.com

2019-11-12 08:26

reporter   ~0035676

PID: 1336 TASK: ffff8800364cbec0 CPU: 1 COMMAND: "rmmod"
 #0 [ffff88081f285e48] crash_nmi_callback at ffffffff8104d342
 #1 [ffff88081f285e58] nmi_handle at ffffffff816901b7
 #2 [ffff88081f285eb0] do_nmi at ffffffff816903c3
 #3 [ffff88081f285ef0] end_repeat_nmi at ffffffff8168f5d3
    [exception RIP: _raw_spin_lock_irqsave+71]
    RIP: ffffffff8168eb47 RSP: ffff88081f2839a8 RFLAGS: 00000002
    RAX: 000000000000463d RBX: ffff880174a50000 RCX: 0000000000000ca4
    RDX: 0000000000000ca2 RSI: 0000000000000ca4 RDI: ffff880174a50000
    RBP: ffff88081f2839a8 R8: 0000000000000086 R9: 0000000000000000
    R10: 0000000000000000 R11: ffff88081f2839fe R12: ffff880174a50000
    R13: ffff8800364822f0 R14: ffff88081f2839e8 R15: 0000000000000e20
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
 #4 [ffff88081f2839a8] _raw_spin_lock_irqsave at ffffffff8168eb47
 #5 [ffff88081f2839b0] lock_timer_base at ffffffff8109700b
 #6 [ffff88081f2839e0] mod_timer at ffffffff81099234
 #7 [ffff88081f283a28] add_timer at ffffffff810993e8
 #8 [ffff88081f283a38] fbcon_add_cursor_timer at ffffffff81381a69
 #9 [ffff88081f283a60] fbcon_cursor at ffffffff81384c2a
#10 [ffff88081f283ab0] hide_cursor at ffffffff813f70b8
#11 [ffff88081f283ac8] vt_console_print at ffffffff813f8ae8
#12 [ffff88081f283b30] call_console_drivers.constprop.17 at ffffffff81087011
#13 [ffff88081f283b58] console_unlock at ffffffff810888b8
#14 [ffff88081f283b98] vprintk_emit at ffffffff81088df4
#15 [ffff88081f283c08] vprintk_default at ffffffff81089169
#16 [ffff88081f283c18] printk at ffffffff816806db
#17 [ffff88081f283c78] no_context at ffffffff8167fbbe
#18 [ffff88081f283cc8] __bad_area_nosemaphore at ffffffff8167fd2c
#19 [ffff88081f283d10] bad_area_nosemaphore at ffffffff8167fe96
#20 [ffff88081f283d20] __do_page_fault at ffffffff81692e4e
#21 [ffff88081f283d80] do_page_fault at ffffffff81692ff5
#22 [ffff88081f283db0] page_fault at ffffffff8168f208
    [exception RIP: run_timer_softirq+248]
    RIP: ffffffff81098a68 RSP: ffff88081f283e68 RFLAGS: 00010086
    RAX: ffff880174a50ee8 RBX: ffff880174a50000 RCX: ffffffffa889b260
    RDX: ffff880174a50ec0 RSI: ffff88081f283e90 RDI: ffff880174a50000
    RBP: ffff88081f283ed0 R8: 0000f005671608c0 R9: ffff88081f283d98
    R10: 0000000000000000 R11: ffff88081f283da0 R12: 00000000000000ec
    R13: 0000000000000001 R14: ffffffff819b70c8 R15: 0000000000000101
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#23 [ffff88081f283e60] run_timer_softirq at ffffffff810989b3
#24 [ffff88081f283ed8] __do_softirq at ffffffff8108f63f
#25 [ffff88081f283f48] call_softirq at ffffffff8169929c
#26 [ffff88081f283f60] do_softirq at ffffffff8102d365
#27 [ffff88081f283f80] irq_exit at ffffffff8108f9d5
#28 [ffff88081f283f98] smp_apic_timer_interrupt at ffffffff81699f15
#29 [ffff88081f283fb0] apic_timer_interrupt at ffffffff8169845d
--- <IRQ stack> ---
#30 [ffff8800d0e17d68] apic_timer_interrupt at ffffffff8169845d
    [exception RIP: free_pages_prepare+246]
    RIP: ffffffff811893f6 RSP: ffff8800d0e17e18 RFLAGS: 00000282
    RAX: 001fffff00000001 RBX: ffffffff81189cab RCX: fff00000ffffffff
    RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffea00034737c0
    RBP: ffff8800d0e17e20 R8: fff00000fe000000 R9: ffff88081f1b4340
    R10: 0000000000000038 R11: 0000000000000001 R12: 0000000000000002
    R13: ffffea0003318d80 R14: ffffea0003318d80 R15: 0000000000000282
    ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018
#31 [ffff8800d0e17e28] free_hot_cold_page at ffffffff81189ee4
#32 [ffff8800d0e17e60] __free_pages at ffffffff8118a095
#33 [ffff8800d0e17e70] __vunmap at ffffffff811c01a3
#34 [ffff8800d0e17e98] vfree at ffffffff811c02a6
#35 [ffff8800d0e17eb0] module_free at ffffffff810fdf71
#36 [ffff8800d0e17ec0] free_module at ffffffff810fe20d
#37 [ffff8800d0e17ef8] sys_delete_module at ffffffff810fe414
#38 [ffff8800d0e17f80] system_call_fastpath at ffffffff81697809
    RIP: 00007f7430810c57 RSP: 00007ffc0a2be808 RFLAGS: 00010246
    RAX: 00000000000000b0 RBX: ffffffff81697809 RCX: 0000000000000000
    RDX: 00007f7430881040 RSI: 0000000000000800 RDI: 00000000020a9338
    RBP: 0000000000000000 R8: 00007f7430ad4060 R9: 00007f7430881040
    R10: 00007ffc0a2be560 R11: 0000000000000206 R12: 0000000000000000
    R13: 00007ffc0a2bef64 R14: 00000000020a92d0 R15: 00000000b11753f2
    ORIG_RAX: 00000000000000b0 CS: 0033 SS: 002b
xqjcool@163.com

xqjcool@163.com

2019-11-13 08:37

reporter   ~0035682

Problem was solved.
This is because the module where the timer is located has been released. At this time, the timer is triggered and the access address is abnormal.

Issue History

Date Modified Username Field Change
2019-11-12 08:25 xqjcool@163.com New Issue
2019-11-12 08:25 xqjcool@163.com File Added: 111crash.txt
2019-11-12 08:25 xqjcool@163.com Tag Attached: Hard LOCKUP
2019-11-12 08:26 xqjcool@163.com Note Added: 0035676
2019-11-13 08:37 xqjcool@163.com Note Added: 0035682