View Issue Details

IDProjectCategoryView StatusLast Update
0017364CentOS-7kernelpublic2020-05-15 11:56
Reporterhejiwen 
PriorityhighSeveritymajorReproducibilityunable to reproduce
Status closedResolutionnot fixable 
Product Version7.4.1708 
Target VersionFixed in Version 
Summary0017364: remove nvme disk, kernle crash
Descriptionwhen remove nvme disk by hand, the kernel crash and the machine reboots.
1、the vmcore backtrace as follows:
PID: 8437 TASK: ffff882fdbe12f70 CPU: 17 COMMAND: "kworker/17:1H"
 #0 [ffff882dee6279d8] machine_kexec at ffffffff8105ea1b
 #1 [ffff882dee627a38] __crash_kexec at ffffffff8110bc72
 #2 [ffff882dee627b08] crash_kexec at ffffffff8110bd60
 #3 [ffff882dee627b20] oops_end at ffffffff816c0758
 #4 [ffff882dee627b48] no_context at ffffffff816af3f9
 #5 [ffff882dee627b98] __bad_area_nosemaphore at ffffffff816af48f
 #6 [ffff882dee627be0] bad_area_nosemaphore at ffffffff816af5f9
 #7 [ffff882dee627bf0] __do_page_fault at ffffffff816c366e
 #8 [ffff882dee627c50] do_page_fault at ffffffff816c37f5
 #9 [ffff882dee627c80] page_fault at ffffffff816bf768
    [exception RIP: native_queued_spin_lock_slowpath+272]
    RIP: ffffffff811014e0 RSP: ffff882dee627d30 RFLAGS: 00010202
    RAX: 0000000000001b86 RBX: ffff882fd2b24000 RCX: 0000000000890000
    RDX: 0000000000019670 RSI: 00000000dc374960 RDI: ffff882fdc374960
    RBP: ffff882dee627d30 R8: ffff882fddb59640 R9: 0000000000000000
    R10: df403d66cad241f8 R11: 0000000000000000 R12: ffff882dee627d78
    R13: ffff882fdc374960 R14: 000000000000001d R15: ffff882fdc372080
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff882dee627d38] queued_spin_lock_slowpath at ffffffff816b0773
#11 [ffff882dee627d48] _raw_spin_lock at ffffffff816be0d0
#12 [ffff882dee627d58] blk_mq_process_rq_list at ffffffff8130eaef
#13 [ffff882dee627de8] __blk_mq_run_hw_queue at ffffffff8130ee2a
#14 [ffff882dee627e10] blk_mq_run_work_fn at ffffffff8130f675
#15 [ffff882dee627e20] process_one_work at ffffffff810af51f
#16 [ffff882dee627e68] worker_thread at ffffffff810b01e6
#17 [ffff882dee627ec8] kthread at ffffffff810b74d1
#18 [ffff882dee627f50] ret_from_fork at ffffffff816c851d

2、the contents of vmcore-dmesg.txt as follows:
[20376.401773] pciehp 0000:d7:00.0:pcie04: Slot(51-1): Link Down
[20376.401872] nvme0n1: detected capacity change from 6401252745216 to 0
[20376.402980] Buffer I/O error on dev nvme0n1p12, logical block 129688312, lost async page write
[20376.403052] Buffer I/O error on dev nvme0n1p12, logical block 129688313, lost async page write
[20376.403102] Buffer I/O error on dev nvme0n1p12, logical block 129688314, lost async page write
[20376.403153] Buffer I/O error on dev nvme0n1p12, logical block 129688315, lost async page write
[20376.403201] Buffer I/O error on dev nvme0n1p12, logical block 129688316, lost async page write
[20376.403250] Buffer I/O error on dev nvme0n1p12, logical block 129688317, lost async page write
[20376.403302] Buffer I/O error on dev nvme0n1p12, logical block 129688318, lost async page write
[20376.403397] Buffer I/O error on dev nvme0n1p12, logical block 129688319, lost async page write
[20376.403448] Buffer I/O error on dev nvme0n1p12, logical block 129688320, lost async page write
[20376.403499] Buffer I/O error on dev nvme0n1p12, logical block 129688321, lost async page write
[20376.404330] XFS (nvme0n1p12): metadata I/O error: block 0x3dc784d0 ("xlog_iodone") error 5 numblks 64
[20376.404385] XFS (nvme0n1p12): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.404395] XFS (nvme0n1p12): Log I/O Error Detected. Shutting down filesystem
[20376.404431] XFS (nvme0n1p12): Please umount the filesystem and rectify the problem(s)
[20376.407140] XFS (nvme0n1p11): metadata I/O error: block 0x3dc78bb8 ("xlog_iodone") error 5 numblks 64
[20376.407177] XFS (nvme0n1p11): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.407184] XFS (nvme0n1p11): Log I/O Error Detected. Shutting down filesystem
[20376.407207] XFS (nvme0n1p11): Please umount the filesystem and rectify the problem(s)
[20376.407962] XFS (nvme0n1p10): metadata I/O error: block 0x3dc79378 ("xlog_iodone") error 5 numblks 64
[20376.408007] XFS (nvme0n1p10): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.408022] XFS (nvme0n1p10): Log I/O Error Detected. Shutting down filesystem
[20376.408053] XFS (nvme0n1p10): Please umount the filesystem and rectify the problem(s)
[20376.408613] XFS (nvme0n1p9): metadata I/O error: block 0x3dc781f8 ("xlog_iodone") error 5 numblks 64
[20376.408644] XFS (nvme0n1p9): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.408650] XFS (nvme0n1p9): Log I/O Error Detected. Shutting down filesystem
[20376.408671] XFS (nvme0n1p9): Please umount the filesystem and rectify the problem(s)
[20376.409549] XFS (nvme0n1p8): metadata I/O error: block 0x3dc777c0 ("xlog_iodone") error 5 numblks 64
[20376.409581] XFS (nvme0n1p8): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.409586] XFS (nvme0n1p8): Log I/O Error Detected. Shutting down filesystem
[20376.409607] XFS (nvme0n1p8): Please umount the filesystem and rectify the problem(s)
[20376.410248] XFS (nvme0n1p7): metadata I/O error: block 0x3dc79d30 ("xlog_iodone") error 5 numblks 64
[20376.410278] XFS (nvme0n1p7): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.410283] XFS (nvme0n1p7): Log I/O Error Detected. Shutting down filesystem
[20376.410304] XFS (nvme0n1p7): Please umount the filesystem and rectify the problem(s)
[20377.173241] nvme 0000:d8:00.0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
[20377.173890] blk_update_request: I/O error, dev nvme0n1, sector 11776776
[20377.173893] blk_update_request: I/O error, dev nvme0n1, sector 22058688
[20377.174014] blk_update_request: I/O error, dev nvme0n1, sector 11777032
[20377.174090] blk_update_request: I/O error, dev nvme0n1, sector 36140400
[20377.174195] blk_update_request: I/O error, dev nvme0n1, sector 36140656
[20377.174256] blk_update_request: I/O error, dev nvme0n1, sector 36140912
[20377.174320] blk_update_request: I/O error, dev nvme0n1, sector 36141168
[20377.174362] blk_update_request: I/O error, dev nvme0n1, sector 36141424
[20377.174401] blk_update_request: I/O error, dev nvme0n1, sector 36141440
[20377.174442] blk_update_request: I/O error, dev nvme0n1, sector 36141696
[20377.174521] nvme nvme0: Removing after probe failure status: -19
[20377.174640] ------------[ cut here ]------------
[20377.174643] ------------[ cut here ]------------
[20377.174652] WARNING: CPU: 12 PID: 4046955 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174656] ------------[ cut here ]------------
[20377.174657] ------------[ cut here ]------------
[ 176.337027] <NVDIMM> Read ADR status from CMOS:0x00[jedec_nvdimm_init][831]
[20377.174658] Modules linked in:
[20377.174667] WARNING: CPU: 26 PID: 3949 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174675] WARNING: CPU: 24 PID: 132 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174677] unix_diag
[20377.174678] ------------[ cut here ]------------
[20377.174679] Modules linked in:
[20377.174680] ------------[ cut here ]------------
[20377.174682] ------------[ cut here ]------------
[20377.174683] Modules linked in:
[20377.174684] nf_conntrack_ipv6
[20377.174686] ------------[ cut here ]------------
[20377.174688] ------------[ cut here ]------------
[20377.174694] WARNING: CPU: 41 PID: 4038 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174701] WARNING: CPU: 42 PID: 222 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174702] unix_diag
[20377.174704] unix_diag
[20377.174710] WARNING: CPU: 44 PID: 33806 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174714] ------------[ cut here ]------------
[20377.174720] WARNING: CPU: 45 PID: 24259 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174722] nf_defrag_ipv6
[20377.174724] ------------[ cut here ]------------
[20377.174725] ------------[ cut here ]------------
[20377.174726] nf_conntrack_ipv6
[20377.174727] ------------[ cut here ]------------
[20377.174728] ------------[ cut here ]------------
[20377.174729] Modules linked in:
[20377.174731] ------------[ cut here ]------------
[20377.174733] Modules linked in:
[20377.174734] nf_conntrack_ipv6
[20377.174735] nf_conntrack_ipv4
[20377.174741] WARNING: CPU: 0 PID: 634945 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174743] Modules linked in:
[20377.174745] Modules linked in:
[20377.174750] WARNING: CPU: 1 PID: 4045642 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174753] WARNING: CPU: 2 PID: 26104 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174754] nf_defrag_ipv6
[20377.174759] WARNING: CPU: 6 PID: 4018486 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174763] WARNING: CPU: 16 PID: 381617 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174764] unix_diag
[20377.174765] unix_diag
[20377.174772] WARNING: CPU: 11 PID: 258071 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.174773] BUG: unable to handle kernel
[20377.174775] nf_defrag_ipv6
[20377.174776] nf_defrag_ipv4
[20377.174779] unix_diag
[20377.174780] Modules linked in:
[20377.174782] unix_diag
[20377.174783] nf_conntrack_ipv4
[20377.174785] Modules linked in:
[20377.174786] Modules linked in:
[20377.174788] nf_conntrack_ipv6
[20377.174790] Modules linked in:
[20377.174791] Modules linked in:
[20377.174792] nf_conntrack_ipv6
[20377.174793] paging request
[20377.174795] Modules linked in:
[20377.174796] nf_conntrack_ipv4
[20377.174798] nf_conntrack
[20377.174799] unix_diag
[20377.174801] nf_conntrack_ipv6
[20377.174802] nf_defrag_ipv4
[20377.174804] nf_conntrack_ipv6
[20377.174805] unix_diag
[20377.174807] nf_defrag_ipv6
[20377.174808] unix_diag
[20377.174809] unix_diag
[20377.174810] unix_diag
[20377.174811] at 0000000000019670
[20377.174813] nf_defrag_ipv6
[20377.174815] unix_diag
[20377.174817] nf_defrag_ipv4
[20377.174818] tcp_diag
[20377.174820] nf_conntrack_ipv6
[20377.174822] nf_defrag_ipv6
[20377.174824] nf_conntrack
[20377.174825] nf_defrag_ipv6
[20377.174827] nf_conntrack_ipv6
[20377.174828] nf_conntrack_ipv4
[20377.174829] nf_conntrack_ipv6
[20377.174832] nf_conntrack_ipv6
[20377.174832] nf_conntrack_ipv6
[20377.174833] nf_conntrack_ipv4
[20377.174835] IP:
[20377.174836] nf_conntrack_ipv6
[20377.174838] nf_conntrack
[20377.174839] inet_diag
[20377.174841] nf_defrag_ipv6
[20377.174842] nf_conntrack_ipv4
[20377.174843] tcp_diag
[20377.174846] nf_conntrack_ipv4
[20377.174847] nf_defrag_ipv6
[20377.174849] nf_defrag_ipv4
[20377.174850] nf_defrag_ipv6
[20377.174851] nf_defrag_ipv6
[20377.174852] nf_defrag_ipv6
[20377.174857] [<ffffffff811014e0>] native_queued_spin_lock_slowpath+0x110/0x200
[20377.174859] nf_defrag_ipv4
[20377.174860] nf_defrag_ipv6
[20377.174862] tcp_diag
[20377.174863] nvdimm_mem(OE)
[20377.174865] nf_conntrack_ipv4
[20377.174867] nf_defrag_ipv4
[20377.174870] inet_diag
[20377.174871] nf_conntrack
[20377.174874] nf_defrag_ipv4
[20377.174875] nf_conntrack_ipv4
[20377.174876] nf_conntrack_ipv4
[20377.174877] nf_conntrack_ipv4
[20377.174878] nf_conntrack
[20377.174880] PGD 0
[20377.174881] nf_conntrack_ipv4
[20377.174882] inet_diag
[20377.174885] nvdimm_smbus(OE)
[20377.174887] nf_defrag_ipv4
[20377.174888] nf_conntrack
[20377.174892] nvdimm_mem(OE)
[20377.174893] tcp_diag
[20377.174894] nf_conntrack
[20377.174895] nf_defrag_ipv4
[20377.174897] nf_defrag_ipv4
[20377.174897] nf_defrag_ipv4
[20377.174898]
[20377.174900] tcp_diag
[20377.174902] nf_defrag_ipv4
[20377.174903] nvdimm_mem(OE)
[20377.174905] jedec_nvdimm(OE)
[20377.174907] nf_conntrack
[20377.174909] tcp_diag
[20377.174910] nvdimm_smbus(OE)
[20377.174911] inet_diag
[20377.174913] tcp_diag
[20377.174914] nf_conntrack
[20377.174915] nf_conntrack
[20377.174916] nf_conntrack
[20377.174918] inet_diag
[20377.174920] Oops: 0002 [#1]
[20377.174922] nf_conntrack
[20377.174924] nvdimm_smbus(OE)
[20377.174926] nvdimm_core(OE)
[20377.174928] tcp_diag
[20377.174929] inet_diag
[20377.174931] jedec_nvdimm(OE)
[20377.174933] nvdimm_mem(OE)
[20377.174934] tcp_diag
[20377.174935] tcp_diag
[20377.174936] inet_diag
[20377.174937] tcp_diag
[20377.174938] nvdimm_mem(OE)
[20377.174939] SMP
[20377.174941] tcp_diag
[20377.174943] jedec_nvdimm(OE)
[20377.174945] imc_smb(OE)
[20377.174946] inet_diag
[20377.174948] nvdimm_mem(OE)
[20377.174949] nvdimm_core(OE)
[20377.174950] inet_diag
[20377.174951] nvdimm_smbus(OE)
[20377.174954] inet_diag
[20377.174954] nvdimm_mem(OE)
[20377.174955]
[20377.174956] inet_diag
[20377.174958] nvdimm_smbus(OE)
[20377.174961] inet_diag
[20377.174962] nvdimm_core(OE)
[20377.174964] ebtable_filter
[20377.174966] nvdimm_mem(OE)
[20377.174967] nvdimm_smbus(OE)
[20377.174968] imc_smb(OE)
[20377.174969] nvdimm_mem(OE)
[20377.174971] jedec_nvdimm(OE)
[20377.174972] nvdimm_mem(OE)
[20377.174973] nvdimm_smbus(OE)
[20377.174973] nvdimm_mem(OE)
[20377.174975] Modules linked in:
[20377.174976] jedec_nvdimm(OE)
[20377.174978] nvdimm_mem(OE)
[20377.174979] imc_smb(OE)
[20377.174980] ebtables
[20377.174982] nvdimm_smbus(OE)
[20377.174984] jedec_nvdimm(OE)
[20377.174984] ebtable_filter
[20377.174986] nvdimm_smbus(OE)
[20377.174987] nvdimm_smbus(OE)
[20377.174988] nvdimm_core(OE)
[20377.174989] jedec_nvdimm(OE)
[20377.174992] nvdimm_smbus(OE)
[20377.174993] unix_diag
[20377.174995] nvdimm_core(OE)
[20377.174996] nvdimm_smbus(OE)
[20377.174998] ebtable_filter
[20377.175000] ip6table_filter
[20377.175001] jedec_nvdimm(OE)
[20377.175003] nvdimm_core(OE)
[20377.175004] ebtables
[20377.175005] jedec_nvdimm(OE)
[20377.175006] jedec_nvdimm(OE)
[20377.175007] nf_conntrack_ipv6
[20377.175008] nvdimm_core(OE)
[20377.175010] imc_smb(OE)
[20377.175011] jedec_nvdimm(OE)
[20377.175012] imc_smb(OE)
[20377.175014] ebtables
[20377.175018] WARNING: CPU: 23 PID: 4058717 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.175020] ip6_tables
[20377.175022] nvdimm_core(OE)
[20377.175023] imc_smb(OE)
[20377.175024] jedec_nvdimm(OE)
[20377.175026] ip6table_filter
[20377.175027] nvdimm_core(OE)
[20377.175027] nvdimm_core(OE)
[20377.175028] nf_defrag_ipv6
[20377.175029] imc_smb(OE)
[20377.175030] ebtable_filter
[20377.175032] nvdimm_core(OE)
[20377.175034] ebtable_filter
[20377.175035] ip6table_filter
[20377.175039] iptable_filter
[20377.175041] Modules linked in:
[20377.175043] imc_smb(OE)
[20377.175043] ebtable_filter
[20377.175045] nvdimm_core(OE)
[20377.175046] ip6_tables
[20377.175047] imc_smb(OE)
[20377.175047] imc_smb(OE)
[20377.175048] nf_conntrack_ipv4
[20377.175049] ebtable_filter
[20377.175051] ebtables
[20377.175052] imc_smb(OE)
[20377.175054] ebtables
[20377.175056] ip6_tables
[20377.175057] bonding
[20377.175059] unix_diag
[20377.175060] ebtable_filter
[20377.175061] ebtables
[20377.175063] imc_smb(OE)
[20377.175064] iptable_filter
[20377.175065] ebtable_filter
[20377.175066] ebtable_filter
[20377.175067] nf_defrag_ipv4
[20377.175068] ebtables
[20377.175069] ip6table_filter
[20377.175071] ebtable_filter
[20377.175073] iptable_filter
[20377.175074] rdma_ucm(OE)
[20377.175074] nf_conntrack_ipv6
[20377.175076] ip6table_filter
[20377.175077] ebtables
[20377.175078] ip6table_filter
[20377.175080] ebtable_filter
[20377.175081] bonding
[20377.175082] ebtables
[20377.175082] ebtables
[20377.175083] nf_conntrack
[20377.175084] ip6table_filter
[20377.175086] ip6_tables
[20377.175087] ebtables
[20377.175090] bonding
[20377.175092] nf_defrag_ipv6
[20377.175093] ip6_tables
[20377.175095] ip6table_filter
[20377.175096] ip6_tables
[20377.175098] ebtables
[20377.175099] rdma_ucm(OE)
[20377.175100] ip6table_filter
[20377.175101] ip6table_filter
[20377.175104] tcp_diag
[20377.175106] iptable_filter
[20377.175108] ip6table_filter
[20377.175110] ip6_tables
[20377.175113] rdma_ucm(OE)
[20377.175115] nf_conntrack_ipv4
[20377.175117] iptable_filter
[20377.175118] ip6_tables
[20377.175120] iptable_filter
[20377.175121] ip6table_filter
[20377.175122] ib_ucm(OE)
[20377.175124] ip6_tables
[20377.175125] ip6_tables
[20377.175127] inet_diag
[20377.175131] bonding
[20377.175133] ip6_tables
[20377.175135] iptable_filter
[20377.175137] ib_ucm(OE)
[20377.175138] nf_defrag_ipv4
[20377.175140] bonding
[20377.175142] iptable_filter
[20377.175143] bonding
[20377.175145] ip6_tables
[20377.175146] rdma_cm(OE)
[20377.175148] iptable_filter
[20377.175150] iptable_filter
[20377.175151] nvdimm_mem(OE)
[20377.175155] rdma_ucm(OE)
[20377.175156] iptable_filter
[20377.175158] bonding
[20377.175160] rdma_cm(OE)
[20377.175161] nf_conntrack
[20377.175163] rdma_ucm(OE)
[20377.175165] bonding
[20377.175167] rdma_ucm(OE)
[20377.175168] iptable_filter
[20377.175169] iw_cm(OE)
[20377.175170] bonding
[20377.175172] bonding
[20377.175174] nvdimm_smbus(OE)
[20377.175177] ib_ucm(OE)
[20377.175178] bonding
[20377.175179] iw_cm(OE)
[20377.175181] tcp_diag
[20377.175183] rdma_ucm(OE)
[20377.175185] ib_ucm(OE)
[20377.175187] rdma_ucm(OE)
[20377.175188] ib_ucm(OE)
[20377.175190] bonding
[20377.175191] ib_ipoib(OE)
[20377.175192] rdma_ucm(OE)
[20377.175194] rdma_ucm(OE)
[20377.175195] jedec_nvdimm(OE)
[20377.175196] rdma_cm(OE)
[20377.175198] rdma_ucm(OE)
[20377.175200] ib_ipoib(OE)
[20377.175201] inet_diag
[20377.175202] ib_ucm(OE)
[20377.175205] rdma_cm(OE)
[20377.175207] ib_ucm(OE)
[20377.175209] rdma_cm(OE)
[20377.175210] rdma_ucm(OE)
[20377.175211] ib_cm(OE)
[20377.175213] ib_ucm(OE)
[20377.175214] ib_ucm(OE)
[20377.175215] nvdimm_core(OE)
[20377.175218] iw_cm(OE)
[20377.175220] ib_ucm(OE)
[20377.175221] ib_cm(OE)
[20377.175222] nvdimm_mem(OE)
[20377.175224] rdma_cm(OE)
[20377.175226] iw_cm(OE)
[20377.175228] rdma_cm(OE)
[20377.175230] iw_cm(OE)
[20377.175232] ib_ucm(OE)
[20377.175233] ib_uverbs(OE)
[20377.175234] rdma_cm(OE)
[20377.175236] imc_smb(OE)
[20377.175237] rdma_cm(OE)
[20377.175239] ib_ipoib(OE)
[20377.175241] rdma_cm(OE)
[20377.175242] nvdimm_smbus(OE)
[20377.175245] iw_cm(OE)
[20377.175246] ib_uverbs(OE)
[20377.175248] ib_ipoib(OE)
[20377.175249] iw_cm(OE)
[20377.175251] ib_ipoib(OE)
[20377.175253] ebtable_filter
[20377.175254] rdma_cm(OE)
[20377.175255] ib_umad(OE)
[20377.175257] iw_cm(OE)
[20377.175258] iw_cm(OE)
[20377.175260] ib_cm(OE)
[20377.175262] iw_cm(OE)
[20377.175264] jedec_nvdimm(OE)
[20377.175266] ib_ipoib(OE)
[20377.175267] ib_umad(OE)
[20377.175269] ib_cm(OE)
[20377.175271] ib_ipoib(OE)
[20377.175272] ib_cm(OE)
[20377.175273] ebtables
[20377.175275] iw_cm(OE)
[20377.175277] mlx5_fpga_tools(OE)
[20377.175279] ib_ipoib(OE)
[20377.175280] ib_ipoib(OE)
[20377.175283] ib_uverbs(OE)
[20377.175284] ib_ipoib(OE)
[20377.175286] nvdimm_core(OE)
[20377.175288] ib_cm(OE)
[20377.175289] mlx5_fpga_tools(OE)
[20377.175291] ib_uverbs(OE)
[20377.175293] ib_cm(OE)
[20377.175294] ib_uverbs(OE)
[20377.175296] ip6table_filter
[20377.175297] ib_ipoib(OE)
[20377.175298] mlx5_ib(OE)
[20377.175301] WARNING: CPU: 40 PID: 25124 at block/blk-mq.c:941 __blk_mq_run_hw_queue+0x89/0xb0
[20377.175302] ib_cm(OE)
[20377.175304] ib_cm(OE)
[20377.175305] ib_umad(OE)
[20377.175307] ib_cm(OE)
[20377.175308] imc_smb(OE)
[20377.175310] ib_uverbs(OE)
[20377.175311] mlx5_ib(OE)
[20377.175313] ib_uverbs(OE)
[20377.175314] ib_umad(OE)
[20377.175315] ib_umad(OE)
[20377.175316] ip6_tables
[20377.175317] ib_cm(OE)
[20377.175318] mlx5_core(OE)
[20377.175320] ib_uverbs(OE)
[20377.175322] Modules linked in:
[20377.175323] ib_uverbs(OE)
[20377.175325] mlx5_fpga_tools(OE)
[20377.175327] ib_uverbs(OE)
[20377.175328] ebtable_filter
[20377.175330] ib_umad(OE)
[20377.175330] mlx5_core(OE)
[20377.175332] ib_umad(OE)
[20377.175334] mlx5_fpga_tools(OE)
[20377.175335] mlx5_fpga_tools(OE)
[20377.175337] iptable_filter
[20377.175338] ib_uverbs(OE)
[20377.175339] mlxfw(OE)
[20377.175340] ib_umad(OE)
[20377.175341] unix_diag
[20377.175343] ib_umad(OE)
[20377.175345] mlx5_ib(OE)
[20377.175347] ib_umad(OE)
[20377.175349] ebtables
[20377.175350] mlxfw(OE)
[20377.175352] mlx5_fpga_tools(OE)
[20377.175353] mlx5_fpga_tools(OE)
[20377.175355] mlx5_ib(OE)
[20377.175356] mlx5_ib(OE)
[20377.175357] bonding
[20377.175357] ib_umad(OE)
[20377.175359] mlx4_en(OE)
[20377.175360] mlx5_fpga_tools(OE)
[20377.175362] nf_conntrack_ipv6
[20377.175363] mlx5_fpga_tools(OE)
[20377.175365] mlx5_core(OE)
[20377.175366] mlx5_fpga_tools(OE)
[20377.175367] ip6table_filter
[20377.175369] mlx4_en(OE)
[20377.175371] mlx5_ib(OE)
[20377.175372] mlx5_ib(OE)
[20377.175374] mlx5_core(OE)
[20377.175375] mlx5_core(OE)
[20377.175376] rdma_ucm(OE)
[20377.175377] mlx5_fpga_tools(OE)
[20377.175378] mlx4_ib(OE)
[20377.175379] mlx5_ib(OE)
[20377.175381] nf_defrag_ipv6
[20377.175382] mlx5_ib(OE)
[20377.175384] mlxfw(OE)
[20377.175385] mlx5_ib(OE)
[20377.175387] ip6_tables
[20377.175388] mlx4_ib(OE)
[20377.175390] mlx5_core(OE)
[20377.175391] mlx5_core(OE)
[20377.175393] mlxfw(OE)
[20377.175394] mlxfw(OE)
[20377.175396] ib_ucm(OE)
[20377.175396] mlx5_ib(OE)
[20377.175397] ib_core(OE)
[20377.175399] mlx5_core(OE)
[20377.175400] nf_conntrack_ipv4
[20377.175402] mlx5_core(OE)
[20377.175403] mlx4_en(OE)
[20377.175405] mlx5_core(OE)
[20377.175407] iptable_filter
[20377.175408] ib_core(OE)
[20377.175410] mlxfw(OE)
[20377.175412] mlxfw(OE)
[20377.175415] mlx4_en(OE)
[20377.175416] rdma_cm(OE)
[20377.175417] mlx5_core(OE)
[20377.175418] mlx4_core(OE)
[20377.175419] mlxfw(OE)
[20377.175420] nf_defrag_ipv4
[20377.175422] mlx4_en(OE)
[20377.175423] mlxfw(OE)
[20377.175425] mlx4_ib(OE)
[20377.175426] mlxfw(OE)
[20377.175428] bonding
[20377.175429] mlx4_core(OE)
[20377.175430] mlx4_en(OE)
[20377.175431] mlx4_ib(OE)
[20377.175433] mlx4_en(OE)
[20377.175434] iw_cm(OE)
[20377.175435] mlxfw(OE)
[20377.175437] mlx_compat(OE)
[20377.175437] mlx4_en(OE)
[20377.175439] nf_conntrack
[20377.175440] mlx4_ib(OE)
[20377.175441] mlx4_en(OE)
[20377.175442] mlx4_en(OE)
[20377.175444] rdma_ucm(OE)
[20377.175445] mlx_compat(OE)
[20377.175447] mlx4_ib(OE)
[20377.175448] ib_core(OE)
[20377.175450] mlx4_ib(OE)
[20377.175451] ib_ipoib(OE)
[20377.175452] mlx4_en(OE)
[20377.175453] devlink
[20377.175455] mlx4_ib(OE)
[20377.175456] tcp_diag
[20377.175457] mlx4_ib(OE)
[20377.175458] mlx4_ib(OE)
[20377.175460] ib_ucm(OE)
[20377.175462] devlink
[20377.175463] ib_core(OE)
[20377.175465] ib_core(OE)
[20377.175466] ib_core(OE)
[20377.175467] mlx4_core(OE)
[20377.175469] ib_core(OE)
[20377.175470] ib_cm(OE)
[20377.175471] mlx4_ib(OE)
[20377.175472] xfs
[20377.175474] ib_core(OE)
[20377.175475] inet_diag
[20377.175476] ib_core(OE)
[20377.175478] ib_core(OE)
[20377.175480] rdma_cm(OE)
[20377.175481] xfs
[20377.175482] mlx4_core(OE)
[20377.175484] mlx4_core(OE)
[20377.175485] mlx4_core(OE)
[20377.175487] mlx_compat(OE)
[20377.175488] mlx4_core(OE)
[20377.175489] ib_uverbs(OE)
[20377.175490] ib_core(OE)
[20377.175491] libcrc32c
[20377.175492] mlx4_core(OE)
[20377.175494] nvdimm_mem(OE)
[20377.175496] mlx4_core(OE)
[20377.175497] mlx4_core(OE)
[20377.175499] iw_cm(OE)
[20377.175501] libcrc32c
[20377.175502] mlx_compat(OE)
[20377.175504] mlx_compat(OE)
[20377.175505] devlink
[20377.175507] ib_umad(OE)
[20377.175508] mlx4_core(OE)
[20377.175509] ipmi_watchdog
[20377.175510] mlx_compat(OE)
[20377.175511] mlx_compat(OE)
[20377.175513] mlx_compat(OE)
[20377.175515] mlx_compat(OE)
[20377.175516] mlx_compat(OE)
[20377.175519] ib_ipoib(OE)
[20377.175520] ipmi_watchdog
[20377.175522] devlink
[20377.175523] devlink
[20377.175525] xfs
[20377.175526] mlx5_fpga_tools(OE)
[20377.175527] mlx_compat(OE)
[20377.175529] dm_mirror
[20377.175530] devlink
[20377.175531] nvdimm_smbus(OE)
[20377.175533] devlink
[20377.175535] devlink
[20377.175536] devlink
[20377.175538] devlink
[20377.175540] ib_cm(OE)
[20377.175542] dm_mirror
[20377.175543] xfs
[20377.175544] xfs
[20377.175546] libcrc32c
[20377.175548] mlx5_ib(OE)
[20377.175549] devlink
[20377.175550] dm_region_hash
[20377.175551] jedec_nvdimm(OE)
[20377.175553] xfs
[20377.175554] xfs
[20377.175557] xfs
[20377.175558] xfs
[20377.175560] ib_uverbs(OE)
[20377.175561] dm_region_hash
[20377.175564] libcrc32c
[20377.175565] libcrc32c
[20377.175567] ipmi_watchdog
[20377.175568] mlx5_core(OE)
[20377.175570] xfs
[20377.175571] xfs
[20377.175572] dm_log
[20377.175573] nvdimm_core(OE)
[20377.175574] libcrc32c
[20377.175575] libcrc32c
[20377.175577] libcrc32c
[20377.175579] libcrc32c
[20377.175581] ib_umad(OE)
[20377.175581] dm_log
[20377.175583] ipmi_watchdog
[20377.175584] dm_mirror
[20377.175585] mlxfw(OE)
[20377.175586] libcrc32c
[20377.175587] dm_mod
[20377.175588] imc_smb(OE)
[20377.175589] ipmi_watchdog
[20377.175591] libcrc32c
[20377.175592] ipmi_watchdog
[20377.175595] ipmi_watchdog
[20377.175596] ipmi_watchdog
[20377.175597] mlx5_fpga_tools(OE)
[20377.175599] ipmi_watchdog
[20377.175600] dm_mod
[20377.175601] dm_mirror
[20377.175602] dm_region_hash
[20377.175604] mlx4_en(OE)
[20377.175605] ipmi_watchdog
[20377.175607] vfat
[20377.175607] ebtable_filter
[20377.175609] dm_mirror
[20377.175610] dm_mirror
[20377.175612] dm_mirror
[20377.175613] dm_mirror
[20377.175616] mlx5_ib(OE)
[20377.175618] dm_mirror
[20377.175619] vfat
[20377.175620] dm_region_hash
[20377.175621] dm_log
[20377.175623] mlx4_ib(OE)
[20377.175623] dm_mirror
[20377.175624] fat
[20377.175626] ipmi_watchdog
[20377.175628] dm_region_hash
[20377.175629] dm_region_hash
[20377.175631] ebtables
[20377.175632] dm_region_hash
[20377.175634] mlx5_core(OE)
[20377.175636] dm_region_hash
[20377.175637] fat
[20377.175638] dm_log
[20377.175639] ib_core(OE)
[20377.175640] dm_region_hash
[20377.175641] skx_edac
[20377.175643] dm_mirror
[20377.175644] dm_log
[20377.175645] dm_log
[20377.175647] ip6table_filter
[20377.175649] dm_region_hash
[20377.175650] dm_log
[20377.175652] mlxfw(OE)
[20377.175653] dm_log
[20377.175655] dm_mod
[20377.175656] skx_edac
[20377.175657] dm_mod
[20377.175658] mlx4_core(OE)
[20377.175659] dm_log
[20377.175660] edac_core
[20377.175662] dm_mod
[20377.175664] dm_region_hash
[20377.175665] dm_mod
[20377.175666] ip6_tables
[20377.175668] dm_log
[20377.175669] dm_mod
[20377.175671] mlx4_en(OE)
[20377.175672] dm_mod
[20377.175675] vfat
[20377.175676] edac_core
[20377.175677] vfat
[20377.175678] mlx_compat(OE)
[20377.175679] dm_mod
[20377.175680] coretemp
[20377.175681] vfat
[20377.175683] dm_log
[20377.175684] vfat
[20377.175685] iptable_filter
[20377.175687] dm_mod
[20377.175688] vfat
[20377.175689] mlx4_ib(OE)
[20377.175691] vfat
[20377.175692] fat
[20377.175693] coretemp
[20377.175694] fat
[20377.175696] devlink
[20377.175697] vfat
[20377.175698] intel_rapl
[20377.175699] fat
[20377.175701] dm_mod
[20377.175702] fat
[20377.175705] vfat
[20377.175706] fat
[20377.175708] ib_core(OE)
[20377.175709] fat
[20377.175711] bonding
[20377.175713] skx_edac
[20377.175713] intel_rapl
[20377.175715] skx_edac
[20377.175716] xfs
[20377.175717] fat
[20377.175718] iosf_mbi
[20377.175719] skx_edac
[20377.175722] vfat
[20377.175723] skx_edac
[20377.175725] fat
[20377.175726] skx_edac
[20377.175728] mlx4_core(OE)
[20377.175729] skx_edac
[20377.175730] rdma_ucm(OE)
[20377.175732] edac_core
[20377.175733] iosf_mbi
[20377.175734] edac_core
[20377.175736] libcrc32c
[20377.175737] skx_edac
[20377.175738] kvm_intel
[20377.175740] edac_core
[20377.175740] edac_core
[20377.175742] edac_core
[20377.175743] mlx_compat(OE)
[20377.175744] skx_edac
[20377.175746] edac_core
[20377.175748] ib_ucm(OE)
[20377.175751] coretemp
[20377.175752] kvm_intel
[20377.175754] fat
[20377.175756] coretemp
[20377.175757] ipmi_watchdog
[20377.175757] edac_core
[20377.175759] kvm
[20377.175760] coretemp
[20377.175761] coretemp
[20377.175762] coretemp
[20377.175764] devlink
[20377.175765] edac_core
[20377.175766] coretemp
[20377.175767] rdma_cm(OE)
[20377.175769] intel_rapl
[20377.175771] kvm
[20377.175773] skx_edac
[20377.175774] intel_rapl
[20377.175776] dm_mirror
[20377.175776] coretemp
[20377.175778] irqbypass
[20377.175779] intel_rapl
[20377.175780] intel_rapl
[20377.175781] intel_rapl
[20377.175782] xfs
[20377.175784] coretemp
[20377.175785] iw_cm(OE)
[20377.175787] iosf_mbi
[20377.175788] irqbypass
[20377.175790] intel_rapl
[20377.175791] iosf_mbi
[20377.175792] dm_region_hash
[20377.175794] edac_core
[20377.175795] intel_rapl
[20377.175796] crc32_pclmul
[20377.175797] iosf_mbi
[20377.175798] iosf_mbi
[20377.175800] iosf_mbi
[20377.175801] intel_rapl
[20377.175802] ib_ipoib(OE)
[20377.175804] libcrc32c
[20377.175806] kvm_intel
[20377.175807] crc32_pclmul
[20377.175809] iosf_mbi
[20377.175810] kvm_intel
[20377.175811] dm_log
[20377.175812] iosf_mbi
[20377.175813] ghash_clmulni_intel
[20377.175815] kvm_intel
[20377.175816] kvm_intel
[20377.175818] coretemp
[20377.175819] kvm_intel
[20377.175821] iosf_mbi
[20377.175822] ib_cm(OE)
[20377.175824] ipmi_watchdog
[20377.175826] ghash_clmulni_intel
[20377.175827] kvm_intel
[20377.175828] kvm
[20377.175830] dm_mod
[20377.175831] kvm_intel
[20377.175832] aesni_intel
[20377.175834] kvm
[20377.175834] kvm
[20377.175837] kvm
[20377.175840] kvm
[20377.175841] intel_rapl
[20377.175842] kvm_intel
[20377.175843] ib_uverbs(OE)
[20377.175844] dm_mirror
[20377.175845] aesni_intel
[20377.175846] irqbypass
[20377.175847] vfat
[20377.175848] kvm
[20377.175850] lrw
[20377.175851] irqbypass
[20377.175852] irqbypass
[20377.175854] irqbypass
[20377.175855] irqbypass
[20377.175856] nf_conntrack_ipv4
[20377.175857] iosf_mbi
[20377.175858] kvm
[20377.175858] ib_umad(OE)
[20377.175859] dm_region_hash
[20377.175861] kvm
[20377.175862] lrw
[20377.175863] crc32_pclmul
[20377.175864] fat
[20377.175865] irqbypass
[20377.175866] iTCO_wdt
[20377.175868] crc32_pclmul
[20377.175869] crc32_pclmul
[20377.175870] crc32_pclmul
[20377.175872] crc32_pclmul
[20377.175873] nf_defrag_ipv4
[20377.175874] kvm_intel
[20377.175875] irqbypass
[20377.175876] dm_log
[20377.175877] irqbypass
[20377.175878] iTCO_wdt
[20377.175879] ghash_clmulni_intel
[20377.175880] skx_edac
[20377.175881] crc32_pclmul
[20377.175882] iTCO_vendor_support
[20377.175884] ghash_clmulni_intel
[20377.175886] ghash_clmulni_intel
[20377.175887] ghash_clmulni_intel
[20377.175889] mlx5_fpga_tools(OE)
[20377.175890] ghash_clmulni_intel
[20377.175891] nf_conntrack
[20377.175892] kvm
[20377.175893] crc32_pclmul
[20377.175894] dm_mod
[20377.175896] iTCO_vendor_support
[20377.175897] aesni_intel
[20377.175898] edac_core
[20377.175899] ghash_clmulni_intel
[20377.175900] gf128mul
[20377.175901] aesni_intel
[20377.175903] crc32_pclmul
[20377.175904] aesni_intel
[20377.175905] aesni_intel
[20377.175907] mlx5_ib(OE)
[20377.175908] aesni_intel
[20377.175909] tcp_diag
[20377.175910] ghash_clmulni_intel
[20377.175911] vfat
[20377.175912] irqbypass
[20377.175913] gf128mul
[20377.175914] lrw
[20377.175915] coretemp
[20377.175916] aesni_intel
[20377.175917] glue_helper
[20377.175918] lrw
[20377.175919] ghash_clmulni_intel
[20377.175920] lrw
[20377.175921] lrw
[20377.175922] mlx5_core(OE)
[20377.175923] lrw
[20377.175925] inet_diag
[20377.175926] aesni_intel
[20377.175926] fat
[20377.175928] crc32_pclmul
[20377.175929] glue_helper
[20377.175931] iTCO_wdt
[20377.175932] intel_rapl
[20377.175932] lrw
[20377.175934] ablk_helper
[20377.175935] iTCO_wdt
[20377.175936] aesni_intel
[20377.175937] iTCO_wdt
[20377.175938] iTCO_wdt
[20377.175939] mlxfw(OE)
[20377.175940] iTCO_wdt
[20377.175942] nvdimm_mem(OE)
[20377.175943] lrw
[20377.175944] skx_edac
[20377.175947] ghash_clmulni_intel
[20377.175948] ablk_helper
[20377.175949] iTCO_vendor_support
[20377.175950] iosf_mbi
[20377.175951] iTCO_wdt
[20377.175952] cryptd
[20377.175953] iTCO_vendor_support
[20377.175954] lrw
[20377.175956] iTCO_vendor_support
[20377.175957] iTCO_vendor_support
[20377.175958] mlx4_en(OE)
[20377.175959] iTCO_vendor_support
[20377.175961] nvdimm_smbus(OE)
[20377.175962] iTCO_wdt
[20377.175963] edac_core
[20377.175964] aesni_intel
[20377.175965] cryptd
[20377.175966] gf128mul
[20377.175967] kvm_intel
[20377.175968] iTCO_vendor_support
[20377.175969] pcspkr
[20377.175970] gf128mul
[20377.175972] gf128mul
[20377.175973] gf128mul
[20377.175974] mlx4_ib(OE)
[20377.175976] gf128mul
[20377.175978] iTCO_wdt
[20377.175979] jedec_nvdimm(OE)
[20377.175980] iTCO_vendor_support
[20377.175981] coretemp
[20377.175982] lrw
[20377.175983] pcspkr
[20377.175985] glue_helper
[20377.175986] kvm
[20377.175986] gf128mul
[20377.175988] joydev
[20377.175989] glue_helper
[20377.175990] glue_helper
[20377.175991] glue_helper
[20377.175992] ib_core(OE)
[20377.175994] glue_helper
[20377.175995] iTCO_vendor_support
[20377.175996] nvdimm_core(OE)
[20377.175998] gf128mul
[20377.175999] intel_rapl
[20377.176000] joydev
[20377.176002] ablk_helper
[20377.176003] irqbypass
[20377.176003] glue_helper
[20377.176005] ses
[20377.176006] ablk_helper
[20377.176007] ablk_helper
[20377.176008] ablk_helper
[20377.176009] mlx4_core(OE)
[20377.176010] ablk_helper
[20377.176012] gf128mul
[20377.176013] imc_smb(OE)
[20377.176014] glue_helper
[20377.176016] iosf_mbi
[20377.176016] ses
[20377.176018] cryptd
[20377.176019] crc32_pclmul
[20377.176020] iTCO_wdt
[20377.176021] ablk_helper
[20377.176022] enclosure
[20377.176024] cryptd
[20377.176025] cryptd
[20377.176026] cryptd
[20377.176027] mlx_compat(OE)
[20377.176028] cryptd
[20377.176030] ebtable_filter
[20377.176031] glue_helper
[20377.176032] ablk_helper
[20377.176033] kvm_intel
[20377.176034] enclosure
[20377.176035] pcspkr
[20377.176036] ghash_clmulni_intel
[20377.176038] iTCO_vendor_support
[20377.176039] cryptd
[20377.176039] sg
[20377.176041] pcspkr
[20377.176042] pcspkr
[20377.176043] pcspkr
[20377.176044] devlink
[20377.176045] pcspkr
[20377.176046] ebtables
[20377.176047] ablk_helper
[20377.176048] cryptd
[20377.176050] kvm
[20377.176050] sg
[20377.176051] joydev
[20377.176052] aesni_intel
[20377.176053] gf128mul
[20377.176054] pcspkr
[20377.176055] mei_me
[20377.176057] joydev
[20377.176058] joydev
[20377.176060] joydev
[20377.176061] joydev
[20377.176062] ip6table_filter
[20377.176063] cryptd
[20377.176064] pcspkr
[20377.176064] xfs
[20377.176066] irqbypass
[20377.176066] mei_me
[20377.176067] ses
[20377.176068] lrw
[20377.176070] glue_helper
[20377.176071] joydev
[20377.176072] mei
[20377.176073] ses
[20377.176074] ses
[20377.176076] ses
[20377.176077] ses
[20377.176079] ip6_tables
[20377.176079] pcspkr
[20377.176080] joydev
[20377.176081] libcrc32c
[20377.176083] crc32_pclmul
[20377.176083] mei
[20377.176085] enclosure
[20377.176085] iTCO_wdt
[20377.176086] ablk_helper
[20377.176087] ses
[20377.176088] shpchp
[20377.176090] enclosure
[20377.176091] enclosure
[20377.176092] enclosure
[20377.176093] enclosure
[20377.176094] iptable_filter
[20377.176095] ses
[20377.176096] ipmi_watchdog
[20377.176097] joydev
[20377.176098] ghash_clmulni_intel
[20377.176099] shpchp
[20377.176100] sg
[20377.176101] iTCO_vendor_support
[20377.176104] enclosure
[20377.176105] lpc_ich
[20377.176109] sg
[20377.176110] sg
[20377.176112] sg
[20377.176113] sg
[20377.176114] bonding
[20377.176115] enclosure
[20377.176115] dm_mirror
[20377.176116] ses
[20377.176118] aesni_intel
[20377.176118] lpc_ich
[20377.176120] mei_me
[20377.176121] gf128mul
[20377.176122] sg
[20377.176123] cryptd
[20377.176124] i2c_i801
[20377.176126] mei_me
[20377.176127] mei_me
[20377.176128] mei_me
[20377.176129] mei_me
[20377.176131] rdma_ucm(OE)
[20377.176132] sg
[20377.176132] dm_region_hash
[20377.176133] enclosure
[20377.176135] lrw
[20377.176137] i2c_i801
[20377.176138] mei
[20377.176139] glue_helper
[20377.176141] mei_me
[20377.176142] pcspkr
[20377.176143] wmi
[20377.176144] mei
[20377.176145] mei
[20377.176147] mei
[20377.176147] mei
[20377.176149] ib_ucm(OE)
[20377.176149] mei_me
[20377.176150] dm_log
[20377.176151] sg
[20377.176152] iTCO_wdt
[20377.176153] wmi
[20377.176154] shpchp
[20377.176155] ablk_helper
[20377.176156] mei
[20377.176158] joydev
[20377.176159] ipmi_si
[20377.176161] shpchp
[20377.176162] shpchp
[20377.176163] shpchp
[20377.176164] shpchp
[20377.176165] rdma_cm(OE)
[20377.176166] mei
[20377.176167] dm_mod
[20377.176168] mei_me
[20377.176169] iTCO_vendor_support
[20377.176170] ipmi_si
[20377.176172] lpc_ich
[20377.176173] cryptd
[20377.176174] shpchp
[20377.176176] ses
[20377.176177] lpc_ich
[20377.176178] lpc_ich
[20377.176180] lpc_ich
[20377.176181] lpc_ich
[20377.176183] iw_cm(OE)
[20377.176184] shpchp
[20377.176185] vfat
[20377.176185] mei
[20377.176187] gf128mul
[20377.176189] ipmi_devintf
[20377.176190] i2c_i801
[20377.176191] pcspkr
[20377.176192] lpc_ich
[20377.176195] enclosure
[20377.176196] i2c_i801
[20377.176198] i2c_i801
[20377.176199] i2c_i801
[20377.176201] i2c_i801
[20377.176202] ib_ipoib(OE)
[20377.176203] lpc_ich
[20377.176204] fat
[20377.176205] glue_helper
[20377.176208] shpchp
[20377.176209] ipmi_msghandler
[20377.176210] wmi
[20377.176212] joydev
[20377.176213] i2c_i801
[20377.176214] sg
[20377.176216] wmi
[20377.176217] wmi
[20377.176219] wmi
[20377.176220] wmi
[20377.176222] ib_cm(OE)
[20377.176223] i2c_i801
[20377.176223] skx_edac
[20377.176225] ablk_helper
[20377.176227] lpc_ich
[20377.176229] nfit
[20377.176230] ipmi_si
[20377.176231] ses
[20377.176233] wmi
[20377.176235] ipmi_si
[20377.176236] ipmi_si
[20377.176238] ipmi_si
[20377.176239] ipmi_si
[20377.176241] edac_core
[20377.176242] cryptd
[20377.176243] wmi
[20377.176244] i2c_i801
[20377.176244] mei_me
[20377.176246] enclosure
[20377.176247] ib_uverbs(OE)
[20377.176249] ipmi_si
[20377.176250] libnvdimm
[20377.176252] ipmi_devintf
[20377.176253] ipmi_devintf
[20377.176254] ipmi_devintf
[20377.176256] ipmi_devintf
[20377.176257] ipmi_devintf
[20377.176258] coretemp
[20377.176259] pcspkr
[20377.176259] ipmi_si
[20377.176260] wmi
[20377.176261] mei
[20377.176262] sg
[20377.176263] ib_umad(OE)
[20377.176266] ipmi_devintf
[20377.176267] acpi_power_meter
[20377.176268] ipmi_msghandler
[20377.176270] ipmi_msghandler
[20377.176271] ipmi_msghandler
[20377.176272] ipmi_msghandler
[20377.176274] ipmi_msghandler
[20377.176275] intel_rapl
[20377.176276] joydev
[20377.176277] ipmi_devintf
[20377.176278] ipmi_si
[20377.176279] shpchp
[20377.176281] mei_me
[20377.176282] mlx5_fpga_tools(OE)
[20377.176284] ipmi_msghandler
[20377.176285] nfsd
[20377.176287] nfit
[20377.176289] nfit
[20377.176290] nfit
[20377.176292] nfit
[20377.176293] nfit
[20377.176294] iosf_mbi
[20377.176295] ses
[20377.176296] ipmi_msghandler
[20377.176298] lpc_ich
[20377.176299] mei
[20377.176300] mlx5_ib(OE)
[20377.176302] nfit
[20377.176303] auth_rpcgss
[20377.176305] libnvdimm
[20377.176306] ipmi_devintf
[20377.176307] libnvdimm
[20377.176308] libnvdimm
[20377.176310] libnvdimm
[20377.176311] libnvdimm
[20377.176314] kvm_intel
[20377.176315] enclosure
[20377.176315] nfit
[20377.176317] i2c_i801
[20377.176318] shpchp
[20377.176320] mlx5_core(OE)
[20377.176321] libnvdimm
[20377.176323] nfs_acl
[20377.176324] acpi_power_meter
[20377.176325] ipmi_msghandler
[20377.176327] acpi_power_meter
[20377.176329] acpi_power_meter
[20377.176331] acpi_power_meter
[20377.176332] acpi_power_meter
[20377.176333] kvm
[20377.176334] sg
[20377.176334] libnvdimm
[20377.176335] wmi
[20377.176337] lpc_ich
[20377.176338] mlxfw(OE)
[20377.176339] acpi_power_meter
[20377.176340] lockd
[20377.176342] nfsd
[20377.176343] nfsd
[20377.176345] nfit
[20377.176346] nfsd
[20377.176347] nfsd
[20377.176348] nfsd
[20377.176350] irqbypass
[20377.176351] mei_me
[20377.176351] acpi_power_meter
[20377.176352] ipmi_si
[20377.176354] i2c_i801
[20377.176355] mlx4_en(OE)
[20377.176356] nfsd
[20377.176357] grace
[20377.176359] auth_rpcgss
[20377.176360] auth_rpcgss
[20377.176362] libnvdimm
[20377.176363] auth_rpcgss
[20377.176365] auth_rpcgss
[20377.176367] auth_rpcgss
[20377.176368] crc32_pclmul
[20377.176369] mei
[20377.176370] nfsd
[20377.176372] ipmi_devintf
[20377.176373] wmi
[20377.176374] mlx4_ib(OE)
[20377.176375] auth_rpcgss
[20377.176377] sunrpc
[20377.176378] nfs_acl
[20377.176379] nfs_acl
[20377.176381] acpi_power_meter
[20377.176382] nfs_acl
[20377.176384] nfs_acl
[20377.176385] nfs_acl
[20377.176387] ghash_clmulni_intel
[20377.176388] shpchp
[20377.176388] auth_rpcgss
[20377.176390] ipmi_msghandler
[20377.176392] ipmi_si
[20377.176393] ib_core(OE)
[20377.176395] nfs_acl
[20377.176396] knem(OE)
[20377.176397] lockd
[20377.176398] lockd
[20377.176400] nfsd
[20377.176401] lockd
[20377.176402] lockd
[20377.176404] lockd
[20377.176405] aesni_intel
[20377.176406] lpc_ich
[20377.176406] nfs_acl
[20377.176407] nfit
[20377.176410] ipmi_devintf
[20377.176411] mlx4_core(OE)
[20377.176412] lockd
[20377.176414] ip_tables
[20377.176415] grace
[20377.176416] grace
[20377.176417] auth_rpcgss
[20377.176418] grace
[20377.176420] grace
[20377.176422] grace
[20377.176423] lrw
[20377.176424] i2c_i801
[20377.176424] lockd
[20377.176425] libnvdimm
[20377.176427] ipmi_msghandler
[20377.176428] mlx_compat(OE)
[20377.176429] grace
[20377.176431] ext4
[20377.176432] sunrpc
[20377.176433] sunrpc
[20377.176434] nfs_acl
[20377.176436] sunrpc
[20377.176437] sunrpc
[20377.176439] sunrpc
[20377.176440] iTCO_wdt
[20377.176441] wmi
[20377.176442] grace
[20377.176443] acpi_power_meter
[20377.176444] nfit
[20377.176445] devlink
[20377.176447] sunrpc
[20377.176448] mbcache
[20377.176449] knem(OE)
[20377.176450] knem(OE)
[20377.176452] lockd
[20377.176454] knem(OE)
[20377.176455] knem(OE)
[20377.176456] knem(OE)
[20377.176458] iTCO_vendor_support
[20377.176458] ipmi_si
[20377.176459] sunrpc
[20377.176460] nfsd
[20377.176462] libnvdimm
[20377.176462] xfs
[20377.176464] knem(OE)
[20377.176465] jbd2
[20377.176466] ip_tables
[20377.176467] ip_tables
[20377.176469] grace
[20377.176470] ip_tables
[20377.176471] ip_tables
[20377.176472] ip_tables
[20377.176474] gf128mul
[20377.176474] ipmi_devintf
[20377.176475] knem(OE)
[20377.176476] auth_rpcgss
[20377.176478] acpi_power_meter
[20377.176479] libcrc32c
[20377.176480] ip_tables
[20377.176482] sd_mod
[20377.176484] ext4
[20377.176485] ext4
[20377.176486] ext4
[20377.176487] sunrpc
[20377.176489] ext4
[20377.176490] ext4
[20377.176491] glue_helper
[20377.176492] ipmi_msghandler
[20377.176493] ip_tables
[20377.176494] nfs_acl
[20377.176495] nfsd
[20377.176496] ipmi_watchdog
[20377.176497] ext4
[20377.176499] crc_t10dif
[20377.176500] mbcache
[20377.176503] mbcache
[20377.176505] mbcache
[20377.176507] mbcache
[20377.176508] mbcache
[20377.176509] ablk_helper
[20377.176510] nfit
[20377.176510] ext4
[20377.176511] knem(OE)
[20377.176512] auth_rpcgss
[20377.176514] dm_mirror
[20377.176515] mbcache
[20377.176516] crct10dif_generic
[20377.176518] lockd
[20377.176520] jbd2
[20377.176521] jbd2
[20377.176522] jbd2
[20377.176524] jbd2
[20377.176526] jbd2
[20377.176527] cryptd
[20377.176528] libnvdimm
[20377.176529] mbcache
[20377.176530] nfs_acl
[20377.176531] ip_tables
[20377.176532] dm_region_hash
[20377.176534] jbd2
[20377.176536] ast
[20377.176538] grace
[20377.176539] sd_mod
[20377.176540] sd_mod
[20377.176541] sd_mod
[20377.176543] sd_mod
[20377.176544] sd_mod
[20377.176545] pcspkr
[20377.176546] acpi_power_meter
[20377.176547] jbd2
[20377.176547] lockd
[20377.176548] ext4
[20377.176550] dm_log
[20377.176552] sd_mod
[20377.176554] i2c_algo_bit
[20377.176555] sunrpc
[20377.176557] crc_t10dif
[20377.176558] crc_t10dif
[20377.176559] crc_t10dif
[20377.176561] crc_t10dif
[20377.176562] crc_t10dif
[20377.176564] joydev
[20377.176565] nfsd
[20377.176565] sd_mod
[20377.176567] grace
[20377.176568] dm_mod
[20377.176570] mbcache
[20377.176571] crc_t10dif
[20377.176572] drm_kms_helper
[20377.176574] knem(OE)
[20377.176576] crct10dif_generic
[20377.176578] crct10dif_generic
[20377.176579] crct10dif_generic
[20377.176580] crct10dif_generic
[20377.176581] crct10dif_generic
[20377.176582] ses
[20377.176583] auth_rpcgss
[20377.176584] crc_t10dif
[20377.176585] sunrpc
[20377.176586] vfat
[20377.176587] jbd2
[20377.176588] crct10dif_generic
[20377.176589] syscopyarea
[20377.176592] ast
[20377.176593] ast
[20377.176594] ip_tables
[20377.176596] ib_ucm(OE)
[20377.176597] ast
[20377.176599] ast
[20377.176600] ast
[20377.176601] enclosure
[20377.176602] nfs_acl
[20377.176603] crct10dif_generic
[20377.176604] knem(OE)
[20377.176605] fat
[20377.176606] ast
[20377.176608] sd_mod
[20377.176609] sysfillrect
[20377.176610] i2c_algo_bit
[20377.176611] i2c_algo_bit
[20377.176612] ext4
[20377.176613] rdma_cm(OE)
[20377.176615] i2c_algo_bit
[20377.176616] i2c_algo_bit
[20377.176617] i2c_algo_bit
[20377.176618] sg
[20377.176619] lockd
[20377.176620] ast
[20377.176621] ip_tables
[20377.176622] skx_edac
[20377.176623] i2c_algo_bit
[20377.176624] crc_t10dif
[20377.176625] sysimgblt
[20377.176626] drm_kms_helper
[20377.176627] drm_kms_helper
[20377.176628] mbcache
[20377.176629] iw_cm(OE)
[20377.176631] drm_kms_helper
[20377.176632] drm_kms_helper
[20377.176634] drm_kms_helper
[20377.176634] mei_me
[20377.176635] grace
[20377.176636] i2c_algo_bit
[20377.176638] ext4
[20377.176638] edac_core
[20377.176640] drm_kms_helper
[20377.176641] crct10dif_generic
[20377.176642] ixgbe
[20377.176643] syscopyarea
[20377.176644] syscopyarea
[20377.176645] jbd2
[20377.176647] ib_ipoib(OE)
[20377.176648] syscopyarea
[20377.176649] syscopyarea
[20377.176650] syscopyarea
[20377.176652] mei
[20377.176652] sunrpc
[20377.176654] drm_kms_helper
[20377.176655] mbcache
[20377.176656] coretemp
[20377.176657] syscopyarea
[20377.176659] ast
[20377.176659] fb_sys_fops
[20377.176661] sysfillrect
[20377.176661] sysfillrect
[20377.176662] sd_mod
[20377.176663] ib_cm(OE)
[20377.176664] sysfillrect
[20377.176666] sysfillrect
[20377.176667] sysfillrect
[20377.176668] shpchp
[20377.176668] knem(OE)
[20377.176669] syscopyarea
[20377.176671] jbd2
[20377.176672] intel_rapl
[20377.176673] sysfillrect
[20377.176674] i2c_algo_bit
[20377.176675] ttm
[20377.176676] sysimgblt
[20377.176677] sysimgblt
[20377.176678] crc_t10dif
[20377.176679] ib_uverbs(OE)
[20377.176680] sysimgblt
[20377.176684] sysimgblt
[20377.176685] sysimgblt
[20377.176685] lpc_ich
[20377.176686] ip_tables
[20377.176687] sysfillrect
[20377.176688] sd_mod
[20377.176689] iosf_mbi
[20377.176690] sysimgblt
[20377.176691] drm_kms_helper
[20377.176692] crct10dif_pclmul
[20377.176693] ixgbe
[20377.176694] ixgbe
[20377.176696] ib_umad(OE)
[20377.176697] ixgbe
[20377.176698] crct10dif_generic
[20377.176700] ixgbe
[20377.176701] ixgbe
[20377.176702] i2c_i801
[20377.176703] ext4
[20377.176704] sysimgblt
[20377.176705] crc_t10dif
[20377.176706] kvm_intel
[20377.176707] ixgbe
[20377.176709] syscopyarea
[20377.176709] crct10dif_common
[20377.176710] fb_sys_fops
[20377.176711] fb_sys_fops
[20377.176712] mlx5_fpga_tools(OE)
[20377.176714] fb_sys_fops
[20377.176715] ast
[20377.176716] fb_sys_fops
[20377.176718] fb_sys_fops
[20377.176719] wmi
[20377.176719] mbcache
[20377.176721] ixgbe
[20377.176722] crct10dif_generic
[20377.176723] kvm
[20377.176724] fb_sys_fops
[20377.176726] sysfillrect
[20377.176727] crc32c_intel
[20377.176728] ttm
[20377.176729] ttm
[20377.176730] mlx5_ib(OE)
[20377.176731] ttm
[20377.176732] i2c_algo_bit
[20377.176735] ttm
[20377.176736] ttm
[20377.176737] ipmi_si
[20377.176738] jbd2
[20377.176739] fb_sys_fops
[20377.176740] ast
[20377.176741] irqbypass
[20377.176742] ttm
[20377.176743] sysimgblt
[20377.176744] drm
[20377.176746] crct10dif_pclmul
[20377.176746] crct10dif_pclmul
[20377.176748] mlx5_core(OE)
[20377.176749] crct10dif_pclmul
[20377.176750] drm_kms_helper
[20377.176752] crct10dif_pclmul
[20377.176754] crct10dif_pclmul
[20377.176754] ipmi_devintf
[20377.176755] sd_mod
[20377.176756] ttm
[20377.176758] i2c_algo_bit
[20377.176759] crc32_pclmul
[20377.176760] crct10dif_pclmul
[20377.176761] ixgbe
[20377.176762] i40e
[20377.176763] crct10dif_common
[20377.176764] crct10dif_common
[20377.176766] mlxfw(OE)
[20377.176767] crct10dif_common
[20377.176768] syscopyarea
[20377.176769] crct10dif_common
[20377.176770] crct10dif_common
[20377.176771] ipmi_msghandler
[20377.176772] crc_t10dif
[20377.176773] crct10dif_pclmul
[20377.176774] drm_kms_helper
[20377.176775] ghash_clmulni_intel
[20377.176777] crct10dif_common
[20377.176779] fb_sys_fops
[20377.176780] nvme
[20377.176781] crc32c_intel
[20377.176781] crc32c_intel
[20377.176783] mlx4_en(OE)
[20377.176784] crc32c_intel
[20377.176786] crc32c_intel
[20377.176787] crc32c_intel
[20377.176787] nfit
[20377.176788] crct10dif_generic
[20377.176789] crct10dif_common
[20377.176790] syscopyarea
[20377.176791] aesni_intel
[20377.176792] crc32c_intel
[20377.176793] sysfillrect
[20377.176795] ttm
[20377.176795] ahci
[20377.176796] drm
[20377.176797] drm
[20377.176799] mlx4_ib(OE)
[20377.176800] drm
[20377.176801] drm
[20377.176802] drm
[20377.176803] libnvdimm
[20377.176804] ast
[20377.176804] crc32c_intel
[20377.176806] sysfillrect
[20377.176806] lrw
[20377.176808] drm
[20377.176809] sysimgblt
[20377.176811] crct10dif_pclmul
[20377.176812] libahci
[20377.176813] i40e
[20377.176814] i40e
[20377.176815] ib_core(OE)
[20377.176816] i40e
[20377.176817] i40e
[20377.176818] i40e
[20377.176819] acpi_power_meter
[20377.176820] i2c_algo_bit
[20377.176820] drm
[20377.176822] sysimgblt
[20377.176822] iTCO_wdt
[20377.176823] i40e
[20377.176824] ixgbe
[20377.176826] crct10dif_common
[20377.176826] nvme_core
[20377.176827] nvme
[20377.176828] nvme
[20377.176829] mlx4_core(OE)
[20377.176831] nvme
[20377.176832] nvme
[20377.176833] nvme
[20377.176834] nfsd
[20377.176834] drm_kms_helper
[20377.176835] i40e
[20377.176837] ixgbe
[20377.176838] iTCO_vendor_support
[20377.176839] nvme
[20377.176840] fb_sys_fops
[20377.176843] crc32c_intel
[20377.176843] libata
[20377.176845] ahci
[20377.176846] ahci
[20377.176847] mlx_compat(OE)
[20377.176848] ahci
[20377.176849] ahci
[20377.176850] ahci
[20377.176851] auth_rpcgss
[20377.176852] syscopyarea
[20377.176853] nvme
[20377.176854] fb_sys_fops
[20377.176855] gf128mul
[20377.176856] ahci
[20377.176858] ttm
[20377.176860] drm
[20377.176861] mdio
[20377.176861] libahci
[20377.176862] libahci
[20377.176864] devlink
[20377.176865] libahci
[20377.176866] libahci
[20377.176867] libahci
[20377.176868] nfs_acl
[20377.176869] sysfillrect
[20377.176870] ahci
[20377.176871] ttm
[20377.176872] glue_helper
[20377.176874] libahci
[20377.176874] crct10dif_pclmul
[20377.176876] i40e
[20377.176876] mpt3sas(OE)
[20377.176878] nvme_core
[20377.176878] nvme_core
[20377.176880] xfs
[20377.176881] nvme_core
[20377.176883] nvme_core
[20377.176884] nvme_core
[20377.176885] sysimgblt
[20377.176886] libahci
[20377.176887] lockd
[20377.176888] crct10dif_pclmul
[20377.176888] ablk_helper
[20377.176890] nvme_core
[20377.176890] crct10dif_common
[20377.176892] nvme
[20377.176893] dca
[20377.176894] libata
[20377.176895] libata
[20377.176897] libcrc32c
[20377.176898] libata
[20377.176900] libata
[20377.176901] libata
[20377.176902] ixgbe
[20377.176903] nvme_core
[20377.176903] grace
[20377.176904] crct10dif_common
[20377.176905] cryptd
[20377.176907] libata
[20377.176907] crc32c_intel
[20377.176909] ahci
[20377.176910] raid_class
[20377.176911] mdio
[20377.176912] mdio
[20377.176914] ipmi_watchdog
[20377.176915] mdio
[20377.176916] mdio
[20377.176918] mdio
[20377.176919] fb_sys_fops
[20377.176919] libata
[20377.176920] sunrpc
[20377.176921] crc32c_intel
[20377.176922] pcspkr
[20377.176923] mdio
[20377.176924] drm
[20377.176926] libahci
[20377.176927] scsi_transport_sas
[20377.176928] mpt3sas(OE)
[20377.176929] mpt3sas(OE)
[20377.176930] dm_mirror
[20377.176932] mpt3sas(OE)
[20377.176933] mpt3sas(OE)
[20377.176934] mpt3sas(OE)
[20377.176935] ttm
[20377.176936] mdio
[20377.176937] knem(OE)
[20377.176938] drm
[20377.176939] joydev
[20377.176941] mpt3sas(OE)
[20377.176941] i40e
[20377.176943] nvme_core
[20377.176944] ptp
[20377.176945] dca
[20377.176946] dca
[20377.176949] dm_region_hash
[20377.176950] dca
[20377.176952] dca
[20377.176953] dca
[20377.176954] crct10dif_pclmul
[20377.176955] mpt3sas(OE)
[20377.176956] i40e
[20377.176957] ip_tables
[20377.176958] ses
[20377.176959] dca
[20377.176960] nvme
[20377.176961] libata
[20377.176962] i2c_core
[20377.176963] raid_class
[20377.176964] raid_class
[20377.176966] dm_log
[20377.176966] raid_class
[20377.176968] raid_class
[20377.176969] raid_class
[20377.176970] crct10dif_common
[20377.176970] dca
[20377.176971] nvme
[20377.176972] ext4
[20377.176973] enclosure
[20377.176975] raid_class
[20377.176976] ahci
[20377.176977] mdio
[20377.176978] pps_core
[20377.176979] scsi_transport_sas
[20377.176980] scsi_transport_sas
[20377.176981] dm_mod
[20377.176983] scsi_transport_sas
[20377.176984] scsi_transport_sas
[20377.176985] scsi_transport_sas
[20377.176986] crc32c_intel
[20377.176987] raid_class
[20377.176988] ahci
[20377.176989] mbcache
[20377.176990] sg
[20377.176991] scsi_transport_sas
[20377.176992] libahci
[20377.176993]
[20377.176994] mpt3sas(OE)
[20377.176996] ptp
[20377.176997] ptp
[20377.176998] vfat
[20377.176999] ptp
[20377.177001] ptp
[20377.177002] ptp
[20377.177003] drm
[20377.177004] scsi_transport_sas
[20377.177004] libahci
[20377.177006] jbd2
[20377.177006] mei_me
[20377.177008] ptp
[20377.177009] nvme_core
[20377.177012] CPU: 24 PID: 132 Comm: kworker/24:0H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177013] dca
[20377.177015] i2c_core
[20377.177015] i2c_core
[20377.177016] fat
[20377.177017] i2c_core
[20377.177019] i2c_core
[20377.177020] i2c_core
[20377.177021] i40e
[20377.177022] ptp
[20377.177022] nvme_core
[20377.177023] sd_mod
[20377.177024] mei
[20377.177026] i2c_core
[20377.177027] libata
[20377.177029] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177030] raid_class
[20377.177032] pps_core
[20377.177032] pps_core
[20377.177034] skx_edac
[20377.177035] pps_core
[20377.177036] pps_core
[20377.177037] pps_core
[20377.177038] nvme
[20377.177039] i2c_core
[20377.177040] libata
[20377.177041] crc_t10dif
[20377.177041] shpchp
[20377.177043] pps_core
[20377.177044] mdio
[20377.177048] scsi_transport_sas
[20377.177049]
[20377.177050]
[20377.177051] Workqueue: kblockd blk_mq_run_work_fn
[20377.177052]
[20377.177053]
[20377.177055]
[20377.177056] edac_core
[20377.177057] ahci
[20377.177058] pps_core
[20377.177059] mdio
[20377.177060]
[20377.177061] lpc_ich
[20377.177062] crct10dif_generic
[20377.177063] mpt3sas(OE)
[20377.177064]
[20377.177067] CPU: 0 PID: 634945 Comm: kworker/0:1H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177069] CPU: 16 PID: 381617 Comm: kworker/16:2H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177070] ptp
[20377.177073] CPU: 2 PID: 26104 Comm: kworker/2:3H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177077] CPU: 45 PID: 24259 Comm: kworker/45:2H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177081] CPU: 11 PID: 258071 Comm: kworker/11:2H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177082] coretemp
[20377.177083]
[20377.177083] libahci
[20377.177084] mpt3sas(OE)
[20377.177086] CPU: 6 PID: 4018486 Comm: kworker/6:2H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177087] i2c_i801
[20377.177088] ast
[20377.177089] dca
[20377.177091] Call Trace:
[20377.177092] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177094] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177095] i2c_core
[20377.177097] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177098] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177100] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177101] intel_rapl
[20377.177102] nvme_core
[20377.177106] CPU: 42 PID: 222 Comm: kworker/42:0H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177108] dca
[20377.177108] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177110] wmi
[20377.177110] i2c_algo_bit
[20377.177111] raid_class
[20377.177112] pps_core
[20377.177115] Workqueue: kblockd blk_mq_run_work_fn
[20377.177118] iosf_mbi
[20377.177123] Workqueue: kblockd blk_mq_run_work_fn
[20377.177126] Workqueue: kblockd blk_mq_run_work_fn
[20377.177127] Workqueue: kblockd blk_mq_run_work_fn
[20377.177130] libata
[20377.177131] Workqueue: kblockd blk_mq_run_work_fn
[20377.177133] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177134] raid_class
[20377.177136] ipmi_si
[20377.177137] Workqueue: kblockd blk_mq_run_work_fn
[20377.177138]
[20377.177139] scsi_transport_sas
[20377.177144] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177144]
[20377.177146] drm_kms_helper
[20377.177147]
[20377.177148]
[20377.177149]
[20377.177150] kvm_intel
[20377.177151]
[20377.177152] mdio
[20377.177154] scsi_transport_sas
[20377.177159] ipmi_devintf
[20377.177160]
[20377.177162] Workqueue: kblockd blk_mq_run_work_fn
[20377.177164] CPU: 41 PID: 4038 Comm: kworker/41:1H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177165] ptp
[20377.177166] syscopyarea
[20377.177167] Call Trace:
[20377.177169] Call Trace:
[20377.177171] Call Trace:
[20377.177172] Call Trace:
[20377.177173] kvm
[20377.177175] Call Trace:
[20377.177176] mpt3sas(OE)
[20377.177178] ptp
[20377.177179]
[20377.177180] Call Trace:
[20377.177181] ipmi_msghandler
[20377.177182] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177183] i2c_core
[20377.177187] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177188] sysfillrect
[20377.177189] irqbypass
[20377.177193] dca
[20377.177194] i2c_core
[20377.177196] Call Trace:
[20377.177196] nfit
[20377.177199] pps_core
[20377.177200] Workqueue: kblockd blk_mq_run_work_fn
[20377.177202] sysimgblt
[20377.177206] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177210] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177213] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177218] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177219] crc32_pclmul
[20377.177219] raid_class
[20377.177223] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177225] pps_core
[20377.177227] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177227] libnvdimm
[20377.177228]
[20377.177229]
[20377.177233] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177234] ixgbe
[20377.177235] ghash_clmulni_intel
[20377.177237]
[20377.177242] scsi_transport_sas
[20377.177243] acpi_power_meter
[20377.177243] Call Trace:
[20377.177245] fb_sys_fops
[20377.177249] CPU: 44 PID: 33806 Comm: kworker/44:3H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177253] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177256] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177258] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177261] CPU: 17 PID: 8437 Comm: kworker/17:1H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177264] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177267] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177268] aesni_intel
[20377.177269] ptp
[20377.177271] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177272] nfsd
[20377.177276] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177278] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177278] ttm
[20377.177282] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177284] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177285] lrw
[20377.177289] i2c_core
[20377.177289] auth_rpcgss
[20377.177292] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177294] crct10dif_pclmul
[20377.177300] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177302] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177304] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177307] Workqueue: kblockd blk_mq_run_work_fn
[20377.177310] Workqueue: kblockd blk_mq_run_work_fn
[20377.177312] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177315] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177316] iTCO_wdt
[20377.177318] pps_core
[20377.177319] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177320] nfs_acl
[20377.177323] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177324]
[20377.177325] crct10dif_common
[20377.177328] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177330]
[20377.177331] iTCO_vendor_support
[20377.177333]
[20377.177335] lockd
[20377.177337] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177338] crc32c_intel
[20377.177340] Call Trace:
[20377.177343] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177344] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177347] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177349] task: ffff882fdbe12f70 ti: ffff882dee624000 task.ti: ffff882dee624000
[20377.177352] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177355] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177358] CPU: 23 PID: 4058717 Comm: kworker/23:2H Tainted: G OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177359] gf128mul
[20377.177361] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177362] grace
[20377.177366] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177367] drm
[20377.177374] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177376] RIP: 0010:[<ffffffff811014e0>]
[20377.177377] glue_helper
[20377.177380] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177382] sunrpc
[20377.177383] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177385] i40e
[20377.177388] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177392] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177394] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177396] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177399] [<ffffffff811014e0>] native_queued_spin_lock_slowpath+0x110/0x200
[20377.177402] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177405] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177409] ablk_helper
[20377.177411] Workqueue: kblockd blk_mq_run_work_fn
[20377.177412] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177413] knem(OE)
[20377.177416] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177417] nvme
[20377.177422] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177424] RSP: 0018:ffff882dee627d30 EFLAGS: 00010202
[20377.177425] cryptd
[20377.177427]
[20377.177428] ip_tables
[20377.177430] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177432] ahci
[20377.177435] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177438] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177442] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177451] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177453] RAX: 0000000000001b86 RBX: ffff882fd2b24000 RCX: 0000000000890000
[20377.177458] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177462] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177463] Call Trace:
[20377.177464] pcspkr
[20377.177467] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177468] ext4
[20377.177473] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177474] libahci
[20377.177480] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177481] RDX: 0000000000019670 RSI: 00000000dc374960 RDI: ffff882fdc374960
[20377.177482] joydev
[20377.177485] mbcache
[20377.177487] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177489] nvme_core
[20377.177491] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177496] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177498] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177500] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177504] RBP: ffff882dee627d30 R08: ffff882fddb59640 R09: 0000000000000000
[20377.177508] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177512] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177515] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177516] ses
[20377.177518] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177519] jbd2
[20377.177523] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177524] libata
[20377.177529] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177531] R10: df403d66cad241f8 R11: 0000000000000000 R12: ffff882dee627d78
[20377.177532] enclosure
[20377.177535] sd_mod
[20377.177538] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177540] mdio
[20377.177543] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177546] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177549] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177552] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177554] R13: ffff882fdc374960 R14: 000000000000001d R15: ffff882fdc372080
[20377.177558] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177562] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177565] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177566] sg
[20377.177568] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177569] crc_t10dif
[20377.177573] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177574] mpt3sas(OE)
[20377.177579] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177581] FS: 0000000000000000(0000) GS:ffff882fddb40000(0000) knlGS:0000000000000000
[20377.177582] mei_me
[20377.177585] crct10dif_generic
[20377.177588] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177589] dca
[20377.177591] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177596] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177599] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177602] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20377.177610] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177613] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177616] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177617] mei
[20377.177619] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177620] ast
[20377.177622] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177624] raid_class
[20377.177631] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177633] CR2: 0000000000019670 CR3: 0000002fb5704000 CR4: 00000000003607e0
[20377.177634] shpchp
[20377.177638] i2c_algo_bit
[20377.177640] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177645] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177649] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177650] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177653] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177654] scsi_transport_sas
[20377.177655] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[20377.177659] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177663] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177665] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177666] lpc_ich
[20377.177668] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177669] drm_kms_helper
[20377.177671] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177673] ptp
[20377.177679] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[20377.177681] i2c_i801
[20377.177685] syscopyarea
[20377.177687] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177691] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177695] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177700] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177703] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177707] ---[ end trace fc8a3c22122718ef ]---
[20377.177707] Call Trace:
[20377.177708] i2c_core
[20377.177713] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177717] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177719] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177720] wmi
[20377.177723] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177724] sysfillrect
[20377.177727] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177729] pps_core
[20377.177733] ipmi_si
[20377.177736] sysimgblt
[20377.177738] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177740]
[20377.177743] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177747] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177749] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177752] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177755] [<ffffffff816b0773>] queued_spin_lock_slowpath+0xb/0xf
[20377.177759] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177762] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177766] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177767] ipmi_devintf
[20377.177768] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177769] ixgbe
[20377.177772] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177774] CPU: 40 PID: 25124 Comm: kworker/40:2H Tainted: G W OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177777] ---[ end trace fc8a3c22122718f0 ]---
[20377.177779] ---[ end trace fc8a3c22122718f1 ]---
[20377.177781] ---[ end trace fc8a3c22122718f2 ]---
[20377.177783] ---[ end trace fc8a3c22122718f3 ]---
[20377.177784] ipmi_msghandler
[20377.177786] ---[ end trace fc8a3c22122718f4 ]---
[20377.177789] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177790] ---[ end trace fc8a3c22122718f5 ]---
[20377.177791] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177794] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177797] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177802] [<ffffffff816be0d0>] _raw_spin_lock+0x20/0x30
[20377.177803] fb_sys_fops
[20377.177805] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177807] nfit
[20377.177808] ttm
[20377.177810] Workqueue: kblockd blk_mq_run_work_fn
[20377.177811] ---[ end trace fc8a3c22122718f6 ]---
[20377.177813] libnvdimm
[20377.177816] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177817]
[20377.177820] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177822] [<ffffffff8130eaef>] blk_mq_process_rq_list+0xcf/0x370
[20377.177822] crct10dif_pclmul
[20377.177825] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177827] acpi_power_meter
[20377.177828] ---[ end trace fc8a3c22122718f7 ]---
[20377.177829] Call Trace:
[20377.177829] crct10dif_common
[20377.177831] nfsd
[20377.177833] crc32c_intel
[20377.177837] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177838] [<ffffffff8130ee2a>] __blk_mq_run_hw_queue+0x9a/0xb0
[20377.177841] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177842] auth_rpcgss
[20377.177844] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177845] drm
[20377.177846] nfs_acl
[20377.177849] i40e
[20377.177852] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177854] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177856] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177858] lockd
[20377.177859] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177862] ---[ end trace fc8a3c22122718f8 ]---
[20377.177862] nvme
[20377.177863] grace
[20377.177866] ahci
[20377.177868] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177871] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177873] sunrpc
[20377.177875] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177876] libahci
[20377.177877] knem(OE)
[20377.177878] nvme_core
[20377.177881] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177884] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177885] ip_tables
[20377.177887] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177888] libata
[20377.177889] ---[ end trace fc8a3c22122718f9 ]---
[20377.177890] ext4
[20377.177891] mdio
[20377.177894] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177895] mbcache
[20377.177897] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.177898] mpt3sas(OE)
[20377.177900] jbd2
[20377.177901] dca
[20377.177903] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177905] sd_mod
[20377.177907] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.177908] raid_class
[20377.177910] crc_t10dif
[20377.177912] scsi_transport_sas
[20377.177914] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177915] crct10dif_generic
[20377.177918] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.177918] ptp
[20377.177921] ast
[20377.177921] i2c_core
[20377.177923] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177925] i2c_algo_bit
[20377.177927] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.177928] pps_core
[20377.177929]
[20377.177930] drm_kms_helper
[20377.177932] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177934] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.177938] CPU: 1 PID: 4045642 Comm: kworker/1:0H Tainted: G W OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.177939] syscopyarea
[20377.177940] Code:
[20377.177940] sysfillrect
[20377.177942] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.177944] 87
[20377.177946] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177947] sysimgblt
[20377.177950] 47
[20377.177952] Workqueue: kblockd blk_mq_run_work_fn
[20377.177952] ixgbe
[20377.177954]
[20377.177955] 02
[20377.177957] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.177958] fb_sys_fops
[20377.177959] Call Trace:
[20377.177960] c1
[20377.177962] ttm
[20377.177963] e0
[20377.177964] crct10dif_pclmul
[20377.177966] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.177967] 10
[20377.177971] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.177972] crct10dif_common
[20377.177973] ---[ end trace fc8a3c22122718fa ]---
[20377.177974] 45
[20377.177974] crc32c_intel
[20377.177977] 31
[20377.177980] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.177981] drm
[20377.177982] c9
[20377.177983] i40e
[20377.177985] 85
[20377.177987] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.177988] nvme
[20377.177989] c0
[20377.177990] ahci
[20377.177992] 74
[20377.177995] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.177996] libahci
[20377.177997] 44
[20377.177998] nvme_core
[20377.178000] 48
[20377.178003] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.178004] libata
[20377.178006] 89
[20377.178007] mdio
[20377.178009] c2
[20377.178013] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.178016] mpt3sas(OE)
[20377.178016] c1
[20377.178017] dca
[20377.178019] e8
[20377.178023] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.178024] raid_class
[20377.178026] 13
[20377.178027] scsi_transport_sas
[20377.178028] 48
[20377.178032] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.178033] ptp
[20377.178034] c1
[20377.178035] i2c_core
[20377.178037] ea
[20377.178040] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.178041] pps_core
[20377.178042]
[20377.178043] 0d
[20377.178047] CPU: 12 PID: 4046955 Comm: kworker/12:0H Tainted: G W OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
[20377.178050] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.178053] Hardware name: Inspur AS13000G5-M12/YZMB-00882-103, BIOS 4.1.13 02/21/2020
[20377.178054] 48 98
[20377.178058] 83
[20377.178062] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.178063] Workqueue: kblockd blk_mq_run_work_fn
[20377.178065]
[20377.178067] e2
[20377.178069] Call Trace:
[20377.178072] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.178074] 30 48
[20377.178075] ---[ end trace fc8a3c22122718fb ]---
[20377.178078] [<ffffffff816b5f65>] dump_stack+0x19/0x1b
[20377.178079] 81 c2
[20377.178081] [<ffffffff8108e118>] __warn+0xd8/0x100
[20377.178082] 40 96
[20377.178084] [<ffffffff8108e25d>] warn_slowpath_null+0x1d/0x20
[20377.178084] 01 00
[20377.178086] [<ffffffff8130ee19>] __blk_mq_run_hw_queue+0x89/0xb0
[20377.178087] 48 03
[20377.178089] [<ffffffff8130f675>] blk_mq_run_work_fn+0x15/0x20
[20377.178090] 14 c5
[20377.178092] [<ffffffff810af51f>] process_one_work+0x17f/0x440
[20377.178093] 20 d5
[20377.178095] [<ffffffff810b01e6>] worker_thread+0x126/0x3c0
[20377.178096] b2 81
[20377.178099] [<ffffffff810b00c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[20377.178099] <4c> 89
[20377.178102] [<ffffffff810b74d1>] kthread+0xd1/0xe0
[20377.178103] 02 41
[20377.178105] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.178106] 8b 40
[20377.178109] [<ffffffff816c851d>] ret_from_fork+0x5d/0xb0
[20377.178112] 08 85
[20377.178114] [<ffffffff810b7400>] ? insert_kthread_work+0x40/0x40
[20377.178115] ---[ end trace fc8a3c22122718fc ]---
[20377.178117] c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40
[20377.178119] RIP [<ffffffff811014e0>] native_queued_spin_lock_slowpath+0x110/0x200
[20377.178119] RSP <ffff882dee627d30>
[20377.178120] CR2: 0000000000019670
Steps To Reproducefirst: read & write nvme disk by a storage cluster, a vendbech
second: remove nvme disk by hand, the kernel crash and machine reboot
TagsNo tags attached.
abrt_hash
URL

Activities

ManuelWolfshant

ManuelWolfshant

2020-05-15 10:01

manager   ~0036933

1. You did not explicitly specify which kernel version is used but the logs show
 [20377.178047] CPU: 12 PID: 4046955 Comm: kworker/12:0H Tainted: G W OE ------------ 3.10.0-693.50.3.el7.x86_64 #1
3.10.693 was used in CentOS 7.4 is not supported for almost 3 years. A lot of security and bugfixes were included in the OS meanwhile so please upgrade ALL your system to CentOS 7.8.2003 and let us know if you still encounter the problem.

2. Your log includes the following lines:
[20376.404330] XFS (nvme0n1p12): metadata I/O error: block 0x3dc784d0 ("xlog_iodone") error 5 numblks 64
[20376.404385] XFS (nvme0n1p12): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0831c20
[20376.404395] XFS (nvme0n1p12): Log I/O Error Detected. Shutting down filesystem
[20376.404431] XFS (nvme0n1p12): Please umount the filesystem and rectify the problem(s)

These lines, together with a few more from the log tell us that you are attempting to remove the NVME disk while it is still actively written to by the OS and without umounting the filesystem first. Are you attempting to voluntarily create a data loss situation ?
hejiwen

hejiwen

2020-05-15 10:24

reporter   ~0036934

Thank you for your comments!

1、CentOS7.4 kernel version: 3.10.0-693.50.3.el7.x86_64
2、Yes, I need this situation (remove nvme disk when it has IOs), and can accept data loss.

Can you give me some advices to solve this problem, looking forward to your reply, thank you very much.
ManuelWolfshant

ManuelWolfshant

2020-05-15 10:39

manager   ~0036935

I have told you already. Update the system. You are missing a huge number of fixes.
hejiwen

hejiwen

2020-05-15 11:16

reporter   ~0036936

Thank you very much。
This issue only appears once, and we try reproduce this issue beyond 100+ times, by now it didn‘t occur.
I am not sure whether solve it or not if I upgrade the kernel。
I really want to know the root cause of this problem,which module trigger this issue, spin_lock or cpumask? how to find the patch?
ManuelWolfshant

ManuelWolfshant

2020-05-15 11:56

manager   ~0036937

For technical and lack of resources reasons, CentOS does not support anything but the latest minor release of each major version of the distribution. Even the RHEL 7.4 Extended Update support ended an year ago.
Feel free to open another bug if you can recreate the crash after using the latest updates.

Issue History

Date Modified Username Field Change
2020-05-15 09:40 hejiwen New Issue
2020-05-15 10:01 ManuelWolfshant Note Added: 0036933
2020-05-15 10:24 hejiwen Note Added: 0036934
2020-05-15 10:39 ManuelWolfshant Note Added: 0036935
2020-05-15 11:16 hejiwen Note Added: 0036936
2020-05-15 11:56 ManuelWolfshant Status new => closed
2020-05-15 11:56 ManuelWolfshant Resolution open => not fixable
2020-05-15 11:56 ManuelWolfshant Note Added: 0036937