0016118: hang on blk_mq_freeze_queue_wait - CentOS Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0016118	CentOS-7	kernel	public	2019-05-28 07:42	2021-01-27 17:32

Reporter	Leno Hou	Assigned To
Priority	high	Severity	block	Reproducibility	random
Status	new	Resolution	open
Product Version	7.6.1810

Summary	0016118: hang on blk_mq_freeze_queue_wait
Description	This description was written by https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7996a8b5511a72465b0b286763c2d8f412b8874a [ 1211.265045] nvme 0000:00:1e.0: irq 63 for MSI/MSI-X [ 1211.266366] do_IRQ: 18.177 No irq handler for vector (irq -1) [ 1324.016738] INFO: task kworker/u146:5:36022 blocked for more than 120 seconds. [ 1324.016742] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1324.016744] kworker/u146:5 D ffff972e18a36180 0 36022 2 0x00000080 [ 1324.016768] Workqueue: nvme-wq nvme_remove_dead_ctrl_work [nvme] [ 1324.016769] Call Trace: [ 1324.016780] [<ffffffff92768e19>] schedule+0x29/0x70 [ 1324.016787] [<ffffffff9234f675>] blk_mq_freeze_queue_wait+0x75/0xe0 [ 1324.016794] [<ffffffff920c2df0>] ? wake_up_atomic_t+0x30/0x30 [ 1324.016797] [<ffffffff92350434>] blk_freeze_queue+0x24/0x50 [ 1324.016801] [<ffffffff9234490f>] blk_cleanup_queue+0x7f/0x1b0 [ 1324.016808] [<ffffffffc036355a>] nvme_ns_remove+0x9a/0xc0 [nvme_core] [ 1324.016812] [<ffffffffc036370b>] nvme_remove_namespaces+0x9b/0xe0 [nvme_core] [ 1324.016816] [<ffffffffc0463882>] nvme_remove+0x92/0x140 [nvme] [ 1324.016822] [<ffffffff923c68ce>] pci_device_remove+0x3e/0xc0 [ 1324.016828] [<ffffffff924a8972>] __device_release_driver+0x82/0xf0 [ 1324.016830] [<ffffffff924a8a03>] device_release_driver+0x23/0x30 [ 1324.016834] [<ffffffffc0461828>] nvme_remove_dead_ctrl_work+0x28/0x40 [nvme] [ 1324.016839] [<ffffffff920b9e3f>] process_one_work+0x17f/0x440 [ 1324.016842] [<ffffffff920baed6>] worker_thread+0x126/0x3c0 [ 1324.016844] [<ffffffff920badb0>] ? manage_workers.isra.25+0x2a0/0x2a0 [ 1324.016847] [<ffffffff920c1d21>] kthread+0xd1/0xe0 [ 1324.016849] [<ffffffff920c1c50>] ? insert_kthread_work+0x40/0x40 [ 1324.016855] [<ffffffff92775c37>] ret_from_fork_nospec_begin+0x21/0x21 [ 1324.016857] [<ffffffff920c1c50>] ? insert_kthread_work+0x40/0x40 [ 1444.015706] INFO: task kworker/u144:1:427 blocked for more than 120 seconds. [ 1444.015710] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1444.015712] kworker/u144:1 D ffff971c2dcca080 0 427 2 0x00000000 [ 1444.015724] Workqueue: kacpi_hotplug acpi_hotplug_work_fn [ 1444.015726] Call Trace: [ 1444.015736] [<ffffffff92769d39>] schedule_preempt_disabled+0x29/0x70 [ 1444.015739] [<ffffffff92767cb7>] __mutex_lock_slowpath+0xc7/0x1d0 [ 1444.015743] [<ffffffff9276709f>] mutex_lock+0x1f/0x2f [ 1444.015748] [<ffffffff924a89fb>] device_release_driver+0x1b/0x30 [ 1444.015754] [<ffffffff923be6f4>] pci_stop_bus_device+0x84/0xa0 [ 1444.015757] [<ffffffff923be802>] pci_stop_and_remove_bus_device+0x12/0x20 [ 1444.015762] [<ffffffff923e1fe6>] disable_slot+0x76/0xd0 [ 1444.015765] [<ffffffff923e205d>] acpiphp_disable_and_eject_slot+0x1d/0x80 [ 1444.015767] [<ffffffff923e306d>] acpiphp_hotplug_notify+0x8d/0x210 [ 1444.015771] [<ffffffff923e2fe0>] ? acpiphp_post_dock_fixup+0xd0/0xd0 [ 1444.015774] [<ffffffff924091de>] acpi_device_hotplug+0x3b7/0x41a [ 1444.015776] [<ffffffff924022a7>] acpi_hotplug_work_fn+0x1e/0x29 [ 1444.015784] [<ffffffff920b9e3f>] process_one_work+0x17f/0x440 [ 1444.015787] [<ffffffff920baed6>] worker_thread+0x126/0x3c0 [ 1444.015789] [<ffffffff920badb0>] ? manage_workers.isra.25+0x2a0/0x2a0 [ 1444.015792] [<ffffffff920c1d21>] kthread+0xd1/0xe0 [ 1444.015799] [<ffffffff920c1c50>] ? insert_kthread_work+0x40/0x40 [ 1444.015805] [<ffffffff92775c37>] ret_from_fork_nospec_begin+0x21/0x21 [ 1444.015807] [<ffffffff920c1c50>] ? insert_kthread_work+0x40/0x40
Steps To Reproduce	Detach NVMe based Disk
Tags	3.10.0-975.12.1.el7.x86_64

abrt_hash
URL

Leno Hou 2019-05-28 07:42 reporter ~0034524	Could you pick up patch and backport into CentOS 7

whou 2021-01-27 11:28 reporter ~0038203 Last edited: 2021-01-27 17:32	@Leno Hou, Do you have the full log in description? Have you observed two kernel warning before the hang task? This kernel defect, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7996a8b5511a72465b0b286763c2d8f412b8874a , is hardly reproduced and if it is, two kernel warning must be shown before. ---- WARNING at lib/percpu-recount.c:309 percpu_ref_kill_and_confirm called more than once ----

Date Modified	Username	Field	Change
2019-05-28 07:42	Leno Hou	New Issue
2019-05-28 07:42	Leno Hou	Tag Attached: 3.10.0-975.12.1.el7.x86_64
2019-05-28 07:42	Leno Hou	Note Added: 0034524
2021-01-27 11:28	whou	Note Added: 0038203
2021-01-27 17:32	toracat	Note Edited: 0038203