View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0016711 | CentOS-7 | kernel | public | 2019-11-09 08:44 | 2021-04-24 09:35 |
Reporter | bruceleeeee | Assigned To | |||
Priority | high | Severity | crash | Reproducibility | random |
Status | new | Resolution | open | ||
Platform | x86_64 | OS | CentOS Linux 7 (Core) | OS Version | 7.7 |
Product Version | 7.7-1908 | ||||
Summary | 0016711: xfsaild blocks after certain time | ||||
Description | problem similar to https://bugs.centos.org/view.php?id=13843 kernel version: 3.10.0-1062.4.1.el7.x86_64 started to have problem where xfsaild becomes blocked after a period of time and processes wanting to access the file system begin to grind to a halt. Most often hits chatty IO applications such as auditd first. A reboot solves the problem until the next time. The file system is in a mirrored volume group and neither the volume group nor individual disk reports any problems. | ||||
Additional Information | Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70 Nov 8 09:23:28 server1 kernel: [<ffffffffc02661ed>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff993bcf1f>] ? get_target_pstate_use_performance+0x8f/0xc0 Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20 Nov 8 09:23:28 server1 kernel: [<ffffffffc02668d9>] xlog_write+0x1a9/0x750 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc0263827>] ? kmem_zone_alloc+0x97/0x130 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc0268708>] xlog_cil_push+0x2a8/0x430 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc02688a5>] xlog_cil_push_work+0x15/0x20 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd1df>] process_one_work+0x17f/0x440 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe448>] worker_thread+0x278/0x3c0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40 Nov 8 09:23:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40 Nov 8 09:23:28 server1 kernel: INFO: task kworker/u32:2:23070 blocked for more than 120 seconds. Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 8 09:23:28 server1 kernel: kworker/u32:2 D ffff95e9fadea0e0 0 23070 2 0x00000080 Nov 8 09:23:28 server1 kernel: Workqueue: writeback bdi_writeback_workfn (flush-9:125) Nov 8 09:23:28 server1 kernel: Call Trace: Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70 Nov 8 09:23:28 server1 kernel: [<ffffffff995804f5>] rwsem_down_read_failed+0x105/0x1c0 Nov 8 09:23:28 server1 kernel: [<ffffffffc023c047>] ? xfs_map_blocks+0x87/0x220 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff991913f8>] call_rwsem_down_read_failed+0x18/0x30 Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc90>] down_read+0x20/0x40 Nov 8 09:23:28 server1 kernel: [<ffffffffc0255d69>] xfs_ilock+0xd9/0x120 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc023c047>] xfs_map_blocks+0x87/0x220 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc023d1a4>] xfs_do_writepage+0x174/0x550 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff98fc82ac>] write_cache_pages+0x21c/0x470 Nov 8 09:23:28 server1 kernel: [<ffffffffc023d030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff9914f710>] ? submit_bio+0x70/0x150 Nov 8 09:23:28 server1 kernel: [<ffffffffc023cffb>] xfs_vm_writepages+0x6b/0xa0 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff98fc92f1>] do_writepages+0x21/0x50 Nov 8 09:23:28 server1 kernel: [<ffffffff99077f70>] __writeback_single_inode+0x40/0x260 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec6265>] ? wake_up_bit+0x25/0x30 Nov 8 09:23:28 server1 kernel: [<ffffffff99078b04>] writeback_sb_inodes+0x1c4/0x430 Nov 8 09:23:28 server1 kernel: [<ffffffff99078e0f>] __writeback_inodes_wb+0x9f/0xd0 Nov 8 09:23:28 server1 kernel: [<ffffffff990792f3>] wb_writeback+0x263/0x2f0 Nov 8 09:23:28 server1 kernel: [<ffffffff9906519c>] ? get_nr_inodes+0x4c/0x70 Nov 8 09:23:28 server1 kernel: [<ffffffff99079eeb>] bdi_writeback_workfn+0x2cb/0x460 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd1df>] process_one_work+0x17f/0x440 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe2f6>] worker_thread+0x126/0x3c0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40 Nov 8 09:23:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21 Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40 Nov 8 09:23:28 server1 kernel: INFO: task dd:27364 blocked for more than 120 seconds. Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 8 09:23:28 server1 kernel: dd D ffff95e5cc4820e0 0 27364 1 0x00000080 Nov 8 09:23:28 server1 kernel: Call Trace: Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70 Nov 8 09:23:28 server1 kernel: [<ffffffff9957c491>] schedule_timeout+0x221/0x2d0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ed6619>] ? ttwu_do_wakeup+0x19/0xe0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ed674f>] ? ttwu_do_activate+0x6f/0x80 Nov 8 09:23:28 server1 kernel: [<ffffffff98ed9ed0>] ? try_to_wake_up+0x190/0x390 Nov 8 09:23:28 server1 kernel: [<ffffffff9957eebd>] wait_for_completion+0xfd/0x140 Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd96a>] flush_work+0x10a/0x1b0 Nov 8 09:23:28 server1 kernel: [<ffffffff98eba680>] ? move_linked_works+0x90/0x90 Nov 8 09:23:28 server1 kernel: [<ffffffffc02690da>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc02674a4>] _xfs_log_force_lsn+0x74/0x310 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff98fbb22f>] ? filemap_fdatawait_range+0x1f/0x30 Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc82>] ? down_read+0x12/0x40 Nov 8 09:23:28 server1 kernel: [<ffffffffc0247a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff9907e2a7>] do_fsync+0x67/0xb0 Nov 8 09:23:28 server1 kernel: [<ffffffff9907e5b3>] SyS_fdatasync+0x13/0x20 Nov 8 09:23:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a Nov 8 09:23:28 server1 kernel: INFO: task exim:27365 blocked for more than 120 seconds. Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 8 09:23:28 server1 kernel: exim D ffff95e95a2e5230 0 27365 27362 0x00000080 Nov 8 09:23:28 server1 kernel: Call Trace: Nov 8 09:23:28 server1 kernel: [<ffffffff98ed56de>] ? resched_curr+0xae/0xc0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ee49a6>] ? check_preempt_wakeup+0x166/0x250 Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70 Nov 8 09:23:28 server1 kernel: [<ffffffff9957c491>] schedule_timeout+0x221/0x2d0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ed6619>] ? ttwu_do_wakeup+0x19/0xe0 Nov 8 09:23:28 server1 kernel: [<ffffffff98ed674f>] ? ttwu_do_activate+0x6f/0x80 Nov 8 09:23:28 server1 kernel: [<ffffffff98ed9ed0>] ? try_to_wake_up+0x190/0x390 Nov 8 09:23:28 server1 kernel: [<ffffffff9957eebd>] wait_for_completion+0xfd/0x140 Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20 Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd96a>] flush_work+0x10a/0x1b0 Nov 8 09:23:28 server1 kernel: [<ffffffff98eba680>] ? move_linked_works+0x90/0x90 Nov 8 09:23:28 server1 kernel: [<ffffffffc02690da>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffffc02674a4>] _xfs_log_force_lsn+0x74/0x310 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff98fbb22f>] ? filemap_fdatawait_range+0x1f/0x30 Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc82>] ? down_read+0x12/0x40 Nov 8 09:23:28 server1 kernel: [<ffffffffc0247a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs] Nov 8 09:23:28 server1 kernel: [<ffffffff9907e2a7>] do_fsync+0x67/0xb0 Nov 8 09:23:28 server1 kernel: [<ffffffff9907e590>] SyS_fsync+0x10/0x20 Nov 8 09:23:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a Nov 8 09:25:28 server1 kernel: INFO: task xfsaild/md125:820 blocked for more than 120 seconds. Nov 8 09:25:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 8 09:25:28 server1 kernel: xfsaild/md125 D ffff95e9fbf69070 0 820 2 0x00000000 Nov 8 09:25:28 server1 kernel: Call Trace: Nov 8 09:25:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70 Nov 8 09:25:28 server1 kernel: [<ffffffffc0267136>] _xfs_log_force+0x1c6/0x2a0 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20 Nov 8 09:25:28 server1 kernel: [<ffffffffc0273550>] ? xfsaild+0x180/0x760 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc026723c>] xfs_log_force+0x2c/0x70 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc02733d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc0273550>] xfsaild+0x180/0x760 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc02733d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0 Nov 8 09:25:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40 Nov 8 09:25:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21 Nov 8 09:25:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40 Nov 8 09:25:28 server1 kernel: INFO: task auditd:1348 blocked for more than 120 seconds. Nov 8 09:25:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 8 09:25:28 server1 kernel: auditd D ffffffff9957e465 0 1348 1 0x00000000 Nov 8 09:25:28 server1 kernel: Call Trace: Nov 8 09:25:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70 Nov 8 09:25:28 server1 kernel: [<ffffffff99580245>] rwsem_down_write_failed+0x215/0x3c0 Nov 8 09:25:28 server1 kernel: [<ffffffffc0263ee4>] ? xlog_grant_head_check+0x54/0x100 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc025307d>] ? xfs_vn_update_time+0xcd/0x150 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffff99191427>] call_rwsem_down_write_failed+0x17/0x30 Nov 8 09:25:28 server1 kernel: [<ffffffff9957dcdd>] down_write+0x2d/0x3d Nov 8 09:25:28 server1 kernel: [<ffffffffc0255d51>] xfs_ilock+0xc1/0x120 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc025307d>] xfs_vn_update_time+0xcd/0x150 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffff990663a8>] update_time+0x28/0xd0 Nov 8 09:25:28 server1 kernel: [<ffffffff98ed2723>] ? __wake_up+0x13/0x20 Nov 8 09:25:28 server1 kernel: [<ffffffff990664f0>] file_update_time+0xa0/0xf0 Nov 8 09:25:28 server1 kernel: [<ffffffffc0248adf>] xfs_file_aio_write_checks+0x16f/0x1c0 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc024935a>] xfs_file_buffered_aio_write+0xca/0x2c0 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffffc02496dd>] xfs_file_aio_write+0x18d/0x1b0 [xfs] Nov 8 09:25:28 server1 kernel: [<ffffffff99048433>] do_sync_write+0x93/0xe0 Nov 8 09:25:28 server1 kernel: [<ffffffff99048f20>] vfs_write+0xc0/0x1f0 Nov 8 09:25:28 server1 kernel: [<ffffffff99049d3f>] SyS_write+0x7f/0xf0 Nov 8 09:25:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a | ||||
Tags | 3.10.0-1062.1.1.el7.x86_64, 7.7, centos 7, file system | ||||
abrt_hash | |||||
URL | |||||
The same here: Centos7 3.10.0-1062.9.1.el7.x86_64 <code> Jan 10 00:12:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds. Jan 10 00:12:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:12:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000 Jan 10 00:12:55 x3421 kernel: Call Trace: Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90 Jan 10 00:12:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:12:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs] Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs] Jan 10 00:12:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs] Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:12:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:14:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds. Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:14:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000 Jan 10 00:14:55 x3421 kernel: Call Trace: Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:14:55 x3421 kernel: INFO: task kworker/0:2:18279 blocked for more than 120 seconds. Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:14:55 x3421 kernel: kworker/0:2 D ffff9b4a1511e2a0 0 18279 2 0x00000080 Jan 10 00:14:55 x3421 kernel: Workqueue: xfs-sync/md125 xfs_log_worker [xfs] Jan 10 00:14:55 x3421 kernel: Call Trace: Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8e83b4>] ? update_group_power+0x174/0x280 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cb81635>] ? cpumask_next_and+0x35/0x50 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90 Jan 10 00:14:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8aecbd>] ? add_timer_on+0x8d/0xf0 Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dcfd6>] _xfs_log_force+0x86/0x2a0 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd296>] ? xfs_log_worker+0x36/0x100 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd296>] xfs_log_worker+0x36/0x100 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf336>] worker_thread+0x126/0x3c0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:14:55 x3421 kernel: INFO: task kworker/0:3:7799 blocked for more than 120 seconds. Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:14:55 x3421 kernel: kworker/0:3 D ffff9b4dfec1ac80 0 7799 2 0x00000080 Jan 10 00:14:55 x3421 kernel: Workqueue: xfs-cil/md125 xlog_cil_push_work [xfs] Jan 10 00:14:55 x3421 kernel: Call Trace: Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dc1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dc8b9>] xlog_write+0x1a9/0x750 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06d9807>] ? kmem_zone_alloc+0x97/0x130 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06de6e8>] xlog_cil_push+0x2a8/0x430 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffffc06de885>] xlog_cil_push_work+0x15/0x20 [xfs] Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf488>] worker_thread+0x278/0x3c0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds. Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:16:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000 Jan 10 00:16:55 x3421 kernel: Call Trace: Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: INFO: task grafana-server:27514 blocked for more than 120 seconds. Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:16:55 x3421 kernel: grafana-server D ffff9b4dfed1ac80 0 27514 4384 0x00000080 Jan 10 00:16:55 x3421 kernel: Call Trace: Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d77a9>] ? ttwu_do_wakeup+0x19/0xe0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d78df>] ? ttwu_do_activate+0x6f/0x80 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db200>] ? try_to_wake_up+0x190/0x390 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd484>] _xfs_log_force_lsn+0x74/0x310 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c9bcaff>] ? filemap_fdatawait_range+0x1f/0x30 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7fd02>] ? down_read+0x12/0x40 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06bda3d>] xfs_file_fsync+0xfd/0x1c0 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fb77>] do_fsync+0x67/0xb0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fe60>] SyS_fsync+0x10/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dede>] system_call_fastpath+0x25/0x2a Jan 10 00:16:55 x3421 kernel: INFO: task kworker/0:2:18279 blocked for more than 120 seconds. Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:16:55 x3421 kernel: kworker/0:2 D ffff9b4a1511e2a0 0 18279 2 0x00000080 Jan 10 00:16:55 x3421 kernel: Workqueue: xfs-sync/md125 xfs_log_worker [xfs] Jan 10 00:16:55 x3421 kernel: Call Trace: Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8e83b4>] ? update_group_power+0x174/0x280 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cb81635>] ? cpumask_next_and+0x35/0x50 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8aecbd>] ? add_timer_on+0x8d/0xf0 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dcfd6>] _xfs_log_force+0x86/0x2a0 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd296>] ? xfs_log_worker+0x36/0x100 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd296>] xfs_log_worker+0x36/0x100 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf336>] worker_thread+0x126/0x3c0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: INFO: task kworker/0:3:7799 blocked for more than 120 seconds. Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:16:55 x3421 kernel: kworker/0:3 D ffff9b4dfec1ac80 0 7799 2 0x00000080 Jan 10 00:16:55 x3421 kernel: Workqueue: xfs-cil/md125 xlog_cil_push_work [xfs] Jan 10 00:16:55 x3421 kernel: Call Trace: Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dc1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dc8b9>] xlog_write+0x1a9/0x750 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06d9807>] ? kmem_zone_alloc+0x97/0x130 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06de6e8>] xlog_cil_push+0x2a8/0x430 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06de885>] xlog_cil_push_work+0x15/0x20 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf488>] worker_thread+0x278/0x3c0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:16:55 x3421 kernel: INFO: task redis-server:14049 blocked for more than 120 seconds. Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:16:55 x3421 kernel: redis-server D ffff9b4dfec1ac80 0 14049 4378 0x00000080 Jan 10 00:16:55 x3421 kernel: Call Trace: Jan 10 00:16:55 x3421 kernel: [<ffffffff9c82b621>] ? __switch_to+0x151/0x580 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d78df>] ? ttwu_do_activate+0x6f/0x80 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd484>] _xfs_log_force_lsn+0x74/0x310 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9c9bcaff>] ? filemap_fdatawait_range+0x1f/0x30 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7fd02>] ? down_read+0x12/0x40 Jan 10 00:16:55 x3421 kernel: [<ffffffffc06bda3d>] xfs_file_fsync+0xfd/0x1c0 [xfs] Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fb77>] do_fsync+0x67/0xb0 Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fe60>] SyS_fsync+0x10/0x20 Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dede>] system_call_fastpath+0x25/0x2a Jan 10 00:18:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds. Jan 10 00:18:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 10 00:18:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000 Jan 10 00:18:55 x3421 kernel: Call Trace: Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90 Jan 10 00:18:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70 Jan 10 00:18:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs] Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20 Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs] Jan 10 00:18:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs] Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0 Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 Jan 10 00:18:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21 Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40 </code> |
|
Centos7 3.10.0-1062.9.1.el7.x86_64 and 3.10.0-1062.4.3.el7.x86_64 kernels here. Several servers with different storage configuration are displaying the issues. All with same motherboard Supermicro X9DRW-iF messages.txt (21,918 bytes)
Jan 14 00:03:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds. Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:03:03 caro kernel: kworker/14:0 D ffff9e847f11ac80 0 84673 2 0x00000080 Jan 14 00:03:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs] Jan 14 00:03:03 caro kernel: Call Trace: Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:03:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:03:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440 Jan 14 00:03:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0 Jan 14 00:03:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 14 00:03:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0 Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:03:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21 Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:03:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds. Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:03:03 caro kernel: kworker/14:2 D ffff9e8470fd9070 0 85328 2 0x00000080 Jan 14 00:03:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs] Jan 14 00:03:03 caro kernel: Call Trace: Jan 14 00:03:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280 Jan 14 00:03:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50 Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:03:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:03:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140 Jan 14 00:03:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:03:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0 Jan 14 00:03:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90 Jan 14 00:03:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0 Jan 14 00:03:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580 Jan 14 00:03:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440 Jan 14 00:03:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0 Jan 14 00:03:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 14 00:03:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0 Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:03:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21 Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:03:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds. Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:03:03 caro kernel: rsync D ffff9e825abe5230 0 86419 86399 0x00000080 Jan 14 00:03:03 caro kernel: Call Trace: Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:03:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:03:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104 Jan 14 00:03:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f Jan 14 00:03:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50 Jan 14 00:03:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs] Jan 14 00:03:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140 Jan 14 00:03:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290 Jan 14 00:03:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0 Jan 14 00:03:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0 Jan 14 00:03:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170 Jan 14 00:03:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220 Jan 14 00:03:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20 Jan 14 00:03:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a Jan 14 00:05:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds. Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:05:03 caro kernel: kworker/14:0 D ffff9e847f11ac80 0 84673 2 0x00000080 Jan 14 00:05:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs] Jan 14 00:05:03 caro kernel: Call Trace: Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:05:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:05:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440 Jan 14 00:05:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0 Jan 14 00:05:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 14 00:05:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0 Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:05:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21 Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:05:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds. Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:05:03 caro kernel: kworker/14:2 D ffff9e8470fd9070 0 85328 2 0x00000080 Jan 14 00:05:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs] Jan 14 00:05:03 caro kernel: Call Trace: Jan 14 00:05:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280 Jan 14 00:05:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50 Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:05:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140 Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:05:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0 Jan 14 00:05:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90 Jan 14 00:05:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0 Jan 14 00:05:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580 Jan 14 00:05:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440 Jan 14 00:05:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0 Jan 14 00:05:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 14 00:05:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0 Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:05:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21 Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:05:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds. Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:05:03 caro kernel: rsync D ffff9e825abe5230 0 86419 86399 0x00000080 Jan 14 00:05:03 caro kernel: Call Trace: Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:05:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104 Jan 14 00:05:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f Jan 14 00:05:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50 Jan 14 00:05:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140 Jan 14 00:05:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290 Jan 14 00:05:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0 Jan 14 00:05:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0 Jan 14 00:05:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170 Jan 14 00:05:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220 Jan 14 00:05:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20 Jan 14 00:05:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a Jan 14 00:05:03 caro kernel: INFO: task xfs_db:87606 blocked for more than 120 seconds. Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:05:03 caro kernel: xfs_db D ffff9e747f89ac80 0 87606 87605 0x00000080 Jan 14 00:05:03 caro kernel: Call Trace: Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:05:03 caro kernel: [<ffffffffad62b621>] ? __switch_to+0x151/0x580 Jan 14 00:05:03 caro kernel: [<ffffffffad635c19>] ? sched_clock+0x9/0x10 Jan 14 00:05:03 caro kernel: [<ffffffffad6de6b5>] ? sched_clock_cpu+0x85/0xc0 Jan 14 00:05:03 caro kernel: [<ffffffffad6db200>] ? try_to_wake_up+0x190/0x390 Jan 14 00:05:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140 Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:05:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0 Jan 14 00:05:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90 Jan 14 00:05:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:05:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02c1e0d>] ? xfs_fs_sync_fs+0x2d/0x50 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffc02c1e0d>] xfs_fs_sync_fs+0x2d/0x50 [xfs] Jan 14 00:05:03 caro kernel: [<ffffffffad87fc70>] sync_filesystem+0x80/0xb0 Jan 14 00:05:03 caro kernel: [<ffffffffad88b304>] fsync_bdev+0x24/0x60 Jan 14 00:05:03 caro kernel: [<ffffffffad963948>] blkdev_ioctl+0x718/0xa20 Jan 14 00:05:03 caro kernel: [<ffffffffad7c07a7>] ? mempool_free_slab+0x17/0x20 Jan 14 00:05:03 caro kernel: [<ffffffffad88a8b1>] block_ioctl+0x41/0x50 Jan 14 00:05:03 caro kernel: [<ffffffffad85fb40>] do_vfs_ioctl+0x3a0/0x5a0 Jan 14 00:05:03 caro kernel: [<ffffffffad95476a>] ? blkdev_issue_flush+0xda/0x110 Jan 14 00:05:03 caro kernel: [<ffffffffad85fde1>] SyS_ioctl+0xa1/0xc0 Jan 14 00:05:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a Jan 14 00:07:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds. Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:07:03 caro kernel: kworker/14:0 D ffff9e847f11ac80 0 84673 2 0x00000080 Jan 14 00:07:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs] Jan 14 00:07:03 caro kernel: Call Trace: Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:07:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:07:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440 Jan 14 00:07:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0 Jan 14 00:07:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 14 00:07:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0 Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:07:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21 Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:07:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds. Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:07:03 caro kernel: kworker/14:2 D ffff9e8470fd9070 0 85328 2 0x00000080 Jan 14 00:07:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs] Jan 14 00:07:03 caro kernel: Call Trace: Jan 14 00:07:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280 Jan 14 00:07:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50 Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:07:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:07:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140 Jan 14 00:07:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20 Jan 14 00:07:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0 Jan 14 00:07:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90 Jan 14 00:07:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0 Jan 14 00:07:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580 Jan 14 00:07:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440 Jan 14 00:07:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0 Jan 14 00:07:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 Jan 14 00:07:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0 Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:07:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21 Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40 Jan 14 00:07:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds. Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 14 00:07:03 caro kernel: rsync D ffff9e825abe5230 0 86419 86399 0x00000080 Jan 14 00:07:03 caro kernel: Call Trace: Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70 Jan 14 00:07:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0 Jan 14 00:07:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104 Jan 14 00:07:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f Jan 14 00:07:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50 Jan 14 00:07:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs] Jan 14 00:07:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140 Jan 14 00:07:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290 Jan 14 00:07:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0 Jan 14 00:07:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0 Jan 14 00:07:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170 Jan 14 00:07:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220 Jan 14 00:07:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20 Jan 14 00:07:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a |
|
# uname -a Linux hostname 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [124679.668777] INFO: task kswapd0:106 blocked for more than 120 seconds. [124679.668841] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.668907] kswapd0 D ffff9e981f45ac80 0 106 2 0x00000000 [124679.668975] Call Trace: [124679.669047] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] [124679.669105] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.669152] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0 [124679.669233] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs] [124679.669321] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs] [124679.669404] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] [124679.669459] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130 [124679.669515] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90 [124679.669566] [<ffffffffa89801b8>] io_schedule+0x18/0x20 [124679.669638] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs] [124679.669695] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40 [124679.669790] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs] [124679.669872] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs] [124679.669952] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] [124679.670037] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] [124679.670120] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] [124679.670186] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0 [124679.670237] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340 [124679.670288] [<ffffffffa84403b7>] ? vmpressure+0x87/0x90 [124679.670338] [<ffffffffa83d53e8>] balance_pgdat+0x3a8/0x5e0 [124679.670389] [<ffffffffa83d5793>] kswapd+0x173/0x440 [124679.670435] [<ffffffffa82c72e0>] ? wake_up_atomic_t+0x30/0x30 [124679.670488] [<ffffffffa83d5620>] ? balance_pgdat+0x5e0/0x5e0 [124679.670539] [<ffffffffa82c61f1>] kthread+0xd1/0xe0 [124679.670584] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.670639] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21 [124679.670696] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.670774] INFO: task kworker/1:5:126 blocked for more than 120 seconds. [124679.670845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.670909] kworker/1:5 D ffff9e981f25ac80 0 126 2 0x00000000 [124679.671010] Workqueue: xfs-cil/md124 xlog_cil_push_work [xfs] [124679.671061] Call Trace: [124679.671088] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.671161] [<ffffffffc08811cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs] [124679.671225] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20 [124679.672983] [<ffffffffc08818b9>] xlog_write+0x1a9/0x750 [xfs] [124679.674779] [<ffffffffc087e807>] ? kmem_zone_alloc+0x97/0x130 [xfs] [124679.676576] [<ffffffffc08836e8>] xlog_cil_push+0x2a8/0x430 [xfs] [124679.678345] [<ffffffffc0883885>] xlog_cil_push_work+0x15/0x20 [xfs] [124679.680030] [<ffffffffa82be21f>] process_one_work+0x17f/0x440 [124679.681548] [<ffffffffa82bf488>] worker_thread+0x278/0x3c0 [124679.683005] [<ffffffffa82bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 [124679.684483] [<ffffffffa82c61f1>] kthread+0xd1/0xe0 [124679.685965] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.687414] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21 [124679.688847] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.690272] INFO: task xfsaild/md124:1947 blocked for more than 120 seconds. [124679.691611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.692952] xfsaild/md124 D ffff9e981f51ac80 0 1947 2 0x00000000 [124679.694312] Call Trace: [124679.695656] [<ffffffffa82ae4de>] ? try_to_del_timer_sync+0x5e/0x90 [124679.697030] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.698419] [<ffffffffc0882116>] _xfs_log_force+0x1c6/0x2a0 [xfs] [124679.699796] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20 [124679.701193] [<ffffffffc088e530>] ? xfsaild+0x180/0x760 [xfs] [124679.702507] [<ffffffffc088221c>] xfs_log_force+0x2c/0x70 [xfs] [124679.703810] [<ffffffffc088e3b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] [124679.705125] [<ffffffffc088e530>] xfsaild+0x180/0x760 [xfs] [124679.706431] [<ffffffffc088e3b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] [124679.707732] [<ffffffffa82c61f1>] kthread+0xd1/0xe0 [124679.709026] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.710331] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21 [124679.711652] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.712907] INFO: task kworker/1:0:16826 blocked for more than 120 seconds. [124679.714149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.715419] kworker/1:0 D ffff9e9818759070 0 16826 2 0x00000080 [124679.716739] Workqueue: xfs-sync/md124 xfs_log_worker [xfs] [124679.718034] Call Trace: [124679.719324] [<ffffffffa82e8278>] ? update_group_power+0x28/0x280 [124679.720645] [<ffffffffa8581685>] ? cpumask_next_and+0x35/0x50 [124679.721967] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.723222] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0 [124679.724464] [<ffffffffa82e150e>] ? account_entity_dequeue+0xae/0xd0 [124679.725709] [<ffffffffa82e4ecc>] ? dequeue_entity+0x11c/0x5c0 [124679.726945] [<ffffffffa82e64d1>] ? put_prev_entity+0x31/0x400 [124679.728167] [<ffffffffa82e2799>] ? pick_next_entity+0xa9/0x190 [124679.729370] [<ffffffffa8980ddd>] wait_for_completion+0xfd/0x140 [124679.730559] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20 [124679.731761] [<ffffffffa82be9aa>] flush_work+0x10a/0x1b0 [124679.732930] [<ffffffffa82bb6c0>] ? move_linked_works+0x90/0x90 [124679.734066] [<ffffffffc08840ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs] [124679.735168] [<ffffffffa82e4ecc>] ? dequeue_entity+0x11c/0x5c0 [124679.736282] [<ffffffffc0881fd6>] _xfs_log_force+0x86/0x2a0 [xfs] [124679.737381] [<ffffffffa822b59e>] ? __switch_to+0xce/0x580 [124679.738493] [<ffffffffc0882296>] ? xfs_log_worker+0x36/0x100 [xfs] [124679.739609] [<ffffffffc088221c>] xfs_log_force+0x2c/0x70 [xfs] [124679.740730] [<ffffffffc0882296>] xfs_log_worker+0x36/0x100 [xfs] [124679.741827] [<ffffffffa82be21f>] process_one_work+0x17f/0x440 [124679.742925] [<ffffffffa82bf336>] worker_thread+0x126/0x3c0 [124679.744020] [<ffffffffa82bf210>] ? manage_workers.isra.26+0x2a0/0x2a0 [124679.745072] [<ffffffffa82c61f1>] kthread+0xd1/0xe0 [124679.746106] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.747143] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21 [124679.748181] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40 [124679.749207] INFO: task ffmpeg:15545 blocked for more than 120 seconds. [124679.750233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.751262] ffmpeg D ffff9e96daf220e0 0 15545 2764 0x00000080 [124679.752301] Call Trace: [124679.753318] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.754349] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0 [124679.755354] [<ffffffffc087f0ab>] ? xlog_bdstrat+0x2b/0x60 [xfs] [124679.756347] [<ffffffffc0880ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs] [124679.757321] [<ffffffffa897fe37>] __down_common+0xaa/0x104 [124679.758311] [<ffffffffc085e210>] ? _xfs_buf_find+0x170/0x330 [xfs] [124679.759283] [<ffffffffa897feae>] __down+0x1d/0x1f [124679.760249] [<ffffffffa82cc191>] down+0x41/0x50 [124679.761219] [<ffffffffc085e00c>] xfs_buf_lock+0x3c/0xd0 [xfs] [124679.762189] [<ffffffffc085e210>] _xfs_buf_find+0x170/0x330 [xfs] [124679.763160] [<ffffffffc085e3fa>] xfs_buf_get_map+0x2a/0x250 [xfs] [124679.764132] [<ffffffffc088f511>] xfs_trans_get_buf_map+0x101/0x170 [xfs] [124679.765103] [<ffffffffc083c5a2>] xfs_da_get_buf+0xc2/0x100 [xfs] [124679.766028] [<ffffffffc0843edf>] xfs_dir3_data_init+0x5f/0x280 [xfs] [124679.766927] [<ffffffffc0842c38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs] [124679.767816] [<ffffffffc086f2b7>] ? xfs_setup_inode+0x87/0x120 [xfs] [124679.768690] [<ffffffffc084ab3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs] [124679.769573] [<ffffffffc087e467>] ? kmem_alloc+0x97/0x130 [xfs] [124679.770446] [<ffffffffc0840c88>] xfs_dir_createname+0x1a8/0x200 [xfs] [124679.771326] [<ffffffffc08725cc>] xfs_create+0x56c/0x6a0 [xfs] [124679.772203] [<ffffffffc086f533>] xfs_generic_create+0xd3/0x290 [xfs] [124679.773085] [<ffffffffc086f724>] xfs_vn_mknod+0x14/0x20 [xfs] [124679.773962] [<ffffffffc086f763>] xfs_vn_create+0x13/0x20 [xfs] [124679.774825] [<ffffffffa8457de3>] vfs_create+0xd3/0x140 [124679.775680] [<ffffffffa8459e7a>] do_last+0x10ca/0x1290 [124679.776540] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0 [124679.777370] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0 [124679.778183] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170 [124679.778992] [<ffffffffa8449704>] do_sys_open+0x124/0x220 [124679.779797] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146 [124679.780602] [<ffffffffa844981e>] SyS_open+0x1e/0x20 [124679.781405] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a [124679.782211] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146 [124679.783019] INFO: task ffmpeg:15554 blocked for more than 120 seconds. [124679.783837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.784665] ffmpeg D ffff9e981f2dac80 0 15554 2764 0x00000080 [124679.785515] Call Trace: [124679.786366] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] [124679.787228] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.788059] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0 [124679.788890] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs] [124679.789729] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs] [124679.790560] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] [124679.791380] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130 [124679.792193] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90 [124679.792990] [<ffffffffa89801b8>] io_schedule+0x18/0x20 [124679.793786] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs] [124679.794563] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40 [124679.795357] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs] [124679.796159] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs] [124679.796961] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] [124679.797766] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] [124679.798537] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] [124679.799289] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0 [124679.800038] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340 [124679.800789] [<ffffffffa8440351>] ? vmpressure+0x21/0x90 [124679.801532] [<ffffffffa83d497a>] do_try_to_free_pages+0x3ca/0x520 [124679.802284] [<ffffffffa83d4bcc>] try_to_free_pages+0xfc/0x180 [124679.803035] [<ffffffffa83c87f1>] __alloc_pages_nodemask+0x831/0xbe0 [124679.803789] [<ffffffffa841a045>] alloc_pages_vma+0xb5/0x200 [124679.804535] [<ffffffffa83f1984>] handle_pte_fault+0x984/0xe20 [124679.805285] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110 [124679.806038] [<ffffffffa83f3f3d>] handle_mm_fault+0x39d/0x9b0 [124679.806788] [<ffffffffa83e95a2>] __get_user_pages+0x1d2/0x7a0 [124679.807528] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110 [124679.808275] [<ffffffffa83ea022>] get_user_pages+0x52/0x60 [124679.809018] [<ffffffffc021b32b>] dl_get_user_pages+0xab/0xf0 [blackmagic] [124679.809754] [<ffffffffc02212da>] _ZN18IOMemoryDescriptor7prepareEv+0x3e/0x50 [blackmagic] [124679.810501] [<ffffffffc02556a0>] _ZN26DaisyCutterDMASupportClass35newVideoFrameDMATransactionNotGatedEP21BlackmagicDMAParamRecP30BlackmagicMainMemoryDescriptorP+0x31a/0x974 [blackmagic] [124679.812043] [<ffffffffc0238d9f>] ? _ZN32HardwareAbstractionDeckLinkClass13BitTestEndianEPVjj+0x11/0x18 [blackmagic] [124679.812869] [<ffffffffc021beca>] ? dl_gate_unlock+0x2a/0x30 [blackmagic] [124679.813698] [<ffffffffc0221bde>] ? _ZN24IOBufferMemoryDescriptor4freeEv+0x40/0x40 [blackmagic] [124679.814556] [<ffffffffc024d4d9>] _ZN22VideoInputSupportClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRec+0x1a1/0x1d2 [blackmagic] [124679.816303] [<ffffffffc025bff7>] _ZN15UserClientClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRecj+0x1b/0x24 [blackmagic] [124679.818159] [<ffffffffc021d6b4>] blackmagic_ioctl_private+0x1466/0x3ef8 [blackmagic] [124679.819126] [<ffffffffc0219fd9>] blackmagic_ioctl+0x49/0x70 [blackmagic] [124679.820058] [<ffffffffa845fbc0>] do_vfs_ioctl+0x3a0/0x5a0 [124679.820976] [<ffffffffa8306152>] ? ktime_get_ts64+0x52/0xf0 [124679.821884] [<ffffffffa845fe61>] SyS_ioctl+0xa1/0xc0 [124679.822777] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146 [124679.823662] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a [124679.824544] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146 [124679.825436] INFO: task ffmpeg:15702 blocked for more than 120 seconds. [124679.826323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.827246] ffmpeg D ffff9e98152d1070 0 15702 2978 0x00000080 [124679.828159] Call Trace: [124679.829060] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.829995] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0 [124679.830896] [<ffffffffc087f0ab>] ? xlog_bdstrat+0x2b/0x60 [xfs] [124679.831785] [<ffffffffc0880ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs] [124679.832654] [<ffffffffa897fe37>] __down_common+0xaa/0x104 [124679.833537] [<ffffffffc085e210>] ? _xfs_buf_find+0x170/0x330 [xfs] [124679.834411] [<ffffffffa897feae>] __down+0x1d/0x1f [124679.835280] [<ffffffffa82cc191>] down+0x41/0x50 [124679.836156] [<ffffffffc085e00c>] xfs_buf_lock+0x3c/0xd0 [xfs] [124679.837039] [<ffffffffc085e210>] _xfs_buf_find+0x170/0x330 [xfs] [124679.837927] [<ffffffffc085e3fa>] xfs_buf_get_map+0x2a/0x250 [xfs] [124679.838816] [<ffffffffc088f511>] xfs_trans_get_buf_map+0x101/0x170 [xfs] [124679.839696] [<ffffffffc083c5a2>] xfs_da_get_buf+0xc2/0x100 [xfs] [124679.840583] [<ffffffffc0843edf>] xfs_dir3_data_init+0x5f/0x280 [xfs] [124679.841473] [<ffffffffc0842c38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs] [124679.842363] [<ffffffffc086f2b7>] ? xfs_setup_inode+0x87/0x120 [xfs] [124679.843253] [<ffffffffc084ab3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs] [124679.844147] [<ffffffffc087e467>] ? kmem_alloc+0x97/0x130 [xfs] [124679.845035] [<ffffffffc0840c88>] xfs_dir_createname+0x1a8/0x200 [xfs] [124679.845930] [<ffffffffc08725cc>] xfs_create+0x56c/0x6a0 [xfs] [124679.846822] [<ffffffffc086f533>] xfs_generic_create+0xd3/0x290 [xfs] [124679.847711] [<ffffffffc086f724>] xfs_vn_mknod+0x14/0x20 [xfs] [124679.848602] [<ffffffffc086f763>] xfs_vn_create+0x13/0x20 [xfs] [124679.849479] [<ffffffffa8457de3>] vfs_create+0xd3/0x140 [124679.850353] [<ffffffffa8459e7a>] do_last+0x10ca/0x1290 [124679.851221] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0 [124679.852079] [<ffffffffa82e49b7>] ? update_curr+0x127/0x1e0 [124679.852920] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0 [124679.853733] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170 [124679.854516] [<ffffffffa8449704>] do_sys_open+0x124/0x220 [124679.855286] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146 [124679.856054] [<ffffffffa844981e>] SyS_open+0x1e/0x20 [124679.856801] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a [124679.857522] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146 [124679.858226] INFO: task ffmpeg:15712 blocked for more than 120 seconds. [124679.858913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.859594] ffmpeg D ffff9e981f21ac80 0 15712 2978 0x00000080 [124679.860290] Call Trace: [124679.860984] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] [124679.861681] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.862381] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0 [124679.863088] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs] [124679.863802] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs] [124679.864510] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs] [124679.865201] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130 [124679.865883] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90 [124679.866548] [<ffffffffa89801b8>] io_schedule+0x18/0x20 [124679.867221] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs] [124679.867888] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40 [124679.868564] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs] [124679.869246] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs] [124679.869931] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs] [124679.870610] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs] [124679.871289] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs] [124679.871956] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0 [124679.872621] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340 [124679.873288] [<ffffffffa8440351>] ? vmpressure+0x21/0x90 [124679.873953] [<ffffffffa83d497a>] do_try_to_free_pages+0x3ca/0x520 [124679.874617] [<ffffffffa83d4bcc>] try_to_free_pages+0xfc/0x180 [124679.875282] [<ffffffffa83c87f1>] __alloc_pages_nodemask+0x831/0xbe0 [124679.875950] [<ffffffffa841a045>] alloc_pages_vma+0xb5/0x200 [124679.876611] [<ffffffffa83f1984>] handle_pte_fault+0x984/0xe20 [124679.877278] [<ffffffffa83c8662>] ? __alloc_pages_nodemask+0x6a2/0xbe0 [124679.877947] [<ffffffffa83f3f3d>] handle_mm_fault+0x39d/0x9b0 [124679.878608] [<ffffffffa83e95a2>] __get_user_pages+0x1d2/0x7a0 [124679.879272] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110 [124679.879936] [<ffffffffa83ea022>] get_user_pages+0x52/0x60 [124679.880604] [<ffffffffc021b32b>] dl_get_user_pages+0xab/0xf0 [blackmagic] [124679.881286] [<ffffffffc02212da>] _ZN18IOMemoryDescriptor7prepareEv+0x3e/0x50 [blackmagic] [124679.881986] [<ffffffffc02556a0>] _ZN26DaisyCutterDMASupportClass35newVideoFrameDMATransactionNotGatedEP21BlackmagicDMAParamRecP30BlackmagicMainMemoryDescriptorP+0x31a/0x974 [blackmagic] [124679.883423] [<ffffffffc0238d9f>] ? _ZN32HardwareAbstractionDeckLinkClass13BitTestEndianEPVjj+0x11/0x18 [blackmagic] [124679.884193] [<ffffffffc021beca>] ? dl_gate_unlock+0x2a/0x30 [blackmagic] [124679.884969] [<ffffffffc0221bde>] ? _ZN24IOBufferMemoryDescriptor4freeEv+0x40/0x40 [blackmagic] [124679.885766] [<ffffffffc024d4d9>] _ZN22VideoInputSupportClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRec+0x1a1/0x1d2 [blackmagic] [124679.887384] [<ffffffffc025bff7>] _ZN15UserClientClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRecj+0x1b/0x24 [blackmagic] [124679.889103] [<ffffffffc021d6b4>] blackmagic_ioctl_private+0x1466/0x3ef8 [blackmagic] [124679.889997] [<ffffffffc0219fd9>] blackmagic_ioctl+0x49/0x70 [blackmagic] [124679.890883] [<ffffffffa845fbc0>] do_vfs_ioctl+0x3a0/0x5a0 [124679.891768] [<ffffffffa8988678>] ? __do_page_fault+0x238/0x500 [124679.892639] [<ffffffffa845fe61>] SyS_ioctl+0xa1/0xc0 [124679.893501] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146 [124679.894361] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a [124679.895213] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146 [124679.896077] INFO: task find:21716 blocked for more than 120 seconds. [124679.896934] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.897805] find D ffff9e981f4dac80 0 21716 21715 0x00000080 [124679.898679] Call Trace: [124679.899545] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.900421] [<ffffffffa8982415>] rwsem_down_read_failed+0x105/0x1c0 [124679.901315] [<ffffffffc0870de0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs] [124679.902203] [<ffffffffa8592d98>] call_rwsem_down_read_failed+0x18/0x30 [124679.903090] [<ffffffffa897fd30>] down_read+0x20/0x40 [124679.903985] [<ffffffffc0870d69>] xfs_ilock+0xd9/0x120 [xfs] [124679.904877] [<ffffffffc0870de0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs] [124679.905772] [<ffffffffc08628c0>] xfs_dir_open+0x40/0x80 [xfs] [124679.906649] [<ffffffffa8448136>] do_dentry_open+0x216/0x2c0 [124679.907535] [<ffffffffa85042b2>] ? security_inode_permission+0x22/0x30 [124679.908438] [<ffffffffc0862880>] ? xfs_file_llseek+0xa0/0xa0 [xfs] [124679.909330] [<ffffffffa844827a>] vfs_open+0x5a/0xb0 [124679.910216] [<ffffffffa84568c3>] ? may_open+0xa3/0x120 [124679.911101] [<ffffffffa8458fa6>] do_last+0x1f6/0x1290 [124679.911987] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0 [124679.912870] [<ffffffffa845d682>] ? user_path_at_empty+0x72/0xc0 [124679.913754] [<ffffffffa84473fa>] ? __check_object_size+0x1ca/0x250 [124679.914634] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0 [124679.915516] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170 [124679.916394] [<ffffffffa8449704>] do_sys_open+0x124/0x220 [124679.917267] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146 [124679.918145] [<ffffffffa8449834>] SyS_openat+0x14/0x20 [124679.919020] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a [124679.919898] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146 [124679.920778] INFO: task find:21734 blocked for more than 120 seconds. [124679.921652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [124679.922539] find D ffff9e981f4dac80 0 21734 21733 0x00000080 [124679.923424] Call Trace: [124679.924271] [<ffffffffa8980a29>] schedule+0x29/0x70 [124679.925108] [<ffffffffa8982415>] rwsem_down_read_failed+0x105/0x1c0 [124679.925947] [<ffffffffc0870de0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs] [124679.926772] [<ffffffffa8592d98>] call_rwsem_down_read_failed+0x18/0x30 [124679.927576] [<ffffffffa897fd30>] down_read+0x20/0x40 [124679.928366] [<ffffffffc0870d69>] xfs_ilock+0xd9/0x120 [xfs] [124679.929132] [<ffffffffc0870de0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs] [124679.929878] [<ffffffffc08628c0>] xfs_dir_open+0x40/0x80 [xfs] [124679.930591] [<ffffffffa8448136>] do_dentry_open+0x216/0x2c0 [124679.931308] [<ffffffffa85042b2>] ? security_inode_permission+0x22/0x30 [124679.932037] [<ffffffffc0862880>] ? xfs_file_llseek+0xa0/0xa0 [xfs] [124679.932756] [<ffffffffa844827a>] vfs_open+0x5a/0xb0 [124679.933469] [<ffffffffa84568c3>] ? may_open+0xa3/0x120 [124679.934181] [<ffffffffa8458fa6>] do_last+0x1f6/0x1290 [124679.934890] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0 [124679.935592] [<ffffffffa845d682>] ? user_path_at_empty+0x72/0xc0 [124679.936293] [<ffffffffa84473fa>] ? __check_object_size+0x1ca/0x250 [124679.936986] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0 [124679.937661] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170 [124679.938329] [<ffffffffa8449704>] do_sys_open+0x124/0x220 [124679.938995] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146 [124679.939661] [<ffffffffa8449834>] SyS_openat+0x14/0x20 [124679.940324] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a [124679.940989] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146 |
|
Reproducibility is from several hours till few weeks. under the same load every time. | |
We are affected too, the latest CentOS/RedHat kernel (3.10.0-1062.....el7.x86_64), happened on multiple mission-critical servers with XFS, kernel updates don't help, same as 3 other reporters above. We had to install (and currently use) the debug(!) kernel as a W/A as it prevents all XFS partitions from being marked read-only. But we are genuinely surprised to see such a critical issue remains unnoticed, as XFS is now default since CentOS7/RHEL7 (and Centos7 is maintained/supported until 30 June 2024). Has it been escalated to maybe RedHat? Is there a patch? |
|
Is anyone still seeing the issue on 3.10.0-1062.18.1.el7.x86_64 ? We had a system that was triggering the issue fairly reliably during RAID resyncs but it hasn't triggered it since we updated to 3.10.0-1062.18.1.el7.x86_64. | |
started my "test" on 3.10.0-1062.18.1.el7.x86_64 just now. previously it was up to a couple of week to fail. will answer at least in a month or earlier. |
|
Any update on those machines updated to 3.10.0-1062.18.1.el7.x86_64 ? Have you seen again the hangs? We are hitting the same on 3.10.0-1062.1.2.el7.x86_64 at least, rebooting being our only solution so far. |
|
Since CentOS 7.8 is out for almost 3 months already and the current kernel is 3.10.0-1127.10.1.el7.x86_64, I suggest to update and retry. If you still encounter the same issue, please open a bug at bugzilla.redhat.com. |
|
We have not seen the issue return on any of the machines we updated. The first machine we patched triggered the issue weekly or bi-weekly before updating and has now been up 50 days without triggering the issue (still 3.10.0-1062.18.1.el7.x86_64). | |
After upgrade no new incidents, but as long as no root cause was identified and fixed I‘m sceptic. Might be just a side effect of unrelated changes (leading to different timings etc.) and that the bug could reoccur in future releases. | |
@maf01: skepticism is good. you are welcome to read the changelog for the kernel package and the release notes so as to verify if any xfs-related changes introduced by RH in the kernel released for 7.8 include the fix for the issue you were facing. @nomuthetart: if you must keep using older kernels, then you are strongly advised to purchase a RHEL EUS subscription. Unlike RedHat, CentOS does not have the resources and cannot provide support for anything but the most recent version of the packages it ships. And this is especially true for the upgrades related to the release of new minor CentOS versions. See also https://wiki.centos.org/FAQ/General#How_does_CentOS_versioning_work.3F , #2.1 and #3.1.1 |
|
@ManuelWolfshant - this bug was a reintroduced version of https://bugs.centos.org/view.php?id=13843 and is sporadic making it difficult to know for sure if it is fixed or not. In our case we kept back hosts that hadn't been patched to 3.10.0-1062.4.1.el7.x86_64 and put those that had been updated onto the debug kernel while waiting for a patch. We only had a handful of hosts that could reliably trigger the bug so we put our most reliable one in a non-critical role and kept it fully patched to test for a fix. Once it made it to a couple weeks uptime on 3.10.0-1062.18.1.el7.x86_64 I posted here and once it hit 30 days without crashing we resumed regular patching. At this point it is only running as a long term test. I didn't see anything XFS related in https://www.redhat.com/archives/rhsa-announce/2020-March/msg00061.html but there were MD changes; since we only saw it on XFS w/ MD setups (our default) we assumed that is what fixed it. |
|
Hello @ManuelWolfshant, @casp or @nomuthetart, has this issue been mitigated for you by using 3.10.0-1062.18.1.el7.x86_64 ? Any further conclusion on this bug? Thanks a lot | |
Hi @xtomi ; after we resumed regular patching the issue hasn't reoccurred. We did eventually patch our long term test host so we aren't running anymore hosts with 3.10.0-1062.18.1.el7.x86_64 but we haven't see the XFS issue on any newer kernels either. In our opinion it is fully resolved and all of our hosts are back on regular patching schedules again. | |
Thanks a lot @nomuthetart for provided info. | |
I don't know what kernel upgrade fixed this, but i haven't seen this issue anymore ever since. | |
3.10.0-1062.18.1.el7.x86_64 uptime 291 days, all ok. ps: more fresh kernels not tested. |
|
@slayerduck @caspy thank both you guys for confirmation. it seems to be very promising :-) | |
Date Modified | Username | Field | Change |
---|---|---|---|
2019-11-09 08:44 | bruceleeeee | New Issue | |
2019-11-09 08:44 | bruceleeeee | Tag Attached: 3.10.0-1062.1.1.el7.x86_64 | |
2019-11-09 08:44 | bruceleeeee | Tag Attached: 7.7 | |
2019-11-09 08:44 | bruceleeeee | Tag Attached: centos 7 | |
2019-11-09 08:44 | bruceleeeee | Tag Attached: file system | |
2020-01-10 13:50 | maf01 | Note Added: 0035998 | |
2020-01-14 00:05 | slayerduck | File Added: messages.txt | |
2020-01-14 00:05 | slayerduck | Note Added: 0036017 | |
2020-02-19 07:30 | caspy | Note Added: 0036316 | |
2020-02-19 07:35 | caspy | Note Added: 0036317 | |
2020-03-19 21:20 | .edu | Note Added: 0036538 | |
2020-03-20 11:38 | kudeyda | File Added: log | |
2020-04-20 18:26 | nomuthetart | Note Added: 0036733 | |
2020-04-20 21:47 | caspy | Note Added: 0036737 | |
2020-06-05 07:28 | dabadaba | Note Added: 0037038 | |
2020-06-05 09:23 | ManuelWolfshant | Note Added: 0037039 | |
2020-06-05 09:25 | ManuelWolfshant | Note Edited: 0037039 | |
2020-06-05 15:32 | nomuthetart | Note Added: 0037043 | |
2020-06-05 15:43 | maf01 | Note Added: 0037044 | |
2020-06-06 00:14 | ManuelWolfshant | Note Added: 0037045 | |
2020-06-06 00:35 | nomuthetart | Note Added: 0037046 | |
2021-04-23 17:03 | xtomi | Note Added: 0038417 | |
2021-04-23 18:28 | nomuthetart | Note Added: 0038418 | |
2021-04-23 21:46 | xtomi | Note Added: 0038419 | |
2021-04-24 05:56 | slayerduck | Note Added: 0038420 | |
2021-04-24 09:22 | caspy | Note Added: 0038421 | |
2021-04-24 09:35 | xtomi | Note Added: 0038422 |