View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0016711CentOS-7kernelpublic2019-11-09 08:442021-04-24 09:35
Reporterbruceleeeee Assigned To 
PriorityhighSeveritycrashReproducibilityrandom
Status newResolutionopen 
Platformx86_64OSCentOS Linux 7 (Core)OS Version7.7
Product Version7.7-1908 
Summary0016711: xfsaild blocks after certain time
Descriptionproblem similar to https://bugs.centos.org/view.php?id=13843

kernel version: 3.10.0-1062.4.1.el7.x86_64

started to have problem where xfsaild becomes blocked after a period of time and processes wanting to access the file system begin to grind to a halt. Most often hits chatty IO applications such as auditd first. A reboot solves the problem until the next time. The file system is in a mirrored volume group and neither the volume group nor individual disk reports any problems.
Additional InformationNov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffffc02661ed>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff993bcf1f>] ? get_target_pstate_use_performance+0x8f/0xc0
Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffffc02668d9>] xlog_write+0x1a9/0x750 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc0263827>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc0268708>] xlog_cil_push+0x2a8/0x430 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc02688a5>] xlog_cil_push_work+0x15/0x20 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd1df>] process_one_work+0x17f/0x440
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe448>] worker_thread+0x278/0x3c0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: INFO: task kworker/u32:2:23070 blocked for more than 120 seconds.
Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:23:28 server1 kernel: kworker/u32:2 D ffff95e9fadea0e0 0 23070 2 0x00000080
Nov 8 09:23:28 server1 kernel: Workqueue: writeback bdi_writeback_workfn (flush-9:125)
Nov 8 09:23:28 server1 kernel: Call Trace:
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff995804f5>] rwsem_down_read_failed+0x105/0x1c0
Nov 8 09:23:28 server1 kernel: [<ffffffffc023c047>] ? xfs_map_blocks+0x87/0x220 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff991913f8>] call_rwsem_down_read_failed+0x18/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc90>] down_read+0x20/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffffc0255d69>] xfs_ilock+0xd9/0x120 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc023c047>] xfs_map_blocks+0x87/0x220 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc023d1a4>] xfs_do_writepage+0x174/0x550 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fc82ac>] write_cache_pages+0x21c/0x470
Nov 8 09:23:28 server1 kernel: [<ffffffffc023d030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff9914f710>] ? submit_bio+0x70/0x150
Nov 8 09:23:28 server1 kernel: [<ffffffffc023cffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fc92f1>] do_writepages+0x21/0x50
Nov 8 09:23:28 server1 kernel: [<ffffffff99077f70>] __writeback_single_inode+0x40/0x260
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec6265>] ? wake_up_bit+0x25/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff99078b04>] writeback_sb_inodes+0x1c4/0x430
Nov 8 09:23:28 server1 kernel: [<ffffffff99078e0f>] __writeback_inodes_wb+0x9f/0xd0
Nov 8 09:23:28 server1 kernel: [<ffffffff990792f3>] wb_writeback+0x263/0x2f0
Nov 8 09:23:28 server1 kernel: [<ffffffff9906519c>] ? get_nr_inodes+0x4c/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff99079eeb>] bdi_writeback_workfn+0x2cb/0x460
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd1df>] process_one_work+0x17f/0x440
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe2f6>] worker_thread+0x126/0x3c0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: INFO: task dd:27364 blocked for more than 120 seconds.
Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:23:28 server1 kernel: dd D ffff95e5cc4820e0 0 27364 1 0x00000080
Nov 8 09:23:28 server1 kernel: Call Trace:
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff9957c491>] schedule_timeout+0x221/0x2d0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed6619>] ? ttwu_do_wakeup+0x19/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed674f>] ? ttwu_do_activate+0x6f/0x80
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed9ed0>] ? try_to_wake_up+0x190/0x390
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eebd>] wait_for_completion+0xfd/0x140
Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd96a>] flush_work+0x10a/0x1b0
Nov 8 09:23:28 server1 kernel: [<ffffffff98eba680>] ? move_linked_works+0x90/0x90
Nov 8 09:23:28 server1 kernel: [<ffffffffc02690da>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc02674a4>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fbb22f>] ? filemap_fdatawait_range+0x1f/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc82>] ? down_read+0x12/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffffc0247a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e2a7>] do_fsync+0x67/0xb0
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e5b3>] SyS_fdatasync+0x13/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a
Nov 8 09:23:28 server1 kernel: INFO: task exim:27365 blocked for more than 120 seconds.
Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:23:28 server1 kernel: exim D ffff95e95a2e5230 0 27365 27362 0x00000080
Nov 8 09:23:28 server1 kernel: Call Trace:
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed56de>] ? resched_curr+0xae/0xc0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ee49a6>] ? check_preempt_wakeup+0x166/0x250
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff9957c491>] schedule_timeout+0x221/0x2d0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed6619>] ? ttwu_do_wakeup+0x19/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed674f>] ? ttwu_do_activate+0x6f/0x80
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed9ed0>] ? try_to_wake_up+0x190/0x390
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eebd>] wait_for_completion+0xfd/0x140
Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd96a>] flush_work+0x10a/0x1b0
Nov 8 09:23:28 server1 kernel: [<ffffffff98eba680>] ? move_linked_works+0x90/0x90
Nov 8 09:23:28 server1 kernel: [<ffffffffc02690da>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc02674a4>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fbb22f>] ? filemap_fdatawait_range+0x1f/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc82>] ? down_read+0x12/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffffc0247a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e2a7>] do_fsync+0x67/0xb0
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e590>] SyS_fsync+0x10/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a

Nov 8 09:25:28 server1 kernel: INFO: task xfsaild/md125:820 blocked for more than 120 seconds.
Nov 8 09:25:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:25:28 server1 kernel: xfsaild/md125 D ffff95e9fbf69070 0 820 2 0x00000000
Nov 8 09:25:28 server1 kernel: Call Trace:
Nov 8 09:25:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:25:28 server1 kernel: [<ffffffffc0267136>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:25:28 server1 kernel: [<ffffffffc0273550>] ? xfsaild+0x180/0x760 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc026723c>] xfs_log_force+0x2c/0x70 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc02733d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc0273550>] xfsaild+0x180/0x760 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc02733d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0
Nov 8 09:25:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:25:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21
Nov 8 09:25:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:25:28 server1 kernel: INFO: task auditd:1348 blocked for more than 120 seconds.
Nov 8 09:25:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:25:28 server1 kernel: auditd D ffffffff9957e465 0 1348 1 0x00000000
Nov 8 09:25:28 server1 kernel: Call Trace:
Nov 8 09:25:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:25:28 server1 kernel: [<ffffffff99580245>] rwsem_down_write_failed+0x215/0x3c0
Nov 8 09:25:28 server1 kernel: [<ffffffffc0263ee4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc025307d>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff99191427>] call_rwsem_down_write_failed+0x17/0x30
Nov 8 09:25:28 server1 kernel: [<ffffffff9957dcdd>] down_write+0x2d/0x3d
Nov 8 09:25:28 server1 kernel: [<ffffffffc0255d51>] xfs_ilock+0xc1/0x120 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc025307d>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff990663a8>] update_time+0x28/0xd0
Nov 8 09:25:28 server1 kernel: [<ffffffff98ed2723>] ? __wake_up+0x13/0x20
Nov 8 09:25:28 server1 kernel: [<ffffffff990664f0>] file_update_time+0xa0/0xf0
Nov 8 09:25:28 server1 kernel: [<ffffffffc0248adf>] xfs_file_aio_write_checks+0x16f/0x1c0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc024935a>] xfs_file_buffered_aio_write+0xca/0x2c0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc02496dd>] xfs_file_aio_write+0x18d/0x1b0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff99048433>] do_sync_write+0x93/0xe0
Nov 8 09:25:28 server1 kernel: [<ffffffff99048f20>] vfs_write+0xc0/0x1f0
Nov 8 09:25:28 server1 kernel: [<ffffffff99049d3f>] SyS_write+0x7f/0xf0
Nov 8 09:25:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a
Tags3.10.0-1062.1.1.el7.x86_64, 7.7, centos 7, file system
abrt_hash
URL

Activities

maf01

maf01

2020-01-10 13:50

reporter   ~0035998

The same here:

Centos7 3.10.0-1062.9.1.el7.x86_64

<code>
Jan 10 00:12:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:12:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:12:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:12:55 x3421 kernel: Call Trace:
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:12:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:12:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:14:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:14:55 x3421 kernel: Call Trace:
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: INFO: task kworker/0:2:18279 blocked for more than 120 seconds.
Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:14:55 x3421 kernel: kworker/0:2 D ffff9b4a1511e2a0 0 18279 2 0x00000080
Jan 10 00:14:55 x3421 kernel: Workqueue: xfs-sync/md125 xfs_log_worker [xfs]
Jan 10 00:14:55 x3421 kernel: Call Trace:
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8e83b4>] ? update_group_power+0x174/0x280
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cb81635>] ? cpumask_next_and+0x35/0x50
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8aecbd>] ? add_timer_on+0x8d/0xf0
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dcfd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf336>] worker_thread+0x126/0x3c0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: INFO: task kworker/0:3:7799 blocked for more than 120 seconds.
Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:14:55 x3421 kernel: kworker/0:3 D ffff9b4dfec1ac80 0 7799 2 0x00000080
Jan 10 00:14:55 x3421 kernel: Workqueue: xfs-cil/md125 xlog_cil_push_work [xfs]
Jan 10 00:14:55 x3421 kernel: Call Trace:
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dc1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dc8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06d9807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06de6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06de885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf488>] worker_thread+0x278/0x3c0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task grafana-server:27514 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: grafana-server D ffff9b4dfed1ac80 0 27514 4384 0x00000080
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d77a9>] ? ttwu_do_wakeup+0x19/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d78df>] ? ttwu_do_activate+0x6f/0x80
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db200>] ? try_to_wake_up+0x190/0x390
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd484>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c9bcaff>] ? filemap_fdatawait_range+0x1f/0x30
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7fd02>] ? down_read+0x12/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06bda3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fb77>] do_fsync+0x67/0xb0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fe60>] SyS_fsync+0x10/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dede>] system_call_fastpath+0x25/0x2a
Jan 10 00:16:55 x3421 kernel: INFO: task kworker/0:2:18279 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: kworker/0:2 D ffff9b4a1511e2a0 0 18279 2 0x00000080
Jan 10 00:16:55 x3421 kernel: Workqueue: xfs-sync/md125 xfs_log_worker [xfs]
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8e83b4>] ? update_group_power+0x174/0x280
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cb81635>] ? cpumask_next_and+0x35/0x50
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8aecbd>] ? add_timer_on+0x8d/0xf0
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dcfd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf336>] worker_thread+0x126/0x3c0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task kworker/0:3:7799 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: kworker/0:3 D ffff9b4dfec1ac80 0 7799 2 0x00000080
Jan 10 00:16:55 x3421 kernel: Workqueue: xfs-cil/md125 xlog_cil_push_work [xfs]
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dc1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dc8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06d9807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06de6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06de885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf488>] worker_thread+0x278/0x3c0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task redis-server:14049 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: redis-server D ffff9b4dfec1ac80 0 14049 4378 0x00000080
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c82b621>] ? __switch_to+0x151/0x580
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d78df>] ? ttwu_do_activate+0x6f/0x80
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd484>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c9bcaff>] ? filemap_fdatawait_range+0x1f/0x30
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7fd02>] ? down_read+0x12/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06bda3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fb77>] do_fsync+0x67/0xb0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fe60>] SyS_fsync+0x10/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dede>] system_call_fastpath+0x25/0x2a
Jan 10 00:18:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:18:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:18:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:18:55 x3421 kernel: Call Trace:
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:18:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:18:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
</code>
slayerduck

slayerduck

2020-01-14 00:05

reporter   ~0036017

Centos7 3.10.0-1062.9.1.el7.x86_64 and 3.10.0-1062.4.3.el7.x86_64 kernels here. Several servers with different storage configuration are displaying the issues. All with same motherboard Supermicro X9DRW-iF
messages.txt (21,918 bytes)    
Jan 14 00:03:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds.
Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:03:03 caro kernel: kworker/14:0    D ffff9e847f11ac80     0 84673      2 0x00000080
Jan 14 00:03:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs]
Jan 14 00:03:03 caro kernel: Call Trace:
Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:03:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:03:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds.
Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:03:03 caro kernel: kworker/14:2    D ffff9e8470fd9070     0 85328      2 0x00000080
Jan 14 00:03:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs]
Jan 14 00:03:03 caro kernel: Call Trace:
Jan 14 00:03:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280
Jan 14 00:03:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50
Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:03:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:03:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:03:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:03:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:03:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:03:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0
Jan 14 00:03:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580
Jan 14 00:03:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds.
Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:03:03 caro kernel: rsync           D ffff9e825abe5230     0 86419  86399 0x00000080
Jan 14 00:03:03 caro kernel: Call Trace:
Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:03:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:03:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104
Jan 14 00:03:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f
Jan 14 00:03:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50
Jan 14 00:03:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140
Jan 14 00:03:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290
Jan 14 00:03:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0
Jan 14 00:03:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0
Jan 14 00:03:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170
Jan 14 00:03:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220
Jan 14 00:03:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20
Jan 14 00:03:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
Jan 14 00:05:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: kworker/14:0    D ffff9e847f11ac80     0 84673      2 0x00000080
Jan 14 00:05:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs]
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: kworker/14:2    D ffff9e8470fd9070     0 85328      2 0x00000080
Jan 14 00:05:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs]
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280
Jan 14 00:05:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:05:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:05:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0
Jan 14 00:05:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: rsync           D ffff9e825abe5230     0 86419  86399 0x00000080
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:05:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104
Jan 14 00:05:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f
Jan 14 00:05:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50
Jan 14 00:05:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140
Jan 14 00:05:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290
Jan 14 00:05:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0
Jan 14 00:05:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0
Jan 14 00:05:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170
Jan 14 00:05:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220
Jan 14 00:05:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
Jan 14 00:05:03 caro kernel: INFO: task xfs_db:87606 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: xfs_db          D ffff9e747f89ac80     0 87606  87605 0x00000080
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:05:03 caro kernel: [<ffffffffad62b621>] ? __switch_to+0x151/0x580
Jan 14 00:05:03 caro kernel: [<ffffffffad635c19>] ? sched_clock+0x9/0x10
Jan 14 00:05:03 caro kernel: [<ffffffffad6de6b5>] ? sched_clock_cpu+0x85/0xc0
Jan 14 00:05:03 caro kernel: [<ffffffffad6db200>] ? try_to_wake_up+0x190/0x390
Jan 14 00:05:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:05:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c1e0d>] ? xfs_fs_sync_fs+0x2d/0x50 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c1e0d>] xfs_fs_sync_fs+0x2d/0x50 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad87fc70>] sync_filesystem+0x80/0xb0
Jan 14 00:05:03 caro kernel: [<ffffffffad88b304>] fsync_bdev+0x24/0x60
Jan 14 00:05:03 caro kernel: [<ffffffffad963948>] blkdev_ioctl+0x718/0xa20
Jan 14 00:05:03 caro kernel: [<ffffffffad7c07a7>] ? mempool_free_slab+0x17/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffad88a8b1>] block_ioctl+0x41/0x50
Jan 14 00:05:03 caro kernel: [<ffffffffad85fb40>] do_vfs_ioctl+0x3a0/0x5a0
Jan 14 00:05:03 caro kernel: [<ffffffffad95476a>] ? blkdev_issue_flush+0xda/0x110
Jan 14 00:05:03 caro kernel: [<ffffffffad85fde1>] SyS_ioctl+0xa1/0xc0
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
Jan 14 00:07:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds.
Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:07:03 caro kernel: kworker/14:0    D ffff9e847f11ac80     0 84673      2 0x00000080
Jan 14 00:07:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs]
Jan 14 00:07:03 caro kernel: Call Trace:
Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:07:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:07:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds.
Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:07:03 caro kernel: kworker/14:2    D ffff9e8470fd9070     0 85328      2 0x00000080
Jan 14 00:07:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs]
Jan 14 00:07:03 caro kernel: Call Trace:
Jan 14 00:07:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280
Jan 14 00:07:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50
Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:07:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:07:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:07:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:07:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:07:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:07:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0
Jan 14 00:07:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580
Jan 14 00:07:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds.
Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:07:03 caro kernel: rsync           D ffff9e825abe5230     0 86419  86399 0x00000080
Jan 14 00:07:03 caro kernel: Call Trace:
Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:07:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:07:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104
Jan 14 00:07:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f
Jan 14 00:07:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50
Jan 14 00:07:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140
Jan 14 00:07:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290
Jan 14 00:07:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0
Jan 14 00:07:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0
Jan 14 00:07:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170
Jan 14 00:07:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220
Jan 14 00:07:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20
Jan 14 00:07:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
messages.txt (21,918 bytes)   
caspy

caspy

2020-02-19 07:30

reporter   ~0036316

# uname -a
Linux hostname 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


[124679.668777] INFO: task kswapd0:106 blocked for more than 120 seconds.
[124679.668841] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.668907] kswapd0 D ffff9e981f45ac80 0 106 2 0x00000000
[124679.668975] Call Trace:
[124679.669047] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.669105] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.669152] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.669233] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs]
[124679.669321] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs]
[124679.669404] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.669459] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130
[124679.669515] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90
[124679.669566] [<ffffffffa89801b8>] io_schedule+0x18/0x20
[124679.669638] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs]
[124679.669695] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40
[124679.669790] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.669872] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs]
[124679.669952] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[124679.670037] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[124679.670120] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[124679.670186] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0
[124679.670237] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340
[124679.670288] [<ffffffffa84403b7>] ? vmpressure+0x87/0x90
[124679.670338] [<ffffffffa83d53e8>] balance_pgdat+0x3a8/0x5e0
[124679.670389] [<ffffffffa83d5793>] kswapd+0x173/0x440
[124679.670435] [<ffffffffa82c72e0>] ? wake_up_atomic_t+0x30/0x30
[124679.670488] [<ffffffffa83d5620>] ? balance_pgdat+0x5e0/0x5e0
[124679.670539] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.670584] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.670639] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.670696] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.670774] INFO: task kworker/1:5:126 blocked for more than 120 seconds.
[124679.670845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.670909] kworker/1:5 D ffff9e981f25ac80 0 126 2 0x00000000
[124679.671010] Workqueue: xfs-cil/md124 xlog_cil_push_work [xfs]
[124679.671061] Call Trace:
[124679.671088] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.671161] [<ffffffffc08811cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
[124679.671225] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20
[124679.672983] [<ffffffffc08818b9>] xlog_write+0x1a9/0x750 [xfs]
[124679.674779] [<ffffffffc087e807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[124679.676576] [<ffffffffc08836e8>] xlog_cil_push+0x2a8/0x430 [xfs]
[124679.678345] [<ffffffffc0883885>] xlog_cil_push_work+0x15/0x20 [xfs]
[124679.680030] [<ffffffffa82be21f>] process_one_work+0x17f/0x440
[124679.681548] [<ffffffffa82bf488>] worker_thread+0x278/0x3c0
[124679.683005] [<ffffffffa82bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[124679.684483] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.685965] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.687414] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.688847] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.690272] INFO: task xfsaild/md124:1947 blocked for more than 120 seconds.
[124679.691611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.692952] xfsaild/md124 D ffff9e981f51ac80 0 1947 2 0x00000000
[124679.694312] Call Trace:
[124679.695656] [<ffffffffa82ae4de>] ? try_to_del_timer_sync+0x5e/0x90
[124679.697030] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.698419] [<ffffffffc0882116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
[124679.699796] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20
[124679.701193] [<ffffffffc088e530>] ? xfsaild+0x180/0x760 [xfs]
[124679.702507] [<ffffffffc088221c>] xfs_log_force+0x2c/0x70 [xfs]
[124679.703810] [<ffffffffc088e3b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[124679.705125] [<ffffffffc088e530>] xfsaild+0x180/0x760 [xfs]
[124679.706431] [<ffffffffc088e3b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[124679.707732] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.709026] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.710331] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.711652] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.712907] INFO: task kworker/1:0:16826 blocked for more than 120 seconds.
[124679.714149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.715419] kworker/1:0 D ffff9e9818759070 0 16826 2 0x00000080
[124679.716739] Workqueue: xfs-sync/md124 xfs_log_worker [xfs]
[124679.718034] Call Trace:
[124679.719324] [<ffffffffa82e8278>] ? update_group_power+0x28/0x280
[124679.720645] [<ffffffffa8581685>] ? cpumask_next_and+0x35/0x50
[124679.721967] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.723222] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.724464] [<ffffffffa82e150e>] ? account_entity_dequeue+0xae/0xd0
[124679.725709] [<ffffffffa82e4ecc>] ? dequeue_entity+0x11c/0x5c0
[124679.726945] [<ffffffffa82e64d1>] ? put_prev_entity+0x31/0x400
[124679.728167] [<ffffffffa82e2799>] ? pick_next_entity+0xa9/0x190
[124679.729370] [<ffffffffa8980ddd>] wait_for_completion+0xfd/0x140
[124679.730559] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20
[124679.731761] [<ffffffffa82be9aa>] flush_work+0x10a/0x1b0
[124679.732930] [<ffffffffa82bb6c0>] ? move_linked_works+0x90/0x90
[124679.734066] [<ffffffffc08840ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[124679.735168] [<ffffffffa82e4ecc>] ? dequeue_entity+0x11c/0x5c0
[124679.736282] [<ffffffffc0881fd6>] _xfs_log_force+0x86/0x2a0 [xfs]
[124679.737381] [<ffffffffa822b59e>] ? __switch_to+0xce/0x580
[124679.738493] [<ffffffffc0882296>] ? xfs_log_worker+0x36/0x100 [xfs]
[124679.739609] [<ffffffffc088221c>] xfs_log_force+0x2c/0x70 [xfs]
[124679.740730] [<ffffffffc0882296>] xfs_log_worker+0x36/0x100 [xfs]
[124679.741827] [<ffffffffa82be21f>] process_one_work+0x17f/0x440
[124679.742925] [<ffffffffa82bf336>] worker_thread+0x126/0x3c0
[124679.744020] [<ffffffffa82bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[124679.745072] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.746106] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.747143] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.748181] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.749207] INFO: task ffmpeg:15545 blocked for more than 120 seconds.
[124679.750233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.751262] ffmpeg D ffff9e96daf220e0 0 15545 2764 0x00000080
[124679.752301] Call Trace:
[124679.753318] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.754349] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.755354] [<ffffffffc087f0ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[124679.756347] [<ffffffffc0880ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
[124679.757321] [<ffffffffa897fe37>] __down_common+0xaa/0x104
[124679.758311] [<ffffffffc085e210>] ? _xfs_buf_find+0x170/0x330 [xfs]
[124679.759283] [<ffffffffa897feae>] __down+0x1d/0x1f
[124679.760249] [<ffffffffa82cc191>] down+0x41/0x50
[124679.761219] [<ffffffffc085e00c>] xfs_buf_lock+0x3c/0xd0 [xfs]
[124679.762189] [<ffffffffc085e210>] _xfs_buf_find+0x170/0x330 [xfs]
[124679.763160] [<ffffffffc085e3fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
[124679.764132] [<ffffffffc088f511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
[124679.765103] [<ffffffffc083c5a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
[124679.766028] [<ffffffffc0843edf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
[124679.766927] [<ffffffffc0842c38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
[124679.767816] [<ffffffffc086f2b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
[124679.768690] [<ffffffffc084ab3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
[124679.769573] [<ffffffffc087e467>] ? kmem_alloc+0x97/0x130 [xfs]
[124679.770446] [<ffffffffc0840c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
[124679.771326] [<ffffffffc08725cc>] xfs_create+0x56c/0x6a0 [xfs]
[124679.772203] [<ffffffffc086f533>] xfs_generic_create+0xd3/0x290 [xfs]
[124679.773085] [<ffffffffc086f724>] xfs_vn_mknod+0x14/0x20 [xfs]
[124679.773962] [<ffffffffc086f763>] xfs_vn_create+0x13/0x20 [xfs]
[124679.774825] [<ffffffffa8457de3>] vfs_create+0xd3/0x140
[124679.775680] [<ffffffffa8459e7a>] do_last+0x10ca/0x1290
[124679.776540] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.777370] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.778183] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.778992] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.779797] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.780602] [<ffffffffa844981e>] SyS_open+0x1e/0x20
[124679.781405] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.782211] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.783019] INFO: task ffmpeg:15554 blocked for more than 120 seconds.
[124679.783837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.784665] ffmpeg D ffff9e981f2dac80 0 15554 2764 0x00000080
[124679.785515] Call Trace:
[124679.786366] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.787228] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.788059] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.788890] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs]
[124679.789729] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs]
[124679.790560] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.791380] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130
[124679.792193] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90
[124679.792990] [<ffffffffa89801b8>] io_schedule+0x18/0x20
[124679.793786] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs]
[124679.794563] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40
[124679.795357] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.796159] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs]
[124679.796961] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[124679.797766] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[124679.798537] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[124679.799289] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0
[124679.800038] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340
[124679.800789] [<ffffffffa8440351>] ? vmpressure+0x21/0x90
[124679.801532] [<ffffffffa83d497a>] do_try_to_free_pages+0x3ca/0x520
[124679.802284] [<ffffffffa83d4bcc>] try_to_free_pages+0xfc/0x180
[124679.803035] [<ffffffffa83c87f1>] __alloc_pages_nodemask+0x831/0xbe0
[124679.803789] [<ffffffffa841a045>] alloc_pages_vma+0xb5/0x200
[124679.804535] [<ffffffffa83f1984>] handle_pte_fault+0x984/0xe20
[124679.805285] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110
[124679.806038] [<ffffffffa83f3f3d>] handle_mm_fault+0x39d/0x9b0
[124679.806788] [<ffffffffa83e95a2>] __get_user_pages+0x1d2/0x7a0
[124679.807528] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110
[124679.808275] [<ffffffffa83ea022>] get_user_pages+0x52/0x60
[124679.809018] [<ffffffffc021b32b>] dl_get_user_pages+0xab/0xf0 [blackmagic]
[124679.809754] [<ffffffffc02212da>] _ZN18IOMemoryDescriptor7prepareEv+0x3e/0x50 [blackmagic]
[124679.810501] [<ffffffffc02556a0>] _ZN26DaisyCutterDMASupportClass35newVideoFrameDMATransactionNotGatedEP21BlackmagicDMAParamRecP30BlackmagicMainMemoryDescriptorP+0x31a/0x974 [blackmagic]
[124679.812043] [<ffffffffc0238d9f>] ? _ZN32HardwareAbstractionDeckLinkClass13BitTestEndianEPVjj+0x11/0x18 [blackmagic]
[124679.812869] [<ffffffffc021beca>] ? dl_gate_unlock+0x2a/0x30 [blackmagic]
[124679.813698] [<ffffffffc0221bde>] ? _ZN24IOBufferMemoryDescriptor4freeEv+0x40/0x40 [blackmagic]
[124679.814556] [<ffffffffc024d4d9>] _ZN22VideoInputSupportClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRec+0x1a1/0x1d2 [blackmagic]
[124679.816303] [<ffffffffc025bff7>] _ZN15UserClientClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRecj+0x1b/0x24 [blackmagic]
[124679.818159] [<ffffffffc021d6b4>] blackmagic_ioctl_private+0x1466/0x3ef8 [blackmagic]
[124679.819126] [<ffffffffc0219fd9>] blackmagic_ioctl+0x49/0x70 [blackmagic]
[124679.820058] [<ffffffffa845fbc0>] do_vfs_ioctl+0x3a0/0x5a0
[124679.820976] [<ffffffffa8306152>] ? ktime_get_ts64+0x52/0xf0
[124679.821884] [<ffffffffa845fe61>] SyS_ioctl+0xa1/0xc0
[124679.822777] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.823662] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.824544] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.825436] INFO: task ffmpeg:15702 blocked for more than 120 seconds.
[124679.826323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.827246] ffmpeg D ffff9e98152d1070 0 15702 2978 0x00000080
[124679.828159] Call Trace:
[124679.829060] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.829995] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.830896] [<ffffffffc087f0ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[124679.831785] [<ffffffffc0880ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
[124679.832654] [<ffffffffa897fe37>] __down_common+0xaa/0x104
[124679.833537] [<ffffffffc085e210>] ? _xfs_buf_find+0x170/0x330 [xfs]
[124679.834411] [<ffffffffa897feae>] __down+0x1d/0x1f
[124679.835280] [<ffffffffa82cc191>] down+0x41/0x50
[124679.836156] [<ffffffffc085e00c>] xfs_buf_lock+0x3c/0xd0 [xfs]
[124679.837039] [<ffffffffc085e210>] _xfs_buf_find+0x170/0x330 [xfs]
[124679.837927] [<ffffffffc085e3fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
[124679.838816] [<ffffffffc088f511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
[124679.839696] [<ffffffffc083c5a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
[124679.840583] [<ffffffffc0843edf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
[124679.841473] [<ffffffffc0842c38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
[124679.842363] [<ffffffffc086f2b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
[124679.843253] [<ffffffffc084ab3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
[124679.844147] [<ffffffffc087e467>] ? kmem_alloc+0x97/0x130 [xfs]
[124679.845035] [<ffffffffc0840c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
[124679.845930] [<ffffffffc08725cc>] xfs_create+0x56c/0x6a0 [xfs]
[124679.846822] [<ffffffffc086f533>] xfs_generic_create+0xd3/0x290 [xfs]
[124679.847711] [<ffffffffc086f724>] xfs_vn_mknod+0x14/0x20 [xfs]
[124679.848602] [<ffffffffc086f763>] xfs_vn_create+0x13/0x20 [xfs]
[124679.849479] [<ffffffffa8457de3>] vfs_create+0xd3/0x140
[124679.850353] [<ffffffffa8459e7a>] do_last+0x10ca/0x1290
[124679.851221] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.852079] [<ffffffffa82e49b7>] ? update_curr+0x127/0x1e0
[124679.852920] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.853733] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.854516] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.855286] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.856054] [<ffffffffa844981e>] SyS_open+0x1e/0x20
[124679.856801] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.857522] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.858226] INFO: task ffmpeg:15712 blocked for more than 120 seconds.
[124679.858913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.859594] ffmpeg D ffff9e981f21ac80 0 15712 2978 0x00000080
[124679.860290] Call Trace:
[124679.860984] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.861681] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.862381] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.863088] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs]
[124679.863802] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs]
[124679.864510] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.865201] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130
[124679.865883] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90
[124679.866548] [<ffffffffa89801b8>] io_schedule+0x18/0x20
[124679.867221] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs]
[124679.867888] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40
[124679.868564] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.869246] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs]
[124679.869931] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[124679.870610] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[124679.871289] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[124679.871956] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0
[124679.872621] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340
[124679.873288] [<ffffffffa8440351>] ? vmpressure+0x21/0x90
[124679.873953] [<ffffffffa83d497a>] do_try_to_free_pages+0x3ca/0x520
[124679.874617] [<ffffffffa83d4bcc>] try_to_free_pages+0xfc/0x180
[124679.875282] [<ffffffffa83c87f1>] __alloc_pages_nodemask+0x831/0xbe0
[124679.875950] [<ffffffffa841a045>] alloc_pages_vma+0xb5/0x200
[124679.876611] [<ffffffffa83f1984>] handle_pte_fault+0x984/0xe20
[124679.877278] [<ffffffffa83c8662>] ? __alloc_pages_nodemask+0x6a2/0xbe0
[124679.877947] [<ffffffffa83f3f3d>] handle_mm_fault+0x39d/0x9b0
[124679.878608] [<ffffffffa83e95a2>] __get_user_pages+0x1d2/0x7a0
[124679.879272] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110
[124679.879936] [<ffffffffa83ea022>] get_user_pages+0x52/0x60
[124679.880604] [<ffffffffc021b32b>] dl_get_user_pages+0xab/0xf0 [blackmagic]
[124679.881286] [<ffffffffc02212da>] _ZN18IOMemoryDescriptor7prepareEv+0x3e/0x50 [blackmagic]
[124679.881986] [<ffffffffc02556a0>] _ZN26DaisyCutterDMASupportClass35newVideoFrameDMATransactionNotGatedEP21BlackmagicDMAParamRecP30BlackmagicMainMemoryDescriptorP+0x31a/0x974 [blackmagic]
[124679.883423] [<ffffffffc0238d9f>] ? _ZN32HardwareAbstractionDeckLinkClass13BitTestEndianEPVjj+0x11/0x18 [blackmagic]
[124679.884193] [<ffffffffc021beca>] ? dl_gate_unlock+0x2a/0x30 [blackmagic]
[124679.884969] [<ffffffffc0221bde>] ? _ZN24IOBufferMemoryDescriptor4freeEv+0x40/0x40 [blackmagic]
[124679.885766] [<ffffffffc024d4d9>] _ZN22VideoInputSupportClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRec+0x1a1/0x1d2 [blackmagic]
[124679.887384] [<ffffffffc025bff7>] _ZN15UserClientClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRecj+0x1b/0x24 [blackmagic]
[124679.889103] [<ffffffffc021d6b4>] blackmagic_ioctl_private+0x1466/0x3ef8 [blackmagic]
[124679.889997] [<ffffffffc0219fd9>] blackmagic_ioctl+0x49/0x70 [blackmagic]
[124679.890883] [<ffffffffa845fbc0>] do_vfs_ioctl+0x3a0/0x5a0
[124679.891768] [<ffffffffa8988678>] ? __do_page_fault+0x238/0x500
[124679.892639] [<ffffffffa845fe61>] SyS_ioctl+0xa1/0xc0
[124679.893501] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.894361] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.895213] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.896077] INFO: task find:21716 blocked for more than 120 seconds.
[124679.896934] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.897805] find D ffff9e981f4dac80 0 21716 21715 0x00000080
[124679.898679] Call Trace:
[124679.899545] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.900421] [<ffffffffa8982415>] rwsem_down_read_failed+0x105/0x1c0
[124679.901315] [<ffffffffc0870de0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.902203] [<ffffffffa8592d98>] call_rwsem_down_read_failed+0x18/0x30
[124679.903090] [<ffffffffa897fd30>] down_read+0x20/0x40
[124679.903985] [<ffffffffc0870d69>] xfs_ilock+0xd9/0x120 [xfs]
[124679.904877] [<ffffffffc0870de0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.905772] [<ffffffffc08628c0>] xfs_dir_open+0x40/0x80 [xfs]
[124679.906649] [<ffffffffa8448136>] do_dentry_open+0x216/0x2c0
[124679.907535] [<ffffffffa85042b2>] ? security_inode_permission+0x22/0x30
[124679.908438] [<ffffffffc0862880>] ? xfs_file_llseek+0xa0/0xa0 [xfs]
[124679.909330] [<ffffffffa844827a>] vfs_open+0x5a/0xb0
[124679.910216] [<ffffffffa84568c3>] ? may_open+0xa3/0x120
[124679.911101] [<ffffffffa8458fa6>] do_last+0x1f6/0x1290
[124679.911987] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.912870] [<ffffffffa845d682>] ? user_path_at_empty+0x72/0xc0
[124679.913754] [<ffffffffa84473fa>] ? __check_object_size+0x1ca/0x250
[124679.914634] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.915516] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.916394] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.917267] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.918145] [<ffffffffa8449834>] SyS_openat+0x14/0x20
[124679.919020] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.919898] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.920778] INFO: task find:21734 blocked for more than 120 seconds.
[124679.921652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.922539] find D ffff9e981f4dac80 0 21734 21733 0x00000080
[124679.923424] Call Trace:
[124679.924271] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.925108] [<ffffffffa8982415>] rwsem_down_read_failed+0x105/0x1c0
[124679.925947] [<ffffffffc0870de0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.926772] [<ffffffffa8592d98>] call_rwsem_down_read_failed+0x18/0x30
[124679.927576] [<ffffffffa897fd30>] down_read+0x20/0x40
[124679.928366] [<ffffffffc0870d69>] xfs_ilock+0xd9/0x120 [xfs]
[124679.929132] [<ffffffffc0870de0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.929878] [<ffffffffc08628c0>] xfs_dir_open+0x40/0x80 [xfs]
[124679.930591] [<ffffffffa8448136>] do_dentry_open+0x216/0x2c0
[124679.931308] [<ffffffffa85042b2>] ? security_inode_permission+0x22/0x30
[124679.932037] [<ffffffffc0862880>] ? xfs_file_llseek+0xa0/0xa0 [xfs]
[124679.932756] [<ffffffffa844827a>] vfs_open+0x5a/0xb0
[124679.933469] [<ffffffffa84568c3>] ? may_open+0xa3/0x120
[124679.934181] [<ffffffffa8458fa6>] do_last+0x1f6/0x1290
[124679.934890] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.935592] [<ffffffffa845d682>] ? user_path_at_empty+0x72/0xc0
[124679.936293] [<ffffffffa84473fa>] ? __check_object_size+0x1ca/0x250
[124679.936986] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.937661] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.938329] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.938995] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.939661] [<ffffffffa8449834>] SyS_openat+0x14/0x20
[124679.940324] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.940989] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
caspy

caspy

2020-02-19 07:35

reporter   ~0036317

Reproducibility is from several hours till few weeks. under the same load every time.
.edu

.edu

2020-03-19 21:20

reporter   ~0036538

We are affected too, the latest CentOS/RedHat kernel (3.10.0-1062.....el7.x86_64),
happened on multiple mission-critical servers with XFS, kernel updates don't help,
same as 3 other reporters above.
We had to install (and currently use) the debug(!) kernel as a W/A as it prevents all
XFS partitions from being marked read-only. But we are genuinely surprised to see
such a critical issue remains unnoticed, as XFS is now default since CentOS7/RHEL7
(and Centos7 is maintained/supported until 30 June 2024).
Has it been escalated to maybe RedHat? Is there a patch?
kudeyda

kudeyda

2020-03-20 11:38

reporter  

log (11 bytes)    
kernel log
log (11 bytes)   
nomuthetart

nomuthetart

2020-04-20 18:26

reporter   ~0036733

Is anyone still seeing the issue on 3.10.0-1062.18.1.el7.x86_64 ? We had a system that was triggering the issue fairly reliably during RAID resyncs but it hasn't triggered it since we updated to 3.10.0-1062.18.1.el7.x86_64.
caspy

caspy

2020-04-20 21:47

reporter   ~0036737

started my "test" on 3.10.0-1062.18.1.el7.x86_64 just now.
previously it was up to a couple of week to fail.
will answer at least in a month or earlier.
dabadaba

dabadaba

2020-06-05 07:28

reporter   ~0037038

Any update on those machines updated to 3.10.0-1062.18.1.el7.x86_64 ? Have you seen again the hangs?

We are hitting the same on 3.10.0-1062.1.2.el7.x86_64 at least, rebooting being our only solution so far.
ManuelWolfshant

ManuelWolfshant

2020-06-05 09:23

manager   ~0037039

Last edited: 2020-06-05 09:25

 Since CentOS 7.8 is out for almost 3 months already and the current kernel is 3.10.0-1127.10.1.el7.x86_64, I suggest to update and retry. If you still encounter the same issue, please open a bug at bugzilla.redhat.com.
nomuthetart

nomuthetart

2020-06-05 15:32

reporter   ~0037043

We have not seen the issue return on any of the machines we updated. The first machine we patched triggered the issue weekly or bi-weekly before updating and has now been up 50 days without triggering the issue (still 3.10.0-1062.18.1.el7.x86_64).
maf01

maf01

2020-06-05 15:43

reporter   ~0037044

After upgrade no new incidents, but as long as no root cause was identified and fixed I‘m sceptic. Might be just a side effect of unrelated changes (leading to different timings etc.) and that the bug could reoccur in future releases.
ManuelWolfshant

ManuelWolfshant

2020-06-06 00:14

manager   ~0037045

@maf01: skepticism is good. you are welcome to read the changelog for the kernel package and the release notes so as to verify if any xfs-related changes introduced by RH in the kernel released for 7.8 include the fix for the issue you were facing.

@nomuthetart: if you must keep using older kernels, then you are strongly advised to purchase a RHEL EUS subscription. Unlike RedHat, CentOS does not have the resources and cannot provide support for anything but the most recent version of the packages it ships. And this is especially true for the upgrades related to the release of new minor CentOS versions. See also https://wiki.centos.org/FAQ/General#How_does_CentOS_versioning_work.3F , #2.1 and #3.1.1
nomuthetart

nomuthetart

2020-06-06 00:35

reporter   ~0037046

@ManuelWolfshant - this bug was a reintroduced version of https://bugs.centos.org/view.php?id=13843 and is sporadic making it difficult to know for sure if it is fixed or not. In our case we kept back hosts that hadn't been patched to 3.10.0-1062.4.1.el7.x86_64 and put those that had been updated onto the debug kernel while waiting for a patch. We only had a handful of hosts that could reliably trigger the bug so we put our most reliable one in a non-critical role and kept it fully patched to test for a fix. Once it made it to a couple weeks uptime on 3.10.0-1062.18.1.el7.x86_64 I posted here and once it hit 30 days without crashing we resumed regular patching. At this point it is only running as a long term test.

I didn't see anything XFS related in https://www.redhat.com/archives/rhsa-announce/2020-March/msg00061.html but there were MD changes; since we only saw it on XFS w/ MD setups (our default) we assumed that is what fixed it.
xtomi

xtomi

2021-04-23 17:03

reporter   ~0038417

Hello @ManuelWolfshant, @casp or @nomuthetart, has this issue been mitigated for you by using 3.10.0-1062.18.1.el7.x86_64 ? Any further conclusion on this bug? Thanks a lot
nomuthetart

nomuthetart

2021-04-23 18:28

reporter   ~0038418

Hi @xtomi ; after we resumed regular patching the issue hasn't reoccurred. We did eventually patch our long term test host so we aren't running anymore hosts with 3.10.0-1062.18.1.el7.x86_64 but we haven't see the XFS issue on any newer kernels either. In our opinion it is fully resolved and all of our hosts are back on regular patching schedules again.
xtomi

xtomi

2021-04-23 21:46

reporter   ~0038419

Thanks a lot @nomuthetart for provided info.
slayerduck

slayerduck

2021-04-24 05:56

reporter   ~0038420

I don't know what kernel upgrade fixed this, but i haven't seen this issue anymore ever since.
caspy

caspy

2021-04-24 09:22

reporter   ~0038421

3.10.0-1062.18.1.el7.x86_64
uptime 291 days, all ok.

ps: more fresh kernels not tested.
xtomi

xtomi

2021-04-24 09:35

reporter   ~0038422

@slayerduck @caspy thank both you guys for confirmation. it seems to be very promising :-)

Issue History

Date Modified Username Field Change
2019-11-09 08:44 bruceleeeee New Issue
2019-11-09 08:44 bruceleeeee Tag Attached: 3.10.0-1062.1.1.el7.x86_64
2019-11-09 08:44 bruceleeeee Tag Attached: 7.7
2019-11-09 08:44 bruceleeeee Tag Attached: centos 7
2019-11-09 08:44 bruceleeeee Tag Attached: file system
2020-01-10 13:50 maf01 Note Added: 0035998
2020-01-14 00:05 slayerduck File Added: messages.txt
2020-01-14 00:05 slayerduck Note Added: 0036017
2020-02-19 07:30 caspy Note Added: 0036316
2020-02-19 07:35 caspy Note Added: 0036317
2020-03-19 21:20 .edu Note Added: 0036538
2020-03-20 11:38 kudeyda File Added: log
2020-04-20 18:26 nomuthetart Note Added: 0036733
2020-04-20 21:47 caspy Note Added: 0036737
2020-06-05 07:28 dabadaba Note Added: 0037038
2020-06-05 09:23 ManuelWolfshant Note Added: 0037039
2020-06-05 09:25 ManuelWolfshant Note Edited: 0037039
2020-06-05 15:32 nomuthetart Note Added: 0037043
2020-06-05 15:43 maf01 Note Added: 0037044
2020-06-06 00:14 ManuelWolfshant Note Added: 0037045
2020-06-06 00:35 nomuthetart Note Added: 0037046
2021-04-23 17:03 xtomi Note Added: 0038417
2021-04-23 18:28 nomuthetart Note Added: 0038418
2021-04-23 21:46 xtomi Note Added: 0038419
2021-04-24 05:56 slayerduck Note Added: 0038420
2021-04-24 09:22 caspy Note Added: 0038421
2021-04-24 09:35 xtomi Note Added: 0038422