View Issue Details

IDProjectCategoryView StatusLast Update
0016711CentOS-7kernelpublic2020-03-20 12:08
Reporterbruceleeeee 
PriorityhighSeveritycrashReproducibilityrandom
Status newResolutionopen 
Platformx86_64OSCentOS Linux 7 (Core)OS Version7.7
Product Version7.7-1908 
Target VersionFixed in Version 
Summary0016711: xfsaild blocks after certain time
Descriptionproblem similar to https://bugs.centos.org/view.php?id=13843

kernel version: 3.10.0-1062.4.1.el7.x86_64

started to have problem where xfsaild becomes blocked after a period of time and processes wanting to access the file system begin to grind to a halt. Most often hits chatty IO applications such as auditd first. A reboot solves the problem until the next time. The file system is in a mirrored volume group and neither the volume group nor individual disk reports any problems.
Additional InformationNov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffffc02661ed>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff993bcf1f>] ? get_target_pstate_use_performance+0x8f/0xc0
Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffffc02668d9>] xlog_write+0x1a9/0x750 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc0263827>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc0268708>] xlog_cil_push+0x2a8/0x430 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc02688a5>] xlog_cil_push_work+0x15/0x20 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd1df>] process_one_work+0x17f/0x440
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe448>] worker_thread+0x278/0x3c0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: INFO: task kworker/u32:2:23070 blocked for more than 120 seconds.
Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:23:28 server1 kernel: kworker/u32:2 D ffff95e9fadea0e0 0 23070 2 0x00000080
Nov 8 09:23:28 server1 kernel: Workqueue: writeback bdi_writeback_workfn (flush-9:125)
Nov 8 09:23:28 server1 kernel: Call Trace:
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff995804f5>] rwsem_down_read_failed+0x105/0x1c0
Nov 8 09:23:28 server1 kernel: [<ffffffffc023c047>] ? xfs_map_blocks+0x87/0x220 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff991913f8>] call_rwsem_down_read_failed+0x18/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc90>] down_read+0x20/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffffc0255d69>] xfs_ilock+0xd9/0x120 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc023c047>] xfs_map_blocks+0x87/0x220 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc023d1a4>] xfs_do_writepage+0x174/0x550 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fc82ac>] write_cache_pages+0x21c/0x470
Nov 8 09:23:28 server1 kernel: [<ffffffffc023d030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff9914f710>] ? submit_bio+0x70/0x150
Nov 8 09:23:28 server1 kernel: [<ffffffffc023cffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fc92f1>] do_writepages+0x21/0x50
Nov 8 09:23:28 server1 kernel: [<ffffffff99077f70>] __writeback_single_inode+0x40/0x260
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec6265>] ? wake_up_bit+0x25/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff99078b04>] writeback_sb_inodes+0x1c4/0x430
Nov 8 09:23:28 server1 kernel: [<ffffffff99078e0f>] __writeback_inodes_wb+0x9f/0xd0
Nov 8 09:23:28 server1 kernel: [<ffffffff990792f3>] wb_writeback+0x263/0x2f0
Nov 8 09:23:28 server1 kernel: [<ffffffff9906519c>] ? get_nr_inodes+0x4c/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff99079eeb>] bdi_writeback_workfn+0x2cb/0x460
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd1df>] process_one_work+0x17f/0x440
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe2f6>] worker_thread+0x126/0x3c0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21
Nov 8 09:23:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:23:28 server1 kernel: INFO: task dd:27364 blocked for more than 120 seconds.
Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:23:28 server1 kernel: dd D ffff95e5cc4820e0 0 27364 1 0x00000080
Nov 8 09:23:28 server1 kernel: Call Trace:
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff9957c491>] schedule_timeout+0x221/0x2d0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed6619>] ? ttwu_do_wakeup+0x19/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed674f>] ? ttwu_do_activate+0x6f/0x80
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed9ed0>] ? try_to_wake_up+0x190/0x390
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eebd>] wait_for_completion+0xfd/0x140
Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd96a>] flush_work+0x10a/0x1b0
Nov 8 09:23:28 server1 kernel: [<ffffffff98eba680>] ? move_linked_works+0x90/0x90
Nov 8 09:23:28 server1 kernel: [<ffffffffc02690da>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc02674a4>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fbb22f>] ? filemap_fdatawait_range+0x1f/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc82>] ? down_read+0x12/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffffc0247a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e2a7>] do_fsync+0x67/0xb0
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e5b3>] SyS_fdatasync+0x13/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a
Nov 8 09:23:28 server1 kernel: INFO: task exim:27365 blocked for more than 120 seconds.
Nov 8 09:23:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:23:28 server1 kernel: exim D ffff95e95a2e5230 0 27365 27362 0x00000080
Nov 8 09:23:28 server1 kernel: Call Trace:
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed56de>] ? resched_curr+0xae/0xc0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ee49a6>] ? check_preempt_wakeup+0x166/0x250
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:23:28 server1 kernel: [<ffffffff9957c491>] schedule_timeout+0x221/0x2d0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed6619>] ? ttwu_do_wakeup+0x19/0xe0
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed674f>] ? ttwu_do_activate+0x6f/0x80
Nov 8 09:23:28 server1 kernel: [<ffffffff98ed9ed0>] ? try_to_wake_up+0x190/0x390
Nov 8 09:23:28 server1 kernel: [<ffffffff9957eebd>] wait_for_completion+0xfd/0x140
Nov 8 09:23:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff98ebd96a>] flush_work+0x10a/0x1b0
Nov 8 09:23:28 server1 kernel: [<ffffffff98eba680>] ? move_linked_works+0x90/0x90
Nov 8 09:23:28 server1 kernel: [<ffffffffc02690da>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffffc02674a4>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff98fbb22f>] ? filemap_fdatawait_range+0x1f/0x30
Nov 8 09:23:28 server1 kernel: [<ffffffff9957dc82>] ? down_read+0x12/0x40
Nov 8 09:23:28 server1 kernel: [<ffffffffc0247a3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e2a7>] do_fsync+0x67/0xb0
Nov 8 09:23:28 server1 kernel: [<ffffffff9907e590>] SyS_fsync+0x10/0x20
Nov 8 09:23:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a

Nov 8 09:25:28 server1 kernel: INFO: task xfsaild/md125:820 blocked for more than 120 seconds.
Nov 8 09:25:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:25:28 server1 kernel: xfsaild/md125 D ffff95e9fbf69070 0 820 2 0x00000000
Nov 8 09:25:28 server1 kernel: Call Trace:
Nov 8 09:25:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:25:28 server1 kernel: [<ffffffffc0267136>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff98eda190>] ? wake_up_state+0x20/0x20
Nov 8 09:25:28 server1 kernel: [<ffffffffc0273550>] ? xfsaild+0x180/0x760 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc026723c>] xfs_log_force+0x2c/0x70 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc02733d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc0273550>] xfsaild+0x180/0x760 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc02733d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff98ec51b1>] kthread+0xd1/0xe0
Nov 8 09:25:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:25:28 server1 kernel: [<ffffffff9958bd37>] ret_from_fork_nospec_begin+0x21/0x21
Nov 8 09:25:28 server1 kernel: [<ffffffff98ec50e0>] ? insert_kthread_work+0x40/0x40
Nov 8 09:25:28 server1 kernel: INFO: task auditd:1348 blocked for more than 120 seconds.
Nov 8 09:25:28 server1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 8 09:25:28 server1 kernel: auditd D ffffffff9957e465 0 1348 1 0x00000000
Nov 8 09:25:28 server1 kernel: Call Trace:
Nov 8 09:25:28 server1 kernel: [<ffffffff9957eb09>] schedule+0x29/0x70
Nov 8 09:25:28 server1 kernel: [<ffffffff99580245>] rwsem_down_write_failed+0x215/0x3c0
Nov 8 09:25:28 server1 kernel: [<ffffffffc0263ee4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc025307d>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff99191427>] call_rwsem_down_write_failed+0x17/0x30
Nov 8 09:25:28 server1 kernel: [<ffffffff9957dcdd>] down_write+0x2d/0x3d
Nov 8 09:25:28 server1 kernel: [<ffffffffc0255d51>] xfs_ilock+0xc1/0x120 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc025307d>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff990663a8>] update_time+0x28/0xd0
Nov 8 09:25:28 server1 kernel: [<ffffffff98ed2723>] ? __wake_up+0x13/0x20
Nov 8 09:25:28 server1 kernel: [<ffffffff990664f0>] file_update_time+0xa0/0xf0
Nov 8 09:25:28 server1 kernel: [<ffffffffc0248adf>] xfs_file_aio_write_checks+0x16f/0x1c0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc024935a>] xfs_file_buffered_aio_write+0xca/0x2c0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffffc02496dd>] xfs_file_aio_write+0x18d/0x1b0 [xfs]
Nov 8 09:25:28 server1 kernel: [<ffffffff99048433>] do_sync_write+0x93/0xe0
Nov 8 09:25:28 server1 kernel: [<ffffffff99048f20>] vfs_write+0xc0/0x1f0
Nov 8 09:25:28 server1 kernel: [<ffffffff99049d3f>] SyS_write+0x7f/0xf0
Nov 8 09:25:28 server1 kernel: [<ffffffff9958bede>] system_call_fastpath+0x25/0x2a
Tags3.10.0-1062.1.1.el7.x86_64, 7.7, centos 7, file system
abrt_hash
URL

Activities

maf01

maf01

2020-01-10 13:50

reporter   ~0035998

The same here:

Centos7 3.10.0-1062.9.1.el7.x86_64

<code>
Jan 10 00:12:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:12:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:12:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:12:55 x3421 kernel: Call Trace:
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:12:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:12:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:12:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:14:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:14:55 x3421 kernel: Call Trace:
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: INFO: task kworker/0:2:18279 blocked for more than 120 seconds.
Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:14:55 x3421 kernel: kworker/0:2 D ffff9b4a1511e2a0 0 18279 2 0x00000080
Jan 10 00:14:55 x3421 kernel: Workqueue: xfs-sync/md125 xfs_log_worker [xfs]
Jan 10 00:14:55 x3421 kernel: Call Trace:
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8e83b4>] ? update_group_power+0x174/0x280
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cb81635>] ? cpumask_next_and+0x35/0x50
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8aecbd>] ? add_timer_on+0x8d/0xf0
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dcfd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dd296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf336>] worker_thread+0x126/0x3c0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: INFO: task kworker/0:3:7799 blocked for more than 120 seconds.
Jan 10 00:14:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:14:55 x3421 kernel: kworker/0:3 D ffff9b4dfec1ac80 0 7799 2 0x00000080
Jan 10 00:14:55 x3421 kernel: Workqueue: xfs-cil/md125 xlog_cil_push_work [xfs]
Jan 10 00:14:55 x3421 kernel: Call Trace:
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dc1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06dc8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06d9807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06de6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffffc06de885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf488>] worker_thread+0x278/0x3c0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:14:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:14:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task grafana-server:27514 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: grafana-server D ffff9b4dfed1ac80 0 27514 4384 0x00000080
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d77a9>] ? ttwu_do_wakeup+0x19/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d78df>] ? ttwu_do_activate+0x6f/0x80
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db200>] ? try_to_wake_up+0x190/0x390
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd484>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c9bcaff>] ? filemap_fdatawait_range+0x1f/0x30
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7fd02>] ? down_read+0x12/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06bda3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fb77>] do_fsync+0x67/0xb0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fe60>] SyS_fsync+0x10/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dede>] system_call_fastpath+0x25/0x2a
Jan 10 00:16:55 x3421 kernel: INFO: task kworker/0:2:18279 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: kworker/0:2 D ffff9b4a1511e2a0 0 18279 2 0x00000080
Jan 10 00:16:55 x3421 kernel: Workqueue: xfs-sync/md125 xfs_log_worker [xfs]
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8e83b4>] ? update_group_power+0x174/0x280
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cb81635>] ? cpumask_next_and+0x35/0x50
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8aecbd>] ? add_timer_on+0x8d/0xf0
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dcfd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf336>] worker_thread+0x126/0x3c0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task kworker/0:3:7799 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: kworker/0:3 D ffff9b4dfec1ac80 0 7799 2 0x00000080
Jan 10 00:16:55 x3421 kernel: Workqueue: xfs-cil/md125 xlog_cil_push_work [xfs]
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dc1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dc8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06d9807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06de6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06de885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be21f>] process_one_work+0x17f/0x440
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf488>] worker_thread+0x278/0x3c0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:16:55 x3421 kernel: INFO: task redis-server:14049 blocked for more than 120 seconds.
Jan 10 00:16:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:16:55 x3421 kernel: redis-server D ffff9b4dfec1ac80 0 14049 4378 0x00000080
Jan 10 00:16:55 x3421 kernel: Call Trace:
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c82b621>] ? __switch_to+0x151/0x580
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7e511>] schedule_timeout+0x221/0x2d0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8d78df>] ? ttwu_do_activate+0x6f/0x80
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf80dbd>] wait_for_completion+0xfd/0x140
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8be9aa>] flush_work+0x10a/0x1b0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c8bb6c0>] ? move_linked_works+0x90/0x90
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06df0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06dd484>] _xfs_log_force_lsn+0x74/0x310 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9c9bcaff>] ? filemap_fdatawait_range+0x1f/0x30
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf7fd02>] ? down_read+0x12/0x40
Jan 10 00:16:55 x3421 kernel: [<ffffffffc06bda3d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fb77>] do_fsync+0x67/0xb0
Jan 10 00:16:55 x3421 kernel: [<ffffffff9ca7fe60>] SyS_fsync+0x10/0x20
Jan 10 00:16:55 x3421 kernel: [<ffffffff9cf8dede>] system_call_fastpath+0x25/0x2a
Jan 10 00:18:55 x3421 kernel: INFO: task xfsaild/md125:827 blocked for more than 120 seconds.
Jan 10 00:18:55 x3421 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 10 00:18:55 x3421 kernel: xfsaild/md125 D ffff9b4dfed1ac80 0 827 2 0x00000000
Jan 10 00:18:55 x3421 kernel: Call Trace:
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8ae4de>] ? try_to_del_timer_sync+0x5e/0x90
Jan 10 00:18:55 x3421 kernel: [<ffffffff9cf80a09>] schedule+0x29/0x70
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06dd116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8db4c0>] ? wake_up_state+0x20/0x20
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e9530>] ? xfsaild+0x180/0x760 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06dd21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e9530>] xfsaild+0x180/0x760 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffffc06e93b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c61f1>] kthread+0xd1/0xe0
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
Jan 10 00:18:55 x3421 kernel: [<ffffffff9cf8dd24>] ret_from_fork_nospec_begin+0xe/0x21
Jan 10 00:18:55 x3421 kernel: [<ffffffff9c8c6120>] ? insert_kthread_work+0x40/0x40
</code>
slayerduck

slayerduck

2020-01-14 00:05

reporter   ~0036017

Centos7 3.10.0-1062.9.1.el7.x86_64 and 3.10.0-1062.4.3.el7.x86_64 kernels here. Several servers with different storage configuration are displaying the issues. All with same motherboard Supermicro X9DRW-iF

messages.txt (21,918 bytes)
Jan 14 00:03:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds.
Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:03:03 caro kernel: kworker/14:0    D ffff9e847f11ac80     0 84673      2 0x00000080
Jan 14 00:03:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs]
Jan 14 00:03:03 caro kernel: Call Trace:
Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:03:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:03:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds.
Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:03:03 caro kernel: kworker/14:2    D ffff9e8470fd9070     0 85328      2 0x00000080
Jan 14 00:03:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs]
Jan 14 00:03:03 caro kernel: Call Trace:
Jan 14 00:03:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280
Jan 14 00:03:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50
Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:03:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:03:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:03:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:03:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:03:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:03:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0
Jan 14 00:03:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580
Jan 14 00:03:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0
Jan 14 00:03:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:03:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:03:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds.
Jan 14 00:03:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:03:03 caro kernel: rsync           D ffff9e825abe5230     0 86419  86399 0x00000080
Jan 14 00:03:03 caro kernel: Call Trace:
Jan 14 00:03:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:03:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:03:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104
Jan 14 00:03:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f
Jan 14 00:03:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50
Jan 14 00:03:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs]
Jan 14 00:03:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140
Jan 14 00:03:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290
Jan 14 00:03:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0
Jan 14 00:03:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0
Jan 14 00:03:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170
Jan 14 00:03:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220
Jan 14 00:03:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20
Jan 14 00:03:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
Jan 14 00:05:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: kworker/14:0    D ffff9e847f11ac80     0 84673      2 0x00000080
Jan 14 00:05:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs]
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: kworker/14:2    D ffff9e8470fd9070     0 85328      2 0x00000080
Jan 14 00:05:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs]
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280
Jan 14 00:05:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:05:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:05:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0
Jan 14 00:05:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:05:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:05:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: rsync           D ffff9e825abe5230     0 86419  86399 0x00000080
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:05:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104
Jan 14 00:05:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f
Jan 14 00:05:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50
Jan 14 00:05:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140
Jan 14 00:05:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290
Jan 14 00:05:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0
Jan 14 00:05:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0
Jan 14 00:05:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170
Jan 14 00:05:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220
Jan 14 00:05:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
Jan 14 00:05:03 caro kernel: INFO: task xfs_db:87606 blocked for more than 120 seconds.
Jan 14 00:05:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:05:03 caro kernel: xfs_db          D ffff9e747f89ac80     0 87606  87605 0x00000080
Jan 14 00:05:03 caro kernel: Call Trace:
Jan 14 00:05:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:05:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:05:03 caro kernel: [<ffffffffad62b621>] ? __switch_to+0x151/0x580
Jan 14 00:05:03 caro kernel: [<ffffffffad635c19>] ? sched_clock+0x9/0x10
Jan 14 00:05:03 caro kernel: [<ffffffffad6de6b5>] ? sched_clock_cpu+0x85/0xc0
Jan 14 00:05:03 caro kernel: [<ffffffffad6db200>] ? try_to_wake_up+0x190/0x390
Jan 14 00:05:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:05:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:05:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c1e0d>] ? xfs_fs_sync_fs+0x2d/0x50 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffc02c1e0d>] xfs_fs_sync_fs+0x2d/0x50 [xfs]
Jan 14 00:05:03 caro kernel: [<ffffffffad87fc70>] sync_filesystem+0x80/0xb0
Jan 14 00:05:03 caro kernel: [<ffffffffad88b304>] fsync_bdev+0x24/0x60
Jan 14 00:05:03 caro kernel: [<ffffffffad963948>] blkdev_ioctl+0x718/0xa20
Jan 14 00:05:03 caro kernel: [<ffffffffad7c07a7>] ? mempool_free_slab+0x17/0x20
Jan 14 00:05:03 caro kernel: [<ffffffffad88a8b1>] block_ioctl+0x41/0x50
Jan 14 00:05:03 caro kernel: [<ffffffffad85fb40>] do_vfs_ioctl+0x3a0/0x5a0
Jan 14 00:05:03 caro kernel: [<ffffffffad95476a>] ? blkdev_issue_flush+0xda/0x110
Jan 14 00:05:03 caro kernel: [<ffffffffad85fde1>] SyS_ioctl+0xa1/0xc0
Jan 14 00:05:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
Jan 14 00:07:03 caro kernel: INFO: task kworker/14:0:84673 blocked for more than 120 seconds.
Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:07:03 caro kernel: kworker/14:0    D ffff9e847f11ac80     0 84673      2 0x00000080
Jan 14 00:07:03 caro kernel: Workqueue: xfs-cil/md0 xlog_cil_push_work [xfs]
Jan 14 00:07:03 caro kernel: Call Trace:
Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:07:03 caro kernel: [<ffffffffc02ca1cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:07:03 caro kernel: [<ffffffffc02ca8b9>] xlog_write+0x1a9/0x750 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02c7807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cc6e8>] xlog_cil_push+0x2a8/0x430 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cc885>] xlog_cil_push_work+0x15/0x20 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf488>] worker_thread+0x278/0x3c0
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: INFO: task kworker/14:2:85328 blocked for more than 120 seconds.
Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:07:03 caro kernel: kworker/14:2    D ffff9e8470fd9070     0 85328      2 0x00000080
Jan 14 00:07:03 caro kernel: Workqueue: xfs-sync/md0 xfs_log_worker [xfs]
Jan 14 00:07:03 caro kernel: Call Trace:
Jan 14 00:07:03 caro kernel: [<ffffffffad6e83b4>] ? update_group_power+0x174/0x280
Jan 14 00:07:03 caro kernel: [<ffffffffad981635>] ? cpumask_next_and+0x35/0x50
Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:07:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:07:03 caro kernel: [<ffffffffadd80dbd>] wait_for_completion+0xfd/0x140
Jan 14 00:07:03 caro kernel: [<ffffffffad6db4c0>] ? wake_up_state+0x20/0x20
Jan 14 00:07:03 caro kernel: [<ffffffffad6be9aa>] flush_work+0x10a/0x1b0
Jan 14 00:07:03 caro kernel: [<ffffffffad6bb6c0>] ? move_linked_works+0x90/0x90
Jan 14 00:07:03 caro kernel: [<ffffffffc02cd0ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6e4ebc>] ? dequeue_entity+0x11c/0x5c0
Jan 14 00:07:03 caro kernel: [<ffffffffc02cafd6>] _xfs_log_force+0x86/0x2a0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad62b59e>] ? __switch_to+0xce/0x580
Jan 14 00:07:03 caro kernel: [<ffffffffc02cb296>] ? xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cb21c>] xfs_log_force+0x2c/0x70 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02cb296>] xfs_log_worker+0x36/0x100 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad6be21f>] process_one_work+0x17f/0x440
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf336>] worker_thread+0x126/0x3c0
Jan 14 00:07:03 caro kernel: [<ffffffffad6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c61f1>] kthread+0xd1/0xe0
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: [<ffffffffadd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
Jan 14 00:07:03 caro kernel: [<ffffffffad6c6120>] ? insert_kthread_work+0x40/0x40
Jan 14 00:07:03 caro kernel: INFO: task rsync:86419 blocked for more than 120 seconds.
Jan 14 00:07:03 caro kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 14 00:07:03 caro kernel: rsync           D ffff9e825abe5230     0 86419  86399 0x00000080
Jan 14 00:07:03 caro kernel: Call Trace:
Jan 14 00:07:03 caro kernel: [<ffffffffadd80a09>] schedule+0x29/0x70
Jan 14 00:07:03 caro kernel: [<ffffffffadd7e511>] schedule_timeout+0x221/0x2d0
Jan 14 00:07:03 caro kernel: [<ffffffffc02c80ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02c9ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffadd7fe17>] __down_common+0xaa/0x104
Jan 14 00:07:03 caro kernel: [<ffffffffc02a7210>] ? _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffadd7fe8e>] __down+0x1d/0x1f
Jan 14 00:07:03 caro kernel: [<ffffffffad6cc191>] down+0x41/0x50
Jan 14 00:07:03 caro kernel: [<ffffffffc02a700c>] xfs_buf_lock+0x3c/0xd0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02a7210>] _xfs_buf_find+0x170/0x330 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02a73fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02d8511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02855a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc028cedf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc028bc38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b82b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc0293b3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02c7467>] ? kmem_alloc+0x97/0x130 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc0289c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02bb5cc>] xfs_create+0x56c/0x6a0 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b8533>] xfs_generic_create+0xd3/0x290 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b8724>] xfs_vn_mknod+0x14/0x20 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffc02b8763>] xfs_vn_create+0x13/0x20 [xfs]
Jan 14 00:07:03 caro kernel: [<ffffffffad857d63>] vfs_create+0xd3/0x140
Jan 14 00:07:03 caro kernel: [<ffffffffad859dfa>] do_last+0x10ca/0x1290
Jan 14 00:07:03 caro kernel: [<ffffffffad85bdbd>] path_openat+0xcd/0x5a0
Jan 14 00:07:03 caro kernel: [<ffffffffad85d72d>] do_filp_open+0x4d/0xb0
Jan 14 00:07:03 caro kernel: [<ffffffffad86b207>] ? __alloc_fd+0x47/0x170
Jan 14 00:07:03 caro kernel: [<ffffffffad849684>] do_sys_open+0x124/0x220
Jan 14 00:07:03 caro kernel: [<ffffffffad84979e>] SyS_open+0x1e/0x20
Jan 14 00:07:03 caro kernel: [<ffffffffadd8dede>] system_call_fastpath+0x25/0x2a
messages.txt (21,918 bytes)
caspy

caspy

2020-02-19 07:30

reporter   ~0036316

# uname -a
Linux hostname 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


[124679.668777] INFO: task kswapd0:106 blocked for more than 120 seconds.
[124679.668841] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.668907] kswapd0 D ffff9e981f45ac80 0 106 2 0x00000000
[124679.668975] Call Trace:
[124679.669047] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.669105] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.669152] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.669233] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs]
[124679.669321] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs]
[124679.669404] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.669459] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130
[124679.669515] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90
[124679.669566] [<ffffffffa89801b8>] io_schedule+0x18/0x20
[124679.669638] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs]
[124679.669695] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40
[124679.669790] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.669872] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs]
[124679.669952] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[124679.670037] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[124679.670120] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[124679.670186] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0
[124679.670237] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340
[124679.670288] [<ffffffffa84403b7>] ? vmpressure+0x87/0x90
[124679.670338] [<ffffffffa83d53e8>] balance_pgdat+0x3a8/0x5e0
[124679.670389] [<ffffffffa83d5793>] kswapd+0x173/0x440
[124679.670435] [<ffffffffa82c72e0>] ? wake_up_atomic_t+0x30/0x30
[124679.670488] [<ffffffffa83d5620>] ? balance_pgdat+0x5e0/0x5e0
[124679.670539] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.670584] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.670639] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.670696] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.670774] INFO: task kworker/1:5:126 blocked for more than 120 seconds.
[124679.670845] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.670909] kworker/1:5 D ffff9e981f25ac80 0 126 2 0x00000000
[124679.671010] Workqueue: xfs-cil/md124 xlog_cil_push_work [xfs]
[124679.671061] Call Trace:
[124679.671088] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.671161] [<ffffffffc08811cd>] xlog_state_get_iclog_space+0x10d/0x320 [xfs]
[124679.671225] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20
[124679.672983] [<ffffffffc08818b9>] xlog_write+0x1a9/0x750 [xfs]
[124679.674779] [<ffffffffc087e807>] ? kmem_zone_alloc+0x97/0x130 [xfs]
[124679.676576] [<ffffffffc08836e8>] xlog_cil_push+0x2a8/0x430 [xfs]
[124679.678345] [<ffffffffc0883885>] xlog_cil_push_work+0x15/0x20 [xfs]
[124679.680030] [<ffffffffa82be21f>] process_one_work+0x17f/0x440
[124679.681548] [<ffffffffa82bf488>] worker_thread+0x278/0x3c0
[124679.683005] [<ffffffffa82bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[124679.684483] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.685965] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.687414] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.688847] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.690272] INFO: task xfsaild/md124:1947 blocked for more than 120 seconds.
[124679.691611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.692952] xfsaild/md124 D ffff9e981f51ac80 0 1947 2 0x00000000
[124679.694312] Call Trace:
[124679.695656] [<ffffffffa82ae4de>] ? try_to_del_timer_sync+0x5e/0x90
[124679.697030] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.698419] [<ffffffffc0882116>] _xfs_log_force+0x1c6/0x2a0 [xfs]
[124679.699796] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20
[124679.701193] [<ffffffffc088e530>] ? xfsaild+0x180/0x760 [xfs]
[124679.702507] [<ffffffffc088221c>] xfs_log_force+0x2c/0x70 [xfs]
[124679.703810] [<ffffffffc088e3b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[124679.705125] [<ffffffffc088e530>] xfsaild+0x180/0x760 [xfs]
[124679.706431] [<ffffffffc088e3b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[124679.707732] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.709026] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.710331] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.711652] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.712907] INFO: task kworker/1:0:16826 blocked for more than 120 seconds.
[124679.714149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.715419] kworker/1:0 D ffff9e9818759070 0 16826 2 0x00000080
[124679.716739] Workqueue: xfs-sync/md124 xfs_log_worker [xfs]
[124679.718034] Call Trace:
[124679.719324] [<ffffffffa82e8278>] ? update_group_power+0x28/0x280
[124679.720645] [<ffffffffa8581685>] ? cpumask_next_and+0x35/0x50
[124679.721967] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.723222] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.724464] [<ffffffffa82e150e>] ? account_entity_dequeue+0xae/0xd0
[124679.725709] [<ffffffffa82e4ecc>] ? dequeue_entity+0x11c/0x5c0
[124679.726945] [<ffffffffa82e64d1>] ? put_prev_entity+0x31/0x400
[124679.728167] [<ffffffffa82e2799>] ? pick_next_entity+0xa9/0x190
[124679.729370] [<ffffffffa8980ddd>] wait_for_completion+0xfd/0x140
[124679.730559] [<ffffffffa82db4d0>] ? wake_up_state+0x20/0x20
[124679.731761] [<ffffffffa82be9aa>] flush_work+0x10a/0x1b0
[124679.732930] [<ffffffffa82bb6c0>] ? move_linked_works+0x90/0x90
[124679.734066] [<ffffffffc08840ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
[124679.735168] [<ffffffffa82e4ecc>] ? dequeue_entity+0x11c/0x5c0
[124679.736282] [<ffffffffc0881fd6>] _xfs_log_force+0x86/0x2a0 [xfs]
[124679.737381] [<ffffffffa822b59e>] ? __switch_to+0xce/0x580
[124679.738493] [<ffffffffc0882296>] ? xfs_log_worker+0x36/0x100 [xfs]
[124679.739609] [<ffffffffc088221c>] xfs_log_force+0x2c/0x70 [xfs]
[124679.740730] [<ffffffffc0882296>] xfs_log_worker+0x36/0x100 [xfs]
[124679.741827] [<ffffffffa82be21f>] process_one_work+0x17f/0x440
[124679.742925] [<ffffffffa82bf336>] worker_thread+0x126/0x3c0
[124679.744020] [<ffffffffa82bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[124679.745072] [<ffffffffa82c61f1>] kthread+0xd1/0xe0
[124679.746106] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.747143] [<ffffffffa898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[124679.748181] [<ffffffffa82c6120>] ? insert_kthread_work+0x40/0x40
[124679.749207] INFO: task ffmpeg:15545 blocked for more than 120 seconds.
[124679.750233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.751262] ffmpeg D ffff9e96daf220e0 0 15545 2764 0x00000080
[124679.752301] Call Trace:
[124679.753318] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.754349] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.755354] [<ffffffffc087f0ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[124679.756347] [<ffffffffc0880ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
[124679.757321] [<ffffffffa897fe37>] __down_common+0xaa/0x104
[124679.758311] [<ffffffffc085e210>] ? _xfs_buf_find+0x170/0x330 [xfs]
[124679.759283] [<ffffffffa897feae>] __down+0x1d/0x1f
[124679.760249] [<ffffffffa82cc191>] down+0x41/0x50
[124679.761219] [<ffffffffc085e00c>] xfs_buf_lock+0x3c/0xd0 [xfs]
[124679.762189] [<ffffffffc085e210>] _xfs_buf_find+0x170/0x330 [xfs]
[124679.763160] [<ffffffffc085e3fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
[124679.764132] [<ffffffffc088f511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
[124679.765103] [<ffffffffc083c5a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
[124679.766028] [<ffffffffc0843edf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
[124679.766927] [<ffffffffc0842c38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
[124679.767816] [<ffffffffc086f2b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
[124679.768690] [<ffffffffc084ab3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
[124679.769573] [<ffffffffc087e467>] ? kmem_alloc+0x97/0x130 [xfs]
[124679.770446] [<ffffffffc0840c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
[124679.771326] [<ffffffffc08725cc>] xfs_create+0x56c/0x6a0 [xfs]
[124679.772203] [<ffffffffc086f533>] xfs_generic_create+0xd3/0x290 [xfs]
[124679.773085] [<ffffffffc086f724>] xfs_vn_mknod+0x14/0x20 [xfs]
[124679.773962] [<ffffffffc086f763>] xfs_vn_create+0x13/0x20 [xfs]
[124679.774825] [<ffffffffa8457de3>] vfs_create+0xd3/0x140
[124679.775680] [<ffffffffa8459e7a>] do_last+0x10ca/0x1290
[124679.776540] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.777370] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.778183] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.778992] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.779797] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.780602] [<ffffffffa844981e>] SyS_open+0x1e/0x20
[124679.781405] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.782211] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.783019] INFO: task ffmpeg:15554 blocked for more than 120 seconds.
[124679.783837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.784665] ffmpeg D ffff9e981f2dac80 0 15554 2764 0x00000080
[124679.785515] Call Trace:
[124679.786366] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.787228] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.788059] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.788890] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs]
[124679.789729] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs]
[124679.790560] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.791380] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130
[124679.792193] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90
[124679.792990] [<ffffffffa89801b8>] io_schedule+0x18/0x20
[124679.793786] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs]
[124679.794563] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40
[124679.795357] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.796159] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs]
[124679.796961] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[124679.797766] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[124679.798537] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[124679.799289] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0
[124679.800038] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340
[124679.800789] [<ffffffffa8440351>] ? vmpressure+0x21/0x90
[124679.801532] [<ffffffffa83d497a>] do_try_to_free_pages+0x3ca/0x520
[124679.802284] [<ffffffffa83d4bcc>] try_to_free_pages+0xfc/0x180
[124679.803035] [<ffffffffa83c87f1>] __alloc_pages_nodemask+0x831/0xbe0
[124679.803789] [<ffffffffa841a045>] alloc_pages_vma+0xb5/0x200
[124679.804535] [<ffffffffa83f1984>] handle_pte_fault+0x984/0xe20
[124679.805285] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110
[124679.806038] [<ffffffffa83f3f3d>] handle_mm_fault+0x39d/0x9b0
[124679.806788] [<ffffffffa83e95a2>] __get_user_pages+0x1d2/0x7a0
[124679.807528] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110
[124679.808275] [<ffffffffa83ea022>] get_user_pages+0x52/0x60
[124679.809018] [<ffffffffc021b32b>] dl_get_user_pages+0xab/0xf0 [blackmagic]
[124679.809754] [<ffffffffc02212da>] _ZN18IOMemoryDescriptor7prepareEv+0x3e/0x50 [blackmagic]
[124679.810501] [<ffffffffc02556a0>] _ZN26DaisyCutterDMASupportClass35newVideoFrameDMATransactionNotGatedEP21BlackmagicDMAParamRecP30BlackmagicMainMemoryDescriptorP+0x31a/0x974 [blackmagic]
[124679.812043] [<ffffffffc0238d9f>] ? _ZN32HardwareAbstractionDeckLinkClass13BitTestEndianEPVjj+0x11/0x18 [blackmagic]
[124679.812869] [<ffffffffc021beca>] ? dl_gate_unlock+0x2a/0x30 [blackmagic]
[124679.813698] [<ffffffffc0221bde>] ? _ZN24IOBufferMemoryDescriptor4freeEv+0x40/0x40 [blackmagic]
[124679.814556] [<ffffffffc024d4d9>] _ZN22VideoInputSupportClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRec+0x1a1/0x1d2 [blackmagic]
[124679.816303] [<ffffffffc025bff7>] _ZN15UserClientClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRecj+0x1b/0x24 [blackmagic]
[124679.818159] [<ffffffffc021d6b4>] blackmagic_ioctl_private+0x1466/0x3ef8 [blackmagic]
[124679.819126] [<ffffffffc0219fd9>] blackmagic_ioctl+0x49/0x70 [blackmagic]
[124679.820058] [<ffffffffa845fbc0>] do_vfs_ioctl+0x3a0/0x5a0
[124679.820976] [<ffffffffa8306152>] ? ktime_get_ts64+0x52/0xf0
[124679.821884] [<ffffffffa845fe61>] SyS_ioctl+0xa1/0xc0
[124679.822777] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.823662] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.824544] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.825436] INFO: task ffmpeg:15702 blocked for more than 120 seconds.
[124679.826323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.827246] ffmpeg D ffff9e98152d1070 0 15702 2978 0x00000080
[124679.828159] Call Trace:
[124679.829060] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.829995] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.830896] [<ffffffffc087f0ab>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[124679.831785] [<ffffffffc0880ea6>] ? xlog_sync+0x2e6/0x3f0 [xfs]
[124679.832654] [<ffffffffa897fe37>] __down_common+0xaa/0x104
[124679.833537] [<ffffffffc085e210>] ? _xfs_buf_find+0x170/0x330 [xfs]
[124679.834411] [<ffffffffa897feae>] __down+0x1d/0x1f
[124679.835280] [<ffffffffa82cc191>] down+0x41/0x50
[124679.836156] [<ffffffffc085e00c>] xfs_buf_lock+0x3c/0xd0 [xfs]
[124679.837039] [<ffffffffc085e210>] _xfs_buf_find+0x170/0x330 [xfs]
[124679.837927] [<ffffffffc085e3fa>] xfs_buf_get_map+0x2a/0x250 [xfs]
[124679.838816] [<ffffffffc088f511>] xfs_trans_get_buf_map+0x101/0x170 [xfs]
[124679.839696] [<ffffffffc083c5a2>] xfs_da_get_buf+0xc2/0x100 [xfs]
[124679.840583] [<ffffffffc0843edf>] xfs_dir3_data_init+0x5f/0x280 [xfs]
[124679.841473] [<ffffffffc0842c38>] xfs_dir2_sf_to_block+0xc8/0x620 [xfs]
[124679.842363] [<ffffffffc086f2b7>] ? xfs_setup_inode+0x87/0x120 [xfs]
[124679.843253] [<ffffffffc084ab3d>] xfs_dir2_sf_addname+0xcd/0x5b0 [xfs]
[124679.844147] [<ffffffffc087e467>] ? kmem_alloc+0x97/0x130 [xfs]
[124679.845035] [<ffffffffc0840c88>] xfs_dir_createname+0x1a8/0x200 [xfs]
[124679.845930] [<ffffffffc08725cc>] xfs_create+0x56c/0x6a0 [xfs]
[124679.846822] [<ffffffffc086f533>] xfs_generic_create+0xd3/0x290 [xfs]
[124679.847711] [<ffffffffc086f724>] xfs_vn_mknod+0x14/0x20 [xfs]
[124679.848602] [<ffffffffc086f763>] xfs_vn_create+0x13/0x20 [xfs]
[124679.849479] [<ffffffffa8457de3>] vfs_create+0xd3/0x140
[124679.850353] [<ffffffffa8459e7a>] do_last+0x10ca/0x1290
[124679.851221] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.852079] [<ffffffffa82e49b7>] ? update_curr+0x127/0x1e0
[124679.852920] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.853733] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.854516] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.855286] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.856054] [<ffffffffa844981e>] SyS_open+0x1e/0x20
[124679.856801] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.857522] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.858226] INFO: task ffmpeg:15712 blocked for more than 120 seconds.
[124679.858913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.859594] ffmpeg D ffff9e981f21ac80 0 15712 2978 0x00000080
[124679.860290] Call Trace:
[124679.860984] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.861681] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.862381] [<ffffffffa897e531>] schedule_timeout+0x221/0x2d0
[124679.863088] [<ffffffffc0870449>] ? __xfs_iunpin_wait+0x99/0x150 [xfs]
[124679.863802] [<ffffffffc08824b6>] ? _xfs_log_force_lsn+0xa6/0x310 [xfs]
[124679.864510] [<ffffffffc08736a9>] ? xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.865201] [<ffffffffa898011d>] io_schedule_timeout+0xad/0x130
[124679.865883] [<ffffffffa82c6dd6>] ? prepare_to_wait+0x56/0x90
[124679.866548] [<ffffffffa89801b8>] io_schedule+0x18/0x20
[124679.867221] [<ffffffffc0870489>] __xfs_iunpin_wait+0xd9/0x150 [xfs]
[124679.867888] [<ffffffffa82c73a0>] ? wake_bit_function+0x40/0x40
[124679.868564] [<ffffffffc08736a9>] xfs_iunpin_wait+0x19/0x20 [xfs]
[124679.869246] [<ffffffffc08671c3>] xfs_reclaim_inode+0x143/0x360 [xfs]
[124679.869931] [<ffffffffc0867647>] xfs_reclaim_inodes_ag+0x267/0x390 [xfs]
[124679.870610] [<ffffffffc08686f3>] xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
[124679.871289] [<ffffffffc0878ae5>] xfs_fs_free_cached_objects+0x15/0x20 [xfs]
[124679.871956] [<ffffffffa844d4a9>] prune_super+0xf9/0x1a0
[124679.872621] [<ffffffffa83d16b5>] shrink_slab+0x175/0x340
[124679.873288] [<ffffffffa8440351>] ? vmpressure+0x21/0x90
[124679.873953] [<ffffffffa83d497a>] do_try_to_free_pages+0x3ca/0x520
[124679.874617] [<ffffffffa83d4bcc>] try_to_free_pages+0xfc/0x180
[124679.875282] [<ffffffffa83c87f1>] __alloc_pages_nodemask+0x831/0xbe0
[124679.875950] [<ffffffffa841a045>] alloc_pages_vma+0xb5/0x200
[124679.876611] [<ffffffffa83f1984>] handle_pte_fault+0x984/0xe20
[124679.877278] [<ffffffffa83c8662>] ? __alloc_pages_nodemask+0x6a2/0xbe0
[124679.877947] [<ffffffffa83f3f3d>] handle_mm_fault+0x39d/0x9b0
[124679.878608] [<ffffffffa83e95a2>] __get_user_pages+0x1d2/0x7a0
[124679.879272] [<ffffffffa8416ba8>] ? alloc_pages_current+0x98/0x110
[124679.879936] [<ffffffffa83ea022>] get_user_pages+0x52/0x60
[124679.880604] [<ffffffffc021b32b>] dl_get_user_pages+0xab/0xf0 [blackmagic]
[124679.881286] [<ffffffffc02212da>] _ZN18IOMemoryDescriptor7prepareEv+0x3e/0x50 [blackmagic]
[124679.881986] [<ffffffffc02556a0>] _ZN26DaisyCutterDMASupportClass35newVideoFrameDMATransactionNotGatedEP21BlackmagicDMAParamRecP30BlackmagicMainMemoryDescriptorP+0x31a/0x974 [blackmagic]
[124679.883423] [<ffffffffc0238d9f>] ? _ZN32HardwareAbstractionDeckLinkClass13BitTestEndianEPVjj+0x11/0x18 [blackmagic]
[124679.884193] [<ffffffffc021beca>] ? dl_gate_unlock+0x2a/0x30 [blackmagic]
[124679.884969] [<ffffffffc0221bde>] ? _ZN24IOBufferMemoryDescriptor4freeEv+0x40/0x40 [blackmagic]
[124679.885766] [<ffffffffc024d4d9>] _ZN22VideoInputSupportClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRec+0x1a1/0x1d2 [blackmagic]
[124679.887384] [<ffffffffc025bff7>] _ZN15UserClientClass25provideFrameForVideoInputEP34BlackmagicVideoInputFrameParamsRecj+0x1b/0x24 [blackmagic]
[124679.889103] [<ffffffffc021d6b4>] blackmagic_ioctl_private+0x1466/0x3ef8 [blackmagic]
[124679.889997] [<ffffffffc0219fd9>] blackmagic_ioctl+0x49/0x70 [blackmagic]
[124679.890883] [<ffffffffa845fbc0>] do_vfs_ioctl+0x3a0/0x5a0
[124679.891768] [<ffffffffa8988678>] ? __do_page_fault+0x238/0x500
[124679.892639] [<ffffffffa845fe61>] SyS_ioctl+0xa1/0xc0
[124679.893501] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.894361] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.895213] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.896077] INFO: task find:21716 blocked for more than 120 seconds.
[124679.896934] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.897805] find D ffff9e981f4dac80 0 21716 21715 0x00000080
[124679.898679] Call Trace:
[124679.899545] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.900421] [<ffffffffa8982415>] rwsem_down_read_failed+0x105/0x1c0
[124679.901315] [<ffffffffc0870de0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.902203] [<ffffffffa8592d98>] call_rwsem_down_read_failed+0x18/0x30
[124679.903090] [<ffffffffa897fd30>] down_read+0x20/0x40
[124679.903985] [<ffffffffc0870d69>] xfs_ilock+0xd9/0x120 [xfs]
[124679.904877] [<ffffffffc0870de0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.905772] [<ffffffffc08628c0>] xfs_dir_open+0x40/0x80 [xfs]
[124679.906649] [<ffffffffa8448136>] do_dentry_open+0x216/0x2c0
[124679.907535] [<ffffffffa85042b2>] ? security_inode_permission+0x22/0x30
[124679.908438] [<ffffffffc0862880>] ? xfs_file_llseek+0xa0/0xa0 [xfs]
[124679.909330] [<ffffffffa844827a>] vfs_open+0x5a/0xb0
[124679.910216] [<ffffffffa84568c3>] ? may_open+0xa3/0x120
[124679.911101] [<ffffffffa8458fa6>] do_last+0x1f6/0x1290
[124679.911987] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.912870] [<ffffffffa845d682>] ? user_path_at_empty+0x72/0xc0
[124679.913754] [<ffffffffa84473fa>] ? __check_object_size+0x1ca/0x250
[124679.914634] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.915516] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.916394] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.917267] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.918145] [<ffffffffa8449834>] SyS_openat+0x14/0x20
[124679.919020] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.919898] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
[124679.920778] INFO: task find:21734 blocked for more than 120 seconds.
[124679.921652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[124679.922539] find D ffff9e981f4dac80 0 21734 21733 0x00000080
[124679.923424] Call Trace:
[124679.924271] [<ffffffffa8980a29>] schedule+0x29/0x70
[124679.925108] [<ffffffffa8982415>] rwsem_down_read_failed+0x105/0x1c0
[124679.925947] [<ffffffffc0870de0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.926772] [<ffffffffa8592d98>] call_rwsem_down_read_failed+0x18/0x30
[124679.927576] [<ffffffffa897fd30>] down_read+0x20/0x40
[124679.928366] [<ffffffffc0870d69>] xfs_ilock+0xd9/0x120 [xfs]
[124679.929132] [<ffffffffc0870de0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
[124679.929878] [<ffffffffc08628c0>] xfs_dir_open+0x40/0x80 [xfs]
[124679.930591] [<ffffffffa8448136>] do_dentry_open+0x216/0x2c0
[124679.931308] [<ffffffffa85042b2>] ? security_inode_permission+0x22/0x30
[124679.932037] [<ffffffffc0862880>] ? xfs_file_llseek+0xa0/0xa0 [xfs]
[124679.932756] [<ffffffffa844827a>] vfs_open+0x5a/0xb0
[124679.933469] [<ffffffffa84568c3>] ? may_open+0xa3/0x120
[124679.934181] [<ffffffffa8458fa6>] do_last+0x1f6/0x1290
[124679.934890] [<ffffffffa845be3d>] path_openat+0xcd/0x5a0
[124679.935592] [<ffffffffa845d682>] ? user_path_at_empty+0x72/0xc0
[124679.936293] [<ffffffffa84473fa>] ? __check_object_size+0x1ca/0x250
[124679.936986] [<ffffffffa845d7ad>] do_filp_open+0x4d/0xb0
[124679.937661] [<ffffffffa846b287>] ? __alloc_fd+0x47/0x170
[124679.938329] [<ffffffffa8449704>] do_sys_open+0x124/0x220
[124679.938995] [<ffffffffa898de15>] ? system_call_after_swapgs+0xa2/0x146
[124679.939661] [<ffffffffa8449834>] SyS_openat+0x14/0x20
[124679.940324] [<ffffffffa898dede>] system_call_fastpath+0x25/0x2a
[124679.940989] [<ffffffffa898de21>] ? system_call_after_swapgs+0xae/0x146
caspy

caspy

2020-02-19 07:35

reporter   ~0036317

Reproducibility is from several hours till few weeks. under the same load every time.
.edu

.edu

2020-03-19 21:20

reporter   ~0036538

We are affected too, the latest CentOS/RedHat kernel (3.10.0-1062.....el7.x86_64),
happened on multiple mission-critical servers with XFS, kernel updates don't help,
same as 3 other reporters above.
We had to install (and currently use) the debug(!) kernel as a W/A as it prevents all
XFS partitions from being marked read-only. But we are genuinely surprised to see
such a critical issue remains unnoticed, as XFS is now default since CentOS7/RHEL7
(and Centos7 is maintained/supported until 30 June 2024).
Has it been escalated to maybe RedHat? Is there a patch?
kudeyda

kudeyda

2020-03-20 11:38

reporter  

log (11 bytes)
kernel log
log (11 bytes)

Issue History

Date Modified Username Field Change
2019-11-09 08:44 bruceleeeee New Issue
2019-11-09 08:44 bruceleeeee Tag Attached: 3.10.0-1062.1.1.el7.x86_64
2019-11-09 08:44 bruceleeeee Tag Attached: 7.7
2019-11-09 08:44 bruceleeeee Tag Attached: centos 7
2019-11-09 08:44 bruceleeeee Tag Attached: file system
2020-01-10 13:50 maf01 Note Added: 0035998
2020-01-14 00:05 slayerduck File Added: messages.txt
2020-01-14 00:05 slayerduck Note Added: 0036017
2020-02-19 07:30 caspy Note Added: 0036316
2020-02-19 07:35 caspy Note Added: 0036317
2020-03-19 21:20 .edu Note Added: 0036538
2020-03-20 11:38 kudeyda File Added: log