View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0018433CentOS-7kernelpublic2022-04-15 02:432022-06-09 01:25
Reporterhejiwen Assigned To 
PriorityhighSeveritymajorReproducibilityalways
Status closedResolutionno change required 
Platformintel 
Product Version7.9.2009 
Summary0018433: sync process enters uninterrupted state
Descriptionkernel-3.10.0-1160.62.1



Steps To Reproduce1. create many sparese file，size=500G
2. use fio to write this files
3. when disk to fault sector，the process enter uninterrupted state， if exec sync command，sync also enter uninterrupted state，call strace as follows：
TagsNo tags attached.
abrt_hash
URL

Activities

hejiwen

hejiwen

2022-04-15 02:47

reporter   ~0038904

Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 Sense Key : Hardware Error [current]
Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 Add. Sense: Mechanical positioning error
Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 CDB: Write(10) 2a 00 08 98 b4 f0 00 02 00 00
Apr 15 02:34:53 inspur01 kernel: blk_update_request: I/O error, dev sdd, sector 144225934
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716382, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716383, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716384, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716385, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716386, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716387, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716388, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716389, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716390, lost async page write
Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716391, lost async page write
Apr 15 02:37:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds.
Apr 15 02:37:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:37:45 inspur01 kernel: <NVDIMM> Read ADR status from CMOS:0x00[jedec_nvdimm_init][831]
Apr 15 02:37:45 inspur01 kernel: filestore_sync D
Apr 15 02:37:45 inspur01 kernel: ffff882034f51fa0 0 8998 1 0x00000080
Apr 15 02:37:45 inspur01 kernel: Call Trace:
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:37:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:37:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds.
Apr 15 02:37:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:37:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080
Apr 15 02:37:45 inspur01 kernel: Call Trace:
Apr 15 02:37:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs]
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:37:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:37:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:39:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds.
Apr 15 02:39:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:39:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000080
Apr 15 02:39:45 inspur01 kernel: Call Trace:
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:39:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:39:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds.
Apr 15 02:39:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:39:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080
Apr 15 02:39:45 inspur01 kernel: Call Trace:
Apr 15 02:39:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs]
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:39:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:39:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:41:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds.
Apr 15 02:41:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:41:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000080
Apr 15 02:41:45 inspur01 kernel: Call Trace:
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:41:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:41:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds.
Apr 15 02:41:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:41:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080
Apr 15 02:41:45 inspur01 kernel: Call Trace:
Apr 15 02:41:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs]
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:41:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:41:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:43:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds.
Apr 15 02:43:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:43:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000080
Apr 15 02:43:45 inspur01 kernel: Call Trace:
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:43:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:43:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds.
Apr 15 02:43:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:43:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080
Apr 15 02:43:45 inspur01 kernel: Call Trace:
Apr 15 02:43:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs]
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:43:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:43:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:45:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds.
Apr 15 02:45:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:45:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000084
Apr 15 02:45:45 inspur01 kernel: Call Trace:
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:45:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
Apr 15 02:45:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds.
Apr 15 02:45:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 02:45:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000084
Apr 15 02:45:45 inspur01 kernel: Call Trace:
Apr 15 02:45:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs]
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90
Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40
Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
Apr 15 02:45:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0
Apr 15 02:45:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27
hejiwen

hejiwen

2022-04-15 02:47

reporter   ~0038905

977061 root 20 0 108052 340 264 D 0.0 0.0 0:00.00 sync
[root@localhost ~]# cat /proc/977061/stack
[<ffffffff81189741>] wait_on_page_bit+0x81/0xa0
[<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190
[<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30
[<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0
[<ffffffff8123e269>] sync_inodes_one_sb+0x19/0x20
[<ffffffff8120f833>] iterate_supers+0xc3/0x120
[<ffffffff8123e544>] sys_sync+0x44/0xb0
[<ffffffff816c86db>] system_call_fastpath+0x22/0x27
[<ffffffffffffffff>] 0xffffffffffffffff
hejiwen

hejiwen

2022-04-15 02:49

reporter   ~0038906

must reboot to solve this problem
ManuelWolfshant

ManuelWolfshant

2022-04-15 06:42

manager   ~0038907

You have faulty hardware ( see the very first 10 lines of your logs ). Nothing that you have included later shows a loop, the system just can't recover past the defective disk
hejiwen

hejiwen

2022-04-15 09:58

reporter   ~0038908

Thanks for your reply。
I update kernel to 4.18， this problem havn't happen，It can tolerate this error, But I can't update kernel in production environment。
I want to know how to find the patch fixed this problem，can you give me some advice？
hejiwen

hejiwen

2022-04-15 10:25

reporter   ~0038910

4.18 dmesg info
111.png (53,528 bytes)   
111.png (53,528 bytes)   
ManuelWolfshant

ManuelWolfshant

2022-06-09 01:25

manager   ~0038942

Even your last paste shows that you have a hardware error. If you believe that ignoring a hardware error is a proper fix, feel free to suggest it to Redhat via a bug opened at bugzilla.redhat.com. If and when RH accepts it and incorporates it into RHEL and releases a patched version, then CentOS will pick it up automatically.

Issue History

Date Modified Username Field Change
2022-04-15 02:43 hejiwen New Issue
2022-04-15 02:47 hejiwen Note Added: 0038904
2022-04-15 02:47 hejiwen Note Added: 0038905
2022-04-15 02:49 hejiwen Note Added: 0038906
2022-04-15 06:42 ManuelWolfshant Note Added: 0038907
2022-04-15 09:58 hejiwen Note Added: 0038908
2022-04-15 10:25 hejiwen Note Added: 0038910
2022-04-15 10:25 hejiwen File Added: 111.png
2022-06-09 01:25 ManuelWolfshant Status new => closed
2022-06-09 01:25 ManuelWolfshant Resolution open => no change required
2022-06-09 01:25 ManuelWolfshant Note Added: 0038942