View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0018433 | CentOS-7 | kernel | public | 2022-04-15 02:43 | 2022-06-09 01:25 |
Reporter | hejiwen | Assigned To | |||
Priority | high | Severity | major | Reproducibility | always |
Status | closed | Resolution | no change required | ||
Platform | intel | ||||
Product Version | 7.9.2009 | ||||
Summary | 0018433: sync process enters uninterrupted state | ||||
Description | kernel-3.10.0-1160.62.1 | ||||
Steps To Reproduce | 1. create many sparese file,size=500G 2. use fio to write this files 3. when disk to fault sector,the process enter uninterrupted state, if exec sync command,sync also enter uninterrupted state,call strace as follows: | ||||
Tags | No tags attached. | ||||
abrt_hash | |||||
URL | |||||
Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 Sense Key : Hardware Error [current] Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 Add. Sense: Mechanical positioning error Apr 15 02:34:53 inspur01 kernel: sd 14:0:6:0: [sdd] tag#11 CDB: Write(10) 2a 00 08 98 b4 f0 00 02 00 00 Apr 15 02:34:53 inspur01 kernel: blk_update_request: I/O error, dev sdd, sector 144225934 Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716382, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716383, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716384, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716385, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716386, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716387, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716388, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716389, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716390, lost async page write Apr 15 02:34:53 inspur01 kernel: Buffer I/O error on dev sdd1, logical block 16716391, lost async page write Apr 15 02:37:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds. Apr 15 02:37:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:37:45 inspur01 kernel: <NVDIMM> Read ADR status from CMOS:0x00[jedec_nvdimm_init][831] Apr 15 02:37:45 inspur01 kernel: filestore_sync D Apr 15 02:37:45 inspur01 kernel: ffff882034f51fa0 0 8998 1 0x00000080 Apr 15 02:37:45 inspur01 kernel: Call Trace: Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:37:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:37:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds. Apr 15 02:37:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:37:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080 Apr 15 02:37:45 inspur01 kernel: Call Trace: Apr 15 02:37:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs] Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:37:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:37:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:37:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:39:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds. Apr 15 02:39:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:39:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000080 Apr 15 02:39:45 inspur01 kernel: Call Trace: Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:39:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:39:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds. Apr 15 02:39:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:39:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080 Apr 15 02:39:45 inspur01 kernel: Call Trace: Apr 15 02:39:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs] Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:39:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:39:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:39:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:41:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds. Apr 15 02:41:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:41:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000080 Apr 15 02:41:45 inspur01 kernel: Call Trace: Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:41:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:41:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds. Apr 15 02:41:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:41:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080 Apr 15 02:41:45 inspur01 kernel: Call Trace: Apr 15 02:41:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs] Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:41:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:41:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:41:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:43:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds. Apr 15 02:43:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:43:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000080 Apr 15 02:43:45 inspur01 kernel: Call Trace: Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:43:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:43:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds. Apr 15 02:43:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:43:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000080 Apr 15 02:43:45 inspur01 kernel: Call Trace: Apr 15 02:43:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs] Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:43:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:43:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:43:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:45:45 inspur01 kernel: INFO: task filestore_sync:8998 blocked for more than 120 seconds. Apr 15 02:45:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:45:45 inspur01 kernel: filestore_sync D ffff882034f51fa0 0 8998 1 0x00000084 Apr 15 02:45:45 inspur01 kernel: Call Trace: Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810d546e>] ? dequeue_task_fair+0x41e/0x660 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810ad74a>] ? __queue_delayed_work+0xaa/0x1a0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810adab1>] ? try_to_grab_pending+0xb1/0x160 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8134a43d>] ? list_del+0xd/0x30 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bbbfd>] ? wait_for_completion+0x11d/0x140 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:45:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 Apr 15 02:45:45 inspur01 kernel: INFO: task journal_write:9216 blocked for more than 120 seconds. Apr 15 02:45:45 inspur01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 02:45:45 inspur01 kernel: journal_write D ffff88201bf38fd0 0 9216 1 0x00000084 Apr 15 02:45:45 inspur01 kernel: Call Trace: Apr 15 02:45:45 inspur01 kernel: [<ffffffffc08561f1>] ? xfs_inode_item_format_attr_fork.isra.7+0xb1/0x270 [xfs] Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bb829>] schedule+0x29/0x70 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b91e9>] schedule_timeout+0x239/0x2c0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff811449ef>] ? delayacct_end+0x8f/0xb0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810f03d2>] ? ktime_get_ts64+0x52/0xf0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9800>] ? bit_wait+0x50/0x50 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bad8d>] io_schedule_timeout+0xad/0x130 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816bae28>] io_schedule+0x18/0x20 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9811>] bit_wait_io+0x11/0x50 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816b9337>] __wait_on_bit+0x67/0x90 Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff810b8520>] ? wake_bit_function+0x40/0x40 Apr 15 02:45:45 inspur01 kernel: [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 Apr 15 02:45:45 inspur01 kernel: [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e4b6>] sync_filesystem+0x66/0xb0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff8123e664>] SyS_syncfs+0x54/0xa0 Apr 15 02:45:45 inspur01 kernel: [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 |
|
977061 root 20 0 108052 340 264 D 0.0 0.0 0:00.00 sync [root@localhost ~]# cat /proc/977061/stack [<ffffffff81189741>] wait_on_page_bit+0x81/0xa0 [<ffffffff81189871>] __filemap_fdatawait_range+0x111/0x190 [<ffffffff8118c6a7>] filemap_fdatawait_keep_errors+0x27/0x30 [<ffffffff81238bbd>] sync_inodes_sb+0x16d/0x1f0 [<ffffffff8123e269>] sync_inodes_one_sb+0x19/0x20 [<ffffffff8120f833>] iterate_supers+0xc3/0x120 [<ffffffff8123e544>] sys_sync+0x44/0xb0 [<ffffffff816c86db>] system_call_fastpath+0x22/0x27 [<ffffffffffffffff>] 0xffffffffffffffff |
|
must reboot to solve this problem | |
You have faulty hardware ( see the very first 10 lines of your logs ). Nothing that you have included later shows a loop, the system just can't recover past the defective disk | |
Thanks for your reply。 I update kernel to 4.18, this problem havn't happen,It can tolerate this error, But I can't update kernel in production environment。 I want to know how to find the patch fixed this problem,can you give me some advice? |
|
4.18 dmesg info | |
Even your last paste shows that you have a hardware error. If you believe that ignoring a hardware error is a proper fix, feel free to suggest it to Redhat via a bug opened at bugzilla.redhat.com. If and when RH accepts it and incorporates it into RHEL and releases a patched version, then CentOS will pick it up automatically. | |
Date Modified | Username | Field | Change |
---|---|---|---|
2022-04-15 02:43 | hejiwen | New Issue | |
2022-04-15 02:47 | hejiwen | Note Added: 0038904 | |
2022-04-15 02:47 | hejiwen | Note Added: 0038905 | |
2022-04-15 02:49 | hejiwen | Note Added: 0038906 | |
2022-04-15 06:42 | ManuelWolfshant | Note Added: 0038907 | |
2022-04-15 09:58 | hejiwen | Note Added: 0038908 | |
2022-04-15 10:25 | hejiwen | Note Added: 0038910 | |
2022-04-15 10:25 | hejiwen | File Added: 111.png | |
2022-06-09 01:25 | ManuelWolfshant | Status | new => closed |
2022-06-09 01:25 | ManuelWolfshant | Resolution | open => no change required |
2022-06-09 01:25 | ManuelWolfshant | Note Added: 0038942 |