View Issue Details

IDProjectCategoryView StatusLast Update
0016960CentOS-7kernelpublic2020-05-11 10:06
Reporterovapostu 
PrioritynormalSeveritymajorReproducibilitysometimes
Status closedResolutionno change required 
Product Version7.7-1908 
Target VersionFixed in Version 
Summary0016960: xfs writeback error on kernel-3.10.0-1062
DescriptionAfter upgrading CentOS from 7.6-1810 to 7.7-1908 we started getting file corruption problems which were traced back to xfs write errors reported in /var/log/messages
The CentOS is hosting a PostgreSQL server with frequent writes and once every 1-2 days writes fail because of the xfs writeback error.

Downgrading the kernel from 3.10.0-1062 back to 3.10.0-957 seems to solve the problem

the error logs show

Dec 14 17:59:30 localhost kernel: sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 14 17:59:30 localhost kernel: sd 2:0:0:0: [sda] Sense Key : Illegal Request [current]
Dec 14 17:59:30 localhost kernel: sd 2:0:0:0: [sda] Add. Sense: Invalid command operation code
Dec 14 17:59:30 localhost kernel: sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 12 aa 82 68 00 00 10 00
Dec 14 17:59:30 localhost kernel: blk_update_request: critical target error, dev sda, sector 313164392
Dec 14 17:59:30 localhost kernel: XFS (dm-0): writeback error on sector 312136296

The system is running in a virtual environment hosted on VMWare ESXi and has a single 500 GB virtual disk attached to a virtual SCSI controller
Tagsfile system, kernel, xfs
abrt_hash
URL

Activities

TrevorH

TrevorH

2020-01-22 10:01

manager   ~0036089

There is at least one xfs fix in the newer kernels that have come out since 7.7 went GA. Worth updating to kernel-3.10.0-1062.9.1.el7.x86_64 to see if the problem is already resolved. Otherwise you need to report this on bugzilla.redhat.com too as CentOS only rebuilds the source used to build RHEL and fixes need to come from them.
ovapostu

ovapostu

2020-01-22 10:29

reporter   ~0036090

Thanks Trevor for the quick reply and explanation.
we can certainly give 3.10.0-1062.9.1 a try. I'm not sure where I can see the fixes that are included with each release
TrevorH

TrevorH

2020-01-22 10:33

manager   ~0036091

rpm -q --changelog kernel-3.10.0-1062.9.1.el7.x86_64 | less

(or repoquery if it's not yet installed)
CovertLeopard

CovertLeopard

2020-03-10 23:01

reporter   ~0036487

What symptoms were you seeing when this happened? We have a rather large MySQL database stored on a xfs network share and recently (on 3/4) upgraded to this kernel. Shortly after, we started seeing long periods of CPU getting stuck at 100%. I'm curious if that is similar to the symptoms you saw in your postgreSQL instance?
ovapostu

ovapostu

2020-04-01 13:31

reporter   ~0036636

Hi, CPU usage was normal in our case.

Just for closure this turned out to be a problem on the hypervisor side of things, we also saw the same problem on a overcrowded VMWare esxi server where at peak times disk latency would increase in the order of hundreds of milliseconds. this in turn generated SCSI errors on the CentOS host en eventually write failures.

so to conclude, this is not an issue in the kernel
Sysadmin CAOS

Sysadmin CAOS

2020-05-11 09:56

reporter   ~0036913

Hello,

My server is running kernel 3.10.0-1062.12.1 in a CentOS Linux release 7.7.1908. Since some weeks ago, server is restarting after XFS errors. I have read this forum and I would like to know if newest kernel version (for a CentOS-7) 3.10.0-1127 solves this problem about XFS erros and automatic system reboots.

Thanks.
ManuelWolfshant

ManuelWolfshant

2020-05-11 10:05

manager   ~0036914

Sysadmin CAOS: This is neither a forum nor a support avenue. If you need help, please use one of the recommend channels ( IRC, mailing list, forum ). As a sidenote, CentOS only offers support for the latest releases, so using 3.10.0-1062.12.1 is not supported. And if have XFS errors, you'd better find and fix the cause for them rather than looking at the reboot part.
ManuelWolfshant

ManuelWolfshant

2020-05-11 10:06

manager   ~0036915

closing as requested by the person who opened the bug

Issue History

Date Modified Username Field Change
2020-01-22 09:52 ovapostu New Issue
2020-01-22 09:52 ovapostu Tag Attached: file system
2020-01-22 09:52 ovapostu Tag Attached: kernel
2020-01-22 09:52 ovapostu Tag Attached: xfs
2020-01-22 10:01 TrevorH Note Added: 0036089
2020-01-22 10:29 ovapostu Note Added: 0036090
2020-01-22 10:33 TrevorH Note Added: 0036091
2020-03-10 23:01 CovertLeopard Note Added: 0036487
2020-04-01 13:31 ovapostu Note Added: 0036636
2020-05-11 09:56 Sysadmin CAOS Note Added: 0036913
2020-05-11 10:05 ManuelWolfshant Note Added: 0036914
2020-05-11 10:06 ManuelWolfshant Status new => closed
2020-05-11 10:06 ManuelWolfshant Resolution open => no change required
2020-05-11 10:06 ManuelWolfshant Note Added: 0036915