View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0006087CentOS-6kernelpublic2012-11-24 10:422013-04-23 21:14
Reporteribox Assigned To 
PriorityurgentSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
OSCentos 6OS Version2.6.32-279.14.1 
Product Version6.3 
Summary0006087: xfsaild always in D state
DescriptionHi, after kernel upgrade to 2.6.32-279.14.1.el6.x86_64 xfsaild is always in D state, what reproduce wrong/high load average in top/uptime commands:
1168 root 20 0 0 0 0 D 0.0 0.0 0:50.68 xfsaild/dm-11
1174 root 20 0 0 0 0 D 0.0 0.0 0:30.28 xfsaild/dm-7
1177 root 20 0 0 0 0 D 0.0 0.0 0:39.18 xfsaild/dm-8
1180 root 20 0 0 0 0 D 0.0 0.0 0:38.64 xfsaild/dm-9

And load average then for 4 processes load average: 4.45, 4.21, 4.15.
Vmstat, iostat shows all is normal.
It's not totally clear to me whether there is actually a problem, or
whether it's safe to ignore the higher load until there's a patch?
TagsNo tags attached.

Activities

avasilenko

avasilenko

2012-11-28 11:55

reporter   ~0016088

Got the same problem. Have same setup on 2 servers, but only 1 of them has this problem.
fbh

fbh

2012-11-30 08:28

reporter   ~0016099

I am seeing this as well.

root 855 0.1 0.0 0 0 ? D 08:37 0:04 [xfsaild/dm-2]

It's a freshly installed 6.3 with all updates applied running under VMWare ESXi.
tru

tru

2012-11-30 08:56

administrator   ~0016100

I am seeing it too, afaik it's a side effect of https://bugzilla.redhat.com/show_bug.cgi?id=860787
torel@dolphingeo.com

torel@dolphingeo.com

2012-12-20 03:47

reporter   ~0016163

cc
noloop

noloop

2013-01-04 22:09

reporter   ~0016238

I got this problem with all CentOS installations with 2.6.32-279.14.1.el6.x86_64 kernel version. Per mounted partition you get an extra load of 1.
noloop

noloop

2013-01-04 22:39

reporter   ~0016239

The same with 2.6.32-279.19.1.el6.x86_64 kernel version.
ibox

ibox

2013-01-19 20:16

reporter   ~0016302

upgraded to third kernel, problem solved :)
toracat

toracat

2013-01-20 03:59

manager   ~0016303

What is a "third kernel"?

According to this upstream BZ:

https://bugzilla.redhat.com/show_bug.cgi?id=883905

there is a proposed fix but planned for EL 6.5 (not a typo).
ibox

ibox

2013-01-20 04:55

reporter   ~0016304

"Third kernel" (last version 3.0.59-1.el6.elrepo) is actually kernel-lt.
More information on http://elrepo.org/tiki/kernel-lt
In my opinion this works much better than 2.6.32
toracat

toracat

2013-01-20 09:03

manager   ~0016305

ELRepo's kernel-lt uses a stable kernel line 3.0.x from kernel.org. Brian Foster of Red Hat submitted a patch that fixes the bug to kernel.org and it is now in the stable/mainline kernels.

Using ELRepo's kernel is fine but may not be well justified in this particular case because this bug is just not "serious enough".

According to BZ 883905 (see the link above):

"The current consensus for this bug is that the fix is fine, but the appropriate target is rhel6.5 and forward, as this is primarily a reporting issue."

More details here:

http://oss.sgi.com/archives/xfs/2012-11/msg00594.html

"There is no side effect other than the load. There are not performance issues with the ailds behaving like this."
ibox

ibox

2013-01-20 09:39

reporter   ~0016306

i forgot to say, that we had also the problem with ksoftirqd to on servers with AMD processors, that was the real reason to upgrade to 3.0.x based kernel.
toracat

toracat

2013-01-20 17:40

manager   ~0016307

@ibox

If the ksoftirqd issue is the one being handled in bug #5813, please add your case there as well as to the upstream BZ ( https://bugzilla.redhat.com/show_bug.cgi?id=870573 ).
toracat

toracat

2013-03-08 16:11

manager   ~0016622

The patch offered in https://bugzilla.redhat.com/show_bug.cgi?id=883905 cleanly applies to the CentOS 6.4 kernel (but that was not the case with the 6.3 kernel). Will try to include the fix in the next release of centosplus kernel.
toracat

toracat

2013-03-08 16:17

manager  

xfs-bug6087.patch (263 bytes)    
--- xfs_trans_ail.c	2013-02-20 07:27:37.000000000 -0800
+++ xfs_trans_ail.c.new	2013-03-08 08:15:13.078745782 -0800
@@ -405,6 +405,7 @@
 		xfs_trans_ail_cursor_done(ailp, cur);
 		spin_unlock(&ailp->xa_lock);
 		*last_lsn = 0;
+		tout = 50;
 		return tout;
 	}
xfs-bug6087.patch (263 bytes)   
toracat

toracat

2013-03-08 16:18

manager   ~0016624

Patch file uploaded: http://bugs.centos.org/file_download.php?file_id=1445&type=bug
toracat

toracat

2013-03-12 20:09

manager   ~0016681

kernel-2.6.32-358.2.1.el6 was released upstream. The patch applied to the cplus kernel.
toracat

toracat

2013-04-23 21:13

manager   ~0017272

Fixed in the distro kernel-2.6.32-358.6.1.el6.

Issue History

Date Modified Username Field Change
2012-11-24 10:42 ibox New Issue
2012-11-28 11:55 avasilenko Note Added: 0016088
2012-11-30 08:28 fbh Note Added: 0016099
2012-11-30 08:56 tru Note Added: 0016100
2012-11-30 08:56 tru Status new => confirmed
2012-12-20 03:47 torel@dolphingeo.com Note Added: 0016163
2013-01-04 22:09 noloop Note Added: 0016238
2013-01-04 22:39 noloop Note Added: 0016239
2013-01-19 20:16 ibox Note Added: 0016302
2013-01-20 03:59 toracat Note Added: 0016303
2013-01-20 04:55 ibox Note Added: 0016304
2013-01-20 09:03 toracat Note Added: 0016305
2013-01-20 09:05 toracat Category -OTHER => kernel
2013-01-20 09:39 ibox Note Added: 0016306
2013-01-20 17:40 toracat Note Added: 0016307
2013-03-08 16:11 toracat Note Added: 0016622
2013-03-08 16:17 toracat File Added: xfs-bug6087.patch
2013-03-08 16:18 toracat Note Added: 0016624
2013-03-12 20:09 toracat Note Added: 0016681
2013-03-12 20:11 toracat Status confirmed => resolved
2013-03-12 20:11 toracat Resolution open => fixed
2013-04-23 21:13 toracat Note Added: 0017272
2013-04-23 21:13 toracat Status resolved => feedback
2013-04-23 21:13 toracat Resolution fixed => reopened
2013-04-23 21:14 toracat Status feedback => resolved
2013-04-23 21:14 toracat Resolution reopened => fixed