View Issue Details

IDProjectCategoryView StatusLast Update
0008803CentOS-7kernelpublic2015-10-05 15:17
Reporteramuraru 
PriorityhighSeveritymajorReproducibilitysometimes
Status resolvedResolutionfixed 
Product Version 
Target VersionFixed in Version 
Summary0008803: Backport: futex: Ensure get_futex_key_refs() always implies a barrier
DescriptionAccording to this report:
https://groups.google.com/forum/#!msg/mechanical-sympathy/QbmpZxp6C64/nMhNjQPTeLEJ
Centos 7.X is affected by this bug

"
To be clear: the bug is not limited to kernels labeled "3.14" thru "3.18". It appears in several production kernels with "earlier" labels that are part s of various distros, and got there through backporting efforts. E.g. RHEL 6.6 uses 2.6.32.xxxxx kernels, and RHEL 7.1 uses 3.10.0.xxx kernels, and both of those have had the bug introduced through backporting of changes from 3.14. "3.0.101 SLES 11" is not the same as 3.0.101. My first suspicion would be that this SLES kernel has had the bug back ported to it. E.g. we originally ran into the bug in RHEL 6.6 with a 2.6.32.504 kernel that included a back port of the buggy update. The fix back port appeared in the 2.6.32.504.16.2 kernel in RHEL 6.6.z.

You should get the specific sources for your 3.0.101 SLES 11 kernel and look at kernel/futex.c. Specifically, you want to see if the 3.14 update (in https://github.com/torvalds/linux/commit/b0c29f79ecea0b6fbcefc999e70f2843ae8306db ) which introduced the bug was backported to it, but the 3.18 fix to that (in https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85beb03c75121419a7dec52f0 ) hadn't been applied on top of that. [See more detailed discussion in the original post on this thread].
"

Can we get the https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85beb03c75121419a7dec52f0 backported in Centos7 kernel?
Additional Informationhttps://groups.google.com/forum/#!msg/mechanical-sympathy/QbmpZxp6C64/nMhNjQPTeLEJ
TagsNo tags attached.
abrt_hash
URLhttps://groups.google.com/forum/#!msg/mechanical-sympathy/QbmpZxp6C64/nMhNjQPTeLEJ

Activities

amuraru

amuraru

2015-05-28 17:27

reporter  

0001-PATCH-Backport-futex-Ensure-get_futex_key_refs-alway.patch (3,354 bytes)
From 7e12c23ade1232022bb4910ba2a041e3446ddc66 Mon Sep 17 00:00:00 2001
From: Adrian Muraru <amuraru@adobe.com>
Date: Thu, 28 May 2015 20:25:54 +0300
Subject: [PATCH] [PATCH] Backport futex: Ensure get_futex_key_refs() always
 implies a barrier

---
 SOURCES/futex-get_futex_key_refs-76835b.patch | 52 +++++++++++++++++++++++++++
 SPECS/kernel.spec                             |  1 +
 2 files changed, 53 insertions(+)
 create mode 100644 SOURCES/futex-get_futex_key_refs-76835b.patch

diff --git a/SOURCES/futex-get_futex_key_refs-76835b.patch b/SOURCES/futex-get_futex_key_refs-76835b.patch
new file mode 100644
index 0000000..97c926e
--- /dev/null
+++ b/SOURCES/futex-get_futex_key_refs-76835b.patch
@@ -0,0 +1,52 @@
+From 76835b0ebf8a7fe85beb03c75121419a7dec52f0 Mon Sep 17 00:00:00 2001
+From: Catalin Marinas <catalin.marinas@arm.com>
+Date: Fri, 17 Oct 2014 17:38:49 +0100
+Subject: [PATCH] futex: Ensure get_futex_key_refs() always implies a barrier
+
+Commit b0c29f79ecea (futexes: Avoid taking the hb->lock if there's
+nothing to wake up) changes the futex code to avoid taking a lock when
+there are no waiters. This code has been subsequently fixed in commit
+11d4616bd07f (futex: revert back to the explicit waiter counting code).
+Both the original commit and the fix-up rely on get_futex_key_refs() to
+always imply a barrier.
+
+However, for private futexes, none of the cases in the switch statement
+of get_futex_key_refs() would be hit and the function completes without
+a memory barrier as required before checking the "waiters" in
+futex_wake() -> hb_waiters_pending(). The consequence is a race with a
+thread waiting on a futex on another CPU, allowing the waker thread to
+read "waiters == 0" while the waiter thread to have read "futex_val ==
+locked" (in kernel).
+
+Without this fix, the problem (user space deadlocks) can be seen with
+Android bionic's mutex implementation on an arm64 multi-cluster system.
+
+Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
+Reported-by: Matteo Franchin <Matteo.Franchin@arm.com>
+Fixes: b0c29f79ecea (futexes: Avoid taking the hb->lock if there's nothing to wake up)
+Acked-by: Davidlohr Bueso <dave@stgolabs.net>
+Tested-by: Mike Galbraith <umgwanakikbuti@gmail.com>
+Cc: <stable@vger.kernel.org>
+Cc: Darren Hart <dvhart@linux.intel.com>
+Cc: Thomas Gleixner <tglx@linutronix.de>
+Cc: Peter Zijlstra <peterz@infradead.org>
+Cc: Ingo Molnar <mingo@kernel.org>
+Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
+Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
+---
+ kernel/futex.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/kernel/futex.c b/kernel/futex.c
+index 815d7af..f3a3a07 100644
+--- a/kernel/futex.c
++++ b/kernel/futex.c
+@@ -343,6 +343,8 @@ static void get_futex_key_refs(union futex_key *key)
+ 	case FUT_OFF_MMSHARED:
+ 		futex_get_mm(key); /* implies MB (B) */
+ 		break;
++	default:
++		smp_mb(); /* explicit MB (B) */
+ 	}
+ }
+ 
diff --git a/SPECS/kernel.spec b/SPECS/kernel.spec
index 9e2edfd..a5af999 100644
--- a/SPECS/kernel.spec
+++ b/SPECS/kernel.spec
@@ -373,6 +373,7 @@ Patch999999: linux-kernel-test.patch
 Patch1000: debrand-single-cpu.patch
 Patch1001: debrand-rh_taint.patch
 Patch1002: debrand-rh-i686-cpu.patch
+Patch1003: futex-get_futex_key_refs-76835b.patch
 
 BuildRoot: %{_tmppath}/kernel-%{KVRA}-root
 
-- 
2.3.5

amuraru

amuraru

2015-05-28 17:43

reporter   ~0023236

Patch sent to centos-devel:
http://lists.centos.org/pipermail/centos-devel/2015-May/013399.html
toracat

toracat

2015-05-28 17:59

manager   ~0023237

The distro kernel cannot be patched because it is a 'bug-for-bug' rebuild of the upstream (RHEL) kernel. What CentOS can offer is to apply the patch to the centosplus kernel.

You'd need to report this at http://bugzilla.redhat.com to get the RHEL kernel fixed. CentOS will then inherit it.
toracat

toracat

2015-05-28 19:13

manager   ~0023239

See also https://access.redhat.com/solutions/1350963 (only RHEL-6 is referenced).
toracat

toracat

2015-05-28 19:29

manager  

futex.patch (516 bytes)
--- a/kernel/futex.c	2015-01-29 15:15:53.000000000 -0800
+++ b/kernel/futex.c	2015-05-28 12:25:44.187442096 -0700
@@ -327,6 +327,13 @@ static void get_futex_key_refs(union fut
 	case FUT_OFF_MMSHARED:
 		futex_get_mm(key); /* implies MB (B) */
 		break;
+	default:
+		/*
+		 * Private futexes do not hold reference on an inode or
+		 * mm, therefore the only purpose of calling get_futex_key_refs
+		 * is because we need the barrier for the lockless waiter check.
+		 */
+		smp_mb(); /* explicit MB (B) */
 	}
 }
 
futex.patch (516 bytes)
amuraru

amuraru

2015-05-29 09:02

reporter   ~0023244

Thanks toracat, filed a bug on RH: https://bugzilla.redhat.com/show_bug.cgi?id=1226222
johnds

johnds

2015-06-03 18:01

reporter   ~0023284

Has this patch been included in some part of the Centos 7 distribution?
We need it for an OS upgrade next week.
toracat

toracat

2015-06-03 19:34

manager   ~0023285

@johnds

Not yet. The patch will be in the next update release of CentOSPlus kernel (kernel-plus.el7).
toracat

toracat

2015-06-03 19:36

manager   ~0023286

@amuraru

Because the BZ you filed is private, could you update the status with any progress you see there?
johnds

johnds

2015-06-03 19:57

reporter   ~0023287

@toracat

When is the next update scheduled to be released?

We are seriuosly hit by this bug.
toracat

toracat

2015-06-03 20:51

manager   ~0023288

@johnds

This is entirely up to upstream (Red Hat). The last 2 updates were on Mar 27 and May 13. So the next one could be sometime this month.

I can provide a non-official [unsigned] version of a patched kernel-plus if you'd like.
amuraru

amuraru

2015-06-03 20:53

reporter   ~0023289

No activity on BZ1226222
johnds

johnds

2015-06-03 21:16

reporter   ~0023290

@toracat
That would be nice to have.
toracat

toracat

2015-06-03 21:34

manager   ~0023291

@johnds

Will work on it.
toracat

toracat

2015-06-04 04:46

manager   ~0023293

@johnds

You can download the patched centosplus kernel from:

http://people.centos.org/toracat/kernel/7/plus/bug8803/

Please test. As noted before, the packages are not official and not signed.
johnds

johnds

2015-06-04 14:50

reporter   ~0023302

@toracat

Got the files. Great service. Thanks.
johnds

johnds

2015-06-05 01:11

reporter   ~0023311

@amuraru

Have you given the BZ1226222 high priority?

The bug is a real show stopper in a HPC cluster of nodes with a high core count.
In our case 28 core nodes with programs with the number of threads ranging between 14 and 28. The bug hits many different popular bioinformatics applications like bwa, NCBI's blast and java based programs.
amuraru

amuraru

2015-06-05 14:45

reporter   ~0023318

@johnds - I have no affiliation with RH so I'm pretty much your situation at the RH mercy.
You can create (a free) account on https://bugzilla.redhat.com and put extra pressure on them?
amuraru

amuraru

2015-06-05 15:02

reporter   ~0023319

@tocarat - IIUC, CentosPLus kernel release is entirely up to Centos community.
Are you going to release an "official" centos-plus kernel to include this fix?
toracat

toracat

2015-06-05 15:37

manager   ~0023320

@amuraru

We usually add new patches to the next update release. There is no plan to publish an "official" version of the current centosplusplus kernel with the patch. But if this problem is so serious, we could make an exception.
johnds

johnds

2015-06-06 14:36

reporter   ~0023331

@amuraru

I was unable to access BZ1226222 so I crated a high priority duplicate Bug 1228879 referring back to this bug report and BZ1226222.
johnds

johnds

2015-06-12 08:10

reporter   ~0023385

@toracat

Just to let you know that we have now upgraded the systems to Centos7 using the patched Centos Plus kernel and we are not seeing the futex_wait_queue_me problem.
toracat

toracat

2015-06-12 13:08

manager   ~0023387

@johnds

Thanks for reporting back with success. By the way is there any progress with your BZ 1228879?
johnds

johnds

2015-06-12 14:02

reporter   ~0023388

@toracat

Nothing has been added to the BZ 1228879. It has been assigned to a named person, though. Maybe the problem is being tracked in BZ1226222.
amuraru

amuraru

2015-06-14 11:51

reporter   ~0023397

BZ1226222 has been marked as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1205862
toracat

toracat

2015-06-23 16:48

manager   ~0023457

kernel 3.10.0-229.7.2.el7 is out upstream (RH). The patch is now in this kernel, so will be removed from the plus kernel.

From the distro kernel changelog:

- [kernel] futex: Ensure get_futex_key_refs() always implies a barrier (Larry Woodman) [1219169 1205862]

Issue History

Date Modified Username Field Change
2015-05-28 17:00 amuraru New Issue
2015-05-28 17:27 amuraru File Added: 0001-PATCH-Backport-futex-Ensure-get_futex_key_refs-alway.patch
2015-05-28 17:43 amuraru Note Added: 0023236
2015-05-28 17:59 toracat Note Added: 0023237
2015-05-28 18:00 toracat Status new => assigned
2015-05-28 19:13 toracat Note Added: 0023239
2015-05-28 19:29 toracat File Added: futex.patch
2015-05-29 09:02 amuraru Note Added: 0023244
2015-06-03 18:01 johnds Note Added: 0023284
2015-06-03 19:34 toracat Note Added: 0023285
2015-06-03 19:36 toracat Note Added: 0023286
2015-06-03 19:57 johnds Note Added: 0023287
2015-06-03 20:51 toracat Note Added: 0023288
2015-06-03 20:53 amuraru Note Added: 0023289
2015-06-03 21:16 johnds Note Added: 0023290
2015-06-03 21:34 toracat Note Added: 0023291
2015-06-04 04:46 toracat Note Added: 0023293
2015-06-04 14:50 johnds Note Added: 0023302
2015-06-05 01:11 johnds Note Added: 0023311
2015-06-05 14:45 amuraru Note Added: 0023318
2015-06-05 15:02 amuraru Note Added: 0023319
2015-06-05 15:37 toracat Note Added: 0023320
2015-06-06 14:36 johnds Note Added: 0023331
2015-06-12 08:10 johnds Note Added: 0023385
2015-06-12 13:08 toracat Note Added: 0023387
2015-06-12 14:02 johnds Note Added: 0023388
2015-06-14 11:51 amuraru Note Added: 0023397
2015-06-23 16:48 toracat Note Added: 0023457
2015-10-05 15:17 toracat Status assigned => resolved
2015-10-05 15:17 toracat Resolution open => fixed