2017-09-21 12:23 UTC
  

View Issue Details [ Jump to Notes ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0013453CentOS-7kernelpublic2017-06-20 17:502017-07-12 15:20
Reporterdnsmichi 
PriorityhighSeveritycrashReproducibilityalways
StatusresolvedResolutionfixed 
Platformx86_64OSCentOSOS Version7
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0013453: Kernel update 3.10.0-514.21.2 for CVE-2017-1000364 breaks applications which set a lower stack size than 4112kbytes
DescriptionSetting a reasonable low stack size breaks any application which attempts to execute anything, i.e. with execvpe().

The stack guard protection patch released as Kernel update 3.10.0-514.21.2 for CVE-2017-1000364 was backported to el7 and centos7 received the updated packages too.

https://access.redhat.com/security/vulnerabilities/stackguard


Steps To Reproduce4112 kbytes works.

# uname -a ; ulimit -s 4112 && /bin/true && echo "works"
Linux icinga2 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
works

4111 kbytes does not (use a new shell)

# uname -a ; ulimit -s 4111 && /bin/true && echo "works"
Linux icinga2 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
-bash: /bin/true: Argument list too long
Additional InformationA bug against RHEL upstream has been reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1463241

This has been reported to the Icinga project on GitHub and we've then tracked it down being a bug in the stack guard patch for el7 only. el6 works. We are collecting details over there: https://github.com/Icinga/icinga2/issues/5367


Debian Stretch/Jessie works too. We've started to analyze the kernel patch diff between el7 and Debian Jessie, but are not Kernel developers. The main difference is the different kernel base versions (3.10 vs 3.16).


An advisory including a workaround for icinga2 users can be found here: https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/
Tagskernel
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0029531

dnsmichi (reporter)

2017-06-20 19:23

 I went into comparing the kernel sources to maybe shed some light for those knowing how to tackle it. I fairly don't understand much what the code exactly does.

As a developer, I would just guess there is a wrong offset calculation somewhere, or something which treats a lower stack size >1MIB && < 4112KiB similar to the default guard.


```
wget http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.21.1.el7.src.rpm
wget http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.21.2.el7.src.rpm

mkdir 1 2

cd 1 && rpm2cpio ../kernel-3.10.0-514.21.1.el7.src.rpm | cpio -ivd && tar xf linux-3.10.0-514.21.1.el7.tar.xz && cd ..
cd 2 && rpm2cpio ../kernel-3.10.0-514.21.2.el7.src.rpm | cpio -ivd && tar xf linux-3.10.0-514.21.2.el7.tar.xz && cd ..

diff -ur 1/linux-3.10.0-514.21.1.el7/ 2/linux-3.10.0-514.21.2.el7/ > diff

```

Debian patches are located here: https://anonscm.debian.org/cgit/kernel/linux.git/log/?h=jessie-security

1)

The patch in RHEL/CentOS in stack_guard_area() returns with less than.

```
mm/mmap.c

+int stack_guard_area(struct vm_area_struct *vma, unsigned long address)
...
+ return vma->vm_end - address < stack_guard_gap;
```

The patch released in Debian Jessie-Security https://anonscm.debian.org/cgit/kernel/linux.git/commit/?h=jessie-security&id=af5f37d1b8feebe4cf4976770a6c37f64de817c7

does a less than EQUAL comparison. This may or may not return different booleans.

```
++ return vma->vm_end - address <= stack_guard_gap;
```



2) task_mmu.c differs too.


CentOS

```
diff -ur 1/linux-3.10.0-514.21.1.el7/fs/proc/task_mmu.c 2/linux-3.10.0-514.21.2.el7/fs/proc/task_mmu.c
--- 1/linux-3.10.0-514.21.1.el7/fs/proc/task_mmu.c 2017-04-22 06:17:16.000000000 +0000
+++ 2/linux-3.10.0-514.21.2.el7/fs/proc/task_mmu.c 2017-05-28 20:42:06.000000000 +0000
@@ -293,11 +293,13 @@

        /* We don't show the stack guard page in /proc/maps */
        start = vma->vm_start;
- if (stack_guard_page_start(vma, start))
- start += PAGE_SIZE;
        end = vma->vm_end;
- if (stack_guard_page_end(vma, end))
- end -= PAGE_SIZE;
+ if (stack_guard_area(vma, start)) {
+ if (vma->vm_flags & VM_GROWSDOWN)
+ start += stack_guard_gap;
+ else
+ end -= stack_guard_gap;
+ }
```

There's no explicit check for VM_GROWSUP.


Debian

```
+--- a/fs/proc/task_mmu.c
++++ b/fs/proc/task_mmu.c
+@@ -276,11 +276,14 @@ show_map_vma(struct seq_file *m, struct
+
+ /* We don't show the stack guard page in /proc/maps */
+ start = vma->vm_start;
+- if (stack_guard_page_start(vma, start))
+- start += PAGE_SIZE;
+ end = vma->vm_end;
+- if (stack_guard_page_end(vma, end))
+- end -= PAGE_SIZE;
++ if (vma->vm_flags & VM_GROWSDOWN) {
++ if (stack_guard_area(vma, start))
++ start += stack_guard_gap;
++ } else if (vma->vm_flags & VM_GROWSUP) {
++ if (stack_guard_area(vma, end))
++ end -= stack_guard_gap;
++ }
```


3)


CentOS

```
@@ -2750,7 +2716,8 @@
                return VM_FAULT_SIGBUS;

        /* Check if we need to add a guard page to the stack */
- if (check_stack_guard_page(vma, address) < 0)
+ if ((vma->vm_flags & (VM_GROWSDOWN|VM_GROWSUP)) &&
+ expand_stack(vma, address) < 0)
                return VM_FAULT_SIGBUS;
```

Debian

```
+@@ -2642,8 +2610,10 @@ static int do_anonymous_page(struct mm_s
+ return VM_FAULT_SIGBUS;
+
+ /* Check if we need to add a guard page to the stack */
+- if (check_stack_guard_page(vma, address) < 0)
+- return VM_FAULT_SIGSEGV;
++ if (stack_guard_area(vma, address)) {
++ if (expand_stack(vma, address) < 0)
++ return VM_FAULT_SIGSEGV;
++ }
```


Debian returns a SIGSEGV instead of SIGBUS, but I think it could also be related to the difference with just checking vm_flags and not calling stack_guard_area() like Debian does.

The main different also is that stack_guard_area() checks against VM_GROWSUP only. Not sure why the CentOS patch differs so much (probably the Kernel base version behaves differently?).


This might be completely stupid, but maybe it helps resolve the issue :)

~0029532

JohnnyHughes (administrator)

2017-06-20 22:33

 Well .. the real question here is, what does the RHEL kernel do. The main CentOS kernel will not deviate technically from the RHEL Source code, so unless this kernel behaves differently than the RHEL one it will not change. It is possible that we MIGHT be able to roll something into the CentOS Plus kernel if it is necessary.

~0029536

dnsmichi (reporter)

2017-06-21 07:45

 Thanks for your fast answer :) I'd appreciate if this bug would be fixed in RHEL and CentOS could just rebuild the new kernel version. I was granted access to https://bugzilla.redhat.com/show_bug.cgi?id=1463241 so I'd continue working on helping upstream getting this fixed.

Just keep this issue open as reference, if other CentOS users run into the same issue.

~0029538

toracat (manager)

2017-06-21 15:34

 @dnsmichi

Thanks for your help in this issue. Yes, please keep us posted as the RH BZ is private.

I see that a workaround is presented here:

https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/

~0029539

dnsmichi (reporter)

2017-06-21 16:16

 Yep, the bad thing about that workaround is that Icinga 2 uses lots of threads and with the default stack size for threads being 8 MB this is a lot more than the 256KB default. Which in turn means that the virtual reserved memory size is a lot more than before. Our users may have to deal with that, so I am eager to help with anything to quickly resolve this issue.

Kurt Seifried and Evgeni Golov helped with getting this issue to the right people at RedHat. Right now it is tagged as regression with priority urgent. I've shared my insights from here with them, and am hoping for the best :)

~0029540

dnsmichi (reporter)

2017-06-21 16:20

 s/priority/severity/ urgent, sorry.

~0029541

dnsmichi (reporter)

2017-06-21 17:19

 https://bugzilla.redhat.com/show_bug.cgi?id=1463241 Is now public :)

~0029542

toracat (manager)

2017-06-21 18:48

 Ah, good. Thanks.

~0029553

dnsmichi (reporter)

2017-06-22 18:29

 RedHat created a patch and a test build which we successfully tested. Though it seems that there might be more issues. Hoping this gets released soon to also allow CentOS getting the fix :)

You might want to poke someone at Redhat for the patch - https://bugzilla.redhat.com/show_bug.cgi?id=1463241#c16

~0029592

toracat (manager)

2017-07-04 18:39

 Is kernel-3.10.0-514.26.2.el7 released today supposed to resolve the issue ?

~0029600

kora (reporter)

2017-07-05 10:11

 I've tested both newer kernel versions (kernel-3.10.0-514.26.1.el7, kernel-3.10.0-514.26.2.el7 ) with the given commandline to reproduce the crash.
The crash seems to be fixed already in the kernel-3.10.0-514.26.1.el7.

~0029651

toracat (manager)

2017-07-12 15:19

 All stack clash related issues have been resolved in the latest kernels ( EL6: kernel-2.6.32-696.6.3.el6 or later and EL7: kernel-3.10.0-514.26.2.el7 or later).
+Notes

-Issue History
Date Modified Username Field Change
2017-06-20 17:50 dnsmichi New Issue
2017-06-20 17:50 dnsmichi Tag Attached: kernel
2017-06-20 19:23 dnsmichi Note Added: 0029531
2017-06-20 22:33 JohnnyHughes Note Added: 0029532
2017-06-21 07:45 dnsmichi Note Added: 0029536
2017-06-21 15:34 toracat Note Added: 0029538
2017-06-21 15:35 toracat Status new => confirmed
2017-06-21 16:16 dnsmichi Note Added: 0029539
2017-06-21 16:20 dnsmichi Note Added: 0029540
2017-06-21 17:19 dnsmichi Note Added: 0029541
2017-06-21 18:48 toracat Note Added: 0029542
2017-06-22 18:29 dnsmichi Note Added: 0029553
2017-07-04 18:39 toracat Note Added: 0029592
2017-07-05 10:11 kora Note Added: 0029600
2017-07-12 15:19 toracat Note Added: 0029651
2017-07-12 15:20 toracat Status confirmed => resolved
2017-07-12 15:20 toracat Resolution open => fixed
+Issue History