View Issue Details
ID | Project | Category | View Status | Date Submitted | Last Update |
---|---|---|---|---|---|
0013453 | CentOS-7 | kernel | public | 2017-06-20 17:50 | 2017-07-12 15:20 |
Reporter | dnsmichi | Assigned To | |||
Priority | high | Severity | crash | Reproducibility | always |
Status | resolved | Resolution | fixed | ||
Platform | x86_64 | OS | CentOS | OS Version | 7 |
Product Version | 7.3.1611 | ||||
Summary | 0013453: Kernel update 3.10.0-514.21.2 for CVE-2017-1000364 breaks applications which set a lower stack size than 4112kbytes | ||||
Description | Setting a reasonable low stack size breaks any application which attempts to execute anything, i.e. with execvpe(). The stack guard protection patch released as Kernel update 3.10.0-514.21.2 for CVE-2017-1000364 was backported to el7 and centos7 received the updated packages too. https://access.redhat.com/security/vulnerabilities/stackguard | ||||
Steps To Reproduce | 4112 kbytes works. # uname -a ; ulimit -s 4112 && /bin/true && echo "works" Linux icinga2 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux works 4111 kbytes does not (use a new shell) # uname -a ; ulimit -s 4111 && /bin/true && echo "works" Linux icinga2 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux -bash: /bin/true: Argument list too long | ||||
Additional Information | A bug against RHEL upstream has been reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1463241 This has been reported to the Icinga project on GitHub and we've then tracked it down being a bug in the stack guard patch for el7 only. el6 works. We are collecting details over there: https://github.com/Icinga/icinga2/issues/5367 Debian Stretch/Jessie works too. We've started to analyze the kernel patch diff between el7 and Debian Jessie, but are not Kernel developers. The main difference is the different kernel base versions (3.10 vs 3.16). An advisory including a workaround for icinga2 users can be found here: https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/ | ||||
Tags | kernel | ||||
abrt_hash | |||||
URL | |||||
I went into comparing the kernel sources to maybe shed some light for those knowing how to tackle it. I fairly don't understand much what the code exactly does. As a developer, I would just guess there is a wrong offset calculation somewhere, or something which treats a lower stack size >1MIB && < 4112KiB similar to the default guard. ``` wget http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.21.1.el7.src.rpm wget http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.21.2.el7.src.rpm mkdir 1 2 cd 1 && rpm2cpio ../kernel-3.10.0-514.21.1.el7.src.rpm | cpio -ivd && tar xf linux-3.10.0-514.21.1.el7.tar.xz && cd .. cd 2 && rpm2cpio ../kernel-3.10.0-514.21.2.el7.src.rpm | cpio -ivd && tar xf linux-3.10.0-514.21.2.el7.tar.xz && cd .. diff -ur 1/linux-3.10.0-514.21.1.el7/ 2/linux-3.10.0-514.21.2.el7/ > diff ``` Debian patches are located here: https://anonscm.debian.org/cgit/kernel/linux.git/log/?h=jessie-security 1) The patch in RHEL/CentOS in stack_guard_area() returns with less than. ``` mm/mmap.c +int stack_guard_area(struct vm_area_struct *vma, unsigned long address) ... + return vma->vm_end - address < stack_guard_gap; ``` The patch released in Debian Jessie-Security https://anonscm.debian.org/cgit/kernel/linux.git/commit/?h=jessie-security&id=af5f37d1b8feebe4cf4976770a6c37f64de817c7 does a less than EQUAL comparison. This may or may not return different booleans. ``` ++ return vma->vm_end - address <= stack_guard_gap; ``` 2) task_mmu.c differs too. CentOS ``` diff -ur 1/linux-3.10.0-514.21.1.el7/fs/proc/task_mmu.c 2/linux-3.10.0-514.21.2.el7/fs/proc/task_mmu.c --- 1/linux-3.10.0-514.21.1.el7/fs/proc/task_mmu.c 2017-04-22 06:17:16.000000000 +0000 +++ 2/linux-3.10.0-514.21.2.el7/fs/proc/task_mmu.c 2017-05-28 20:42:06.000000000 +0000 @@ -293,11 +293,13 @@ /* We don't show the stack guard page in /proc/maps */ start = vma->vm_start; - if (stack_guard_page_start(vma, start)) - start += PAGE_SIZE; end = vma->vm_end; - if (stack_guard_page_end(vma, end)) - end -= PAGE_SIZE; + if (stack_guard_area(vma, start)) { + if (vma->vm_flags & VM_GROWSDOWN) + start += stack_guard_gap; + else + end -= stack_guard_gap; + } ``` There's no explicit check for VM_GROWSUP. Debian ``` +--- a/fs/proc/task_mmu.c ++++ b/fs/proc/task_mmu.c +@@ -276,11 +276,14 @@ show_map_vma(struct seq_file *m, struct + + /* We don't show the stack guard page in /proc/maps */ + start = vma->vm_start; +- if (stack_guard_page_start(vma, start)) +- start += PAGE_SIZE; + end = vma->vm_end; +- if (stack_guard_page_end(vma, end)) +- end -= PAGE_SIZE; ++ if (vma->vm_flags & VM_GROWSDOWN) { ++ if (stack_guard_area(vma, start)) ++ start += stack_guard_gap; ++ } else if (vma->vm_flags & VM_GROWSUP) { ++ if (stack_guard_area(vma, end)) ++ end -= stack_guard_gap; ++ } ``` 3) CentOS ``` @@ -2750,7 +2716,8 @@ return VM_FAULT_SIGBUS; /* Check if we need to add a guard page to the stack */ - if (check_stack_guard_page(vma, address) < 0) + if ((vma->vm_flags & (VM_GROWSDOWN|VM_GROWSUP)) && + expand_stack(vma, address) < 0) return VM_FAULT_SIGBUS; ``` Debian ``` +@@ -2642,8 +2610,10 @@ static int do_anonymous_page(struct mm_s + return VM_FAULT_SIGBUS; + + /* Check if we need to add a guard page to the stack */ +- if (check_stack_guard_page(vma, address) < 0) +- return VM_FAULT_SIGSEGV; ++ if (stack_guard_area(vma, address)) { ++ if (expand_stack(vma, address) < 0) ++ return VM_FAULT_SIGSEGV; ++ } ``` Debian returns a SIGSEGV instead of SIGBUS, but I think it could also be related to the difference with just checking vm_flags and not calling stack_guard_area() like Debian does. The main different also is that stack_guard_area() checks against VM_GROWSUP only. Not sure why the CentOS patch differs so much (probably the Kernel base version behaves differently?). This might be completely stupid, but maybe it helps resolve the issue :) |
|
Well .. the real question here is, what does the RHEL kernel do. The main CentOS kernel will not deviate technically from the RHEL Source code, so unless this kernel behaves differently than the RHEL one it will not change. It is possible that we MIGHT be able to roll something into the CentOS Plus kernel if it is necessary. | |
Thanks for your fast answer :) I'd appreciate if this bug would be fixed in RHEL and CentOS could just rebuild the new kernel version. I was granted access to https://bugzilla.redhat.com/show_bug.cgi?id=1463241 so I'd continue working on helping upstream getting this fixed. Just keep this issue open as reference, if other CentOS users run into the same issue. |
|
@dnsmichi Thanks for your help in this issue. Yes, please keep us posted as the RH BZ is private. I see that a workaround is presented here: https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/ |
|
Yep, the bad thing about that workaround is that Icinga 2 uses lots of threads and with the default stack size for threads being 8 MB this is a lot more than the 256KB default. Which in turn means that the virtual reserved memory size is a lot more than before. Our users may have to deal with that, so I am eager to help with anything to quickly resolve this issue. Kurt Seifried and Evgeni Golov helped with getting this issue to the right people at RedHat. Right now it is tagged as regression with priority urgent. I've shared my insights from here with them, and am hoping for the best :) |
|
s/priority/severity/ urgent, sorry. | |
https://bugzilla.redhat.com/show_bug.cgi?id=1463241 Is now public :) | |
Ah, good. Thanks. | |
RedHat created a patch and a test build which we successfully tested. Though it seems that there might be more issues. Hoping this gets released soon to also allow CentOS getting the fix :) You might want to poke someone at Redhat for the patch - https://bugzilla.redhat.com/show_bug.cgi?id=1463241#c16 |
|
Is kernel-3.10.0-514.26.2.el7 released today supposed to resolve the issue ? | |
I've tested both newer kernel versions (kernel-3.10.0-514.26.1.el7, kernel-3.10.0-514.26.2.el7 ) with the given commandline to reproduce the crash. The crash seems to be fixed already in the kernel-3.10.0-514.26.1.el7. |
|
All stack clash related issues have been resolved in the latest kernels ( EL6: kernel-2.6.32-696.6.3.el6 or later and EL7: kernel-3.10.0-514.26.2.el7 or later). | |
Date Modified | Username | Field | Change |
---|---|---|---|
2017-06-20 17:50 | dnsmichi | New Issue | |
2017-06-20 17:50 | dnsmichi | Tag Attached: kernel | |
2017-06-20 19:23 | dnsmichi | Note Added: 0029531 | |
2017-06-20 22:33 | JohnnyHughes | Note Added: 0029532 | |
2017-06-21 07:45 | dnsmichi | Note Added: 0029536 | |
2017-06-21 15:34 | toracat | Note Added: 0029538 | |
2017-06-21 15:35 | toracat | Status | new => confirmed |
2017-06-21 16:16 | dnsmichi | Note Added: 0029539 | |
2017-06-21 16:20 | dnsmichi | Note Added: 0029540 | |
2017-06-21 17:19 | dnsmichi | Note Added: 0029541 | |
2017-06-21 18:48 | toracat | Note Added: 0029542 | |
2017-06-22 18:29 | dnsmichi | Note Added: 0029553 | |
2017-07-04 18:39 | toracat | Note Added: 0029592 | |
2017-07-05 10:11 | kora | Note Added: 0029600 | |
2017-07-12 15:19 | toracat | Note Added: 0029651 | |
2017-07-12 15:20 | toracat | Status | confirmed => resolved |
2017-07-12 15:20 | toracat | Resolution | open => fixed |