 |
I went into comparing the kernel sources to maybe shed some light for those knowing how to tackle it. I fairly don't understand much what the code exactly does. As a developer, I would just guess there is a wrong offset calculation somewhere, or something which treats a lower stack size >1MIB && < 4112KiB similar to the default guard. ``` wget http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.21.1.el7.src.rpm wget http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.21.2.el7.src.rpm mkdir 1 2 cd 1 && rpm2cpio ../kernel-3.10.0-514.21.1.el7.src.rpm | cpio -ivd && tar xf linux-3.10.0-514.21.1.el7.tar.xz && cd .. cd 2 && rpm2cpio ../kernel-3.10.0-514.21.2.el7.src.rpm | cpio -ivd && tar xf linux-3.10.0-514.21.2.el7.tar.xz && cd .. diff -ur 1/linux-3.10.0-514.21.1.el7/ 2/linux-3.10.0-514.21.2.el7/ > diff ``` Debian patches are located here: https://anonscm.debian.org/cgit/kernel/linux.git/log/?h=jessie-security 1) The patch in RHEL/CentOS in stack_guard_area() returns with less than. ``` mm/mmap.c +int stack_guard_area(struct vm_area_struct *vma, unsigned long address) ... + return vma->vm_end - address < stack_guard_gap; ``` The patch released in Debian Jessie-Security https://anonscm.debian.org/cgit/kernel/linux.git/commit/?h=jessie-security&id=af5f37d1b8feebe4cf4976770a6c37f64de817c7 does a less than EQUAL comparison. This may or may not return different booleans. ``` ++ return vma->vm_end - address <= stack_guard_gap; ``` 2) task_mmu.c differs too. CentOS ``` diff -ur 1/linux-3.10.0-514.21.1.el7/fs/proc/task_mmu.c 2/linux-3.10.0-514.21.2.el7/fs/proc/task_mmu.c --- 1/linux-3.10.0-514.21.1.el7/fs/proc/task_mmu.c 2017-04-22 06:17:16.000000000 +0000 +++ 2/linux-3.10.0-514.21.2.el7/fs/proc/task_mmu.c 2017-05-28 20:42:06.000000000 +0000 @@ -293,11 +293,13 @@ /* We don't show the stack guard page in /proc/maps */ start = vma->vm_start; - if (stack_guard_page_start(vma, start)) - start += PAGE_SIZE; end = vma->vm_end; - if (stack_guard_page_end(vma, end)) - end -= PAGE_SIZE; + if (stack_guard_area(vma, start)) { + if (vma->vm_flags & VM_GROWSDOWN) + start += stack_guard_gap; + else + end -= stack_guard_gap; + } ``` There's no explicit check for VM_GROWSUP. Debian ``` +--- a/fs/proc/task_mmu.c ++++ b/fs/proc/task_mmu.c +@@ -276,11 +276,14 @@ show_map_vma(struct seq_file *m, struct + + /* We don't show the stack guard page in /proc/maps */ + start = vma->vm_start; +- if (stack_guard_page_start(vma, start)) +- start += PAGE_SIZE; + end = vma->vm_end; +- if (stack_guard_page_end(vma, end)) +- end -= PAGE_SIZE; ++ if (vma->vm_flags & VM_GROWSDOWN) { ++ if (stack_guard_area(vma, start)) ++ start += stack_guard_gap; ++ } else if (vma->vm_flags & VM_GROWSUP) { ++ if (stack_guard_area(vma, end)) ++ end -= stack_guard_gap; ++ } ``` 3) CentOS ``` @@ -2750,7 +2716,8 @@ return VM_FAULT_SIGBUS; /* Check if we need to add a guard page to the stack */ - if (check_stack_guard_page(vma, address) < 0) + if ((vma->vm_flags & (VM_GROWSDOWN|VM_GROWSUP)) && + expand_stack(vma, address) < 0) return VM_FAULT_SIGBUS; ``` Debian ``` +@@ -2642,8 +2610,10 @@ static int do_anonymous_page(struct mm_s + return VM_FAULT_SIGBUS; + + /* Check if we need to add a guard page to the stack */ +- if (check_stack_guard_page(vma, address) < 0) +- return VM_FAULT_SIGSEGV; ++ if (stack_guard_area(vma, address)) { ++ if (expand_stack(vma, address) < 0) ++ return VM_FAULT_SIGSEGV; ++ } ``` Debian returns a SIGSEGV instead of SIGBUS, but I think it could also be related to the difference with just checking vm_flags and not calling stack_guard_area() like Debian does. The main different also is that stack_guard_area() checks against VM_GROWSUP only. Not sure why the CentOS patch differs so much (probably the Kernel base version behaves differently?). This might be completely stupid, but maybe it helps resolve the issue :) |
 |
Well .. the real question here is, what does the RHEL kernel do. The main CentOS kernel will not deviate technically from the RHEL Source code, so unless this kernel behaves differently than the RHEL one it will not change. It is possible that we MIGHT be able to roll something into the CentOS Plus kernel if it is necessary. |
|
|
Thanks for your fast answer :) I'd appreciate if this bug would be fixed in RHEL and CentOS could just rebuild the new kernel version. I was granted access to https://bugzilla.redhat.com/show_bug.cgi?id=1463241 so I'd continue working on helping upstream getting this fixed. Just keep this issue open as reference, if other CentOS users run into the same issue. |
 |
@dnsmichi Thanks for your help in this issue. Yes, please keep us posted as the RH BZ is private. I see that a workaround is presented here: https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/ |
|
|
Yep, the bad thing about that workaround is that Icinga 2 uses lots of threads and with the default stack size for threads being 8 MB this is a lot more than the 256KB default. Which in turn means that the virtual reserved memory size is a lot more than before. Our users may have to deal with that, so I am eager to help with anything to quickly resolve this issue. Kurt Seifried and Evgeni Golov helped with getting this issue to the right people at RedHat. Right now it is tagged as regression with priority urgent. I've shared my insights from here with them, and am hoping for the best :) |
|
|
s/priority/severity/ urgent, sorry. |
|
|
https://bugzilla.redhat.com/show_bug.cgi?id=1463241 Is now public :) |
|
|
Ah, good. Thanks. |
|
|
RedHat created a patch and a test build which we successfully tested. Though it seems that there might be more issues. Hoping this gets released soon to also allow CentOS getting the fix :) You might want to poke someone at Redhat for the patch - https://bugzilla.redhat.com/show_bug.cgi?id=1463241#c16 |
|
|
Is kernel-3.10.0-514.26.2.el7 released today supposed to resolve the issue ? |
|
|
I've tested both newer kernel versions (kernel-3.10.0-514.26.1.el7, kernel-3.10.0-514.26.2.el7 ) with the given commandline to reproduce the crash. The crash seems to be fixed already in the kernel-3.10.0-514.26.1.el7. |
|
|
All stack clash related issues have been resolved in the latest kernels ( EL6: kernel-2.6.32-696.6.3.el6 or later and EL7: kernel-3.10.0-514.26.2.el7 or later). |
- Anonymous
