View Issue Details

IDProjectCategoryView StatusLast Update
0005722CentOS-6glibcpublic2012-06-29 13:25
Reportergstanley 
PriorityurgentSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformXen / Virtualized EnviornmentsOSCentOSOS Version6.2
Product Version6.2 
Target VersionFixed in Version 
Summary0005722: glibc 2.12 has regression with AVX/XSave under Xen (and other virt enviorments)
DescriptionIn glibc 2.12, drepper added a patch to support AVX, so that stopped SIGILLs on dedicated hardware, however under Xen environments random sigills still occur regardless of -march=native support with CFLAGS.

This patch was merged in from http://sourceware.org/git/?p=glibc.git;a=history;f=sysdeps/x86_64/dl-trampoline.S;h=7691662867ab0da1596ededa745109dfb45345c8;hb=HEAD from last year with more AVX/XSave fixes. This does work in my test enviorments.

-bash-3.2# cat glibc-avx-fix-gary.patch
--- sysdeps/x86_64/dl-trampoline.S.old 2012-05-10 05:56:27.000000000 -0500
+++ sysdeps/x86_64/dl-trampoline.S 2012-05-10 07:20:08.000000000 -0500
@@ -1,5 +1,5 @@
 /* PLT trampolines. x86-64 version.
- Copyright (C) 2004, 2005, 2007, 2009 Free Software Foundation, Inc.
+ Copyright (C) 2004, 2005, 2007, 2009, 2011 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
 
    The GNU C Library is free software; you can redistribute it and/or
@@ -13,9 +13,8 @@
    Lesser General Public License for more details.
 
    You should have received a copy of the GNU Lesser General Public
- License along with the GNU C Library; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
- 02111-1307 USA. */
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
 
 #include <config.h>
 #include <sysdep.h>
@@ -27,8 +26,9 @@
        .align 16
        cfi_startproc
 _dl_runtime_resolve:
+ cfi_adjust_cfa_offset(16) # Incorporate PLT
        subq $56,%rsp
- cfi_adjust_cfa_offset(72) # Incorporate PLT
+ cfi_adjust_cfa_offset(56)
        movq %rax,(%rsp) # Preserve registers otherwise clobbered.
        movq %rcx, 8(%rsp)
        movq %rdx, 16(%rsp)
@@ -139,24 +139,31 @@
        movl $1, %eax
        cpuid
        movq %r11,%rbx # Restore rbx
- movl $1, %eax
- testl $(1 << 28), %ecx
+ xorl %eax, %eax
+ // AVX and XSAVE supported?
+ andl $((1 << 28) | (1 << 27)), %ecx
+ cmpl $((1 << 28) | (1 << 27)), %ecx
        jne 2f
- negl %eax
-2: movl %eax, L(have_avx)(%rip)
+ xorl %ecx, %ecx
+ // Get XFEATURE_ENABLED_MASK
+ xgetbv
+ andl $0x6, %eax
+2: subl $0x5, %eax
+ movl %eax, L(have_avx)(%rip)
        cmpl $0, %eax
 
 1: js L(no_avx)
 
 # define RESTORE_AVX
+# define MORE_CODE
 # include "dl-trampoline.h"
 
        .align 16
 L(no_avx):
 # endif
 
-# undef RESTORE_AVX
-# include "dl-trampoline.h"
+# undef RESTORE_AVX
+# include "dl-trampoline.h"
 
        cfi_endproc
        .size _dl_runtime_profile, .-_dl_runtime_profile
@@ -176,11 +183,20 @@
        movl $1, %eax
        cpuid
        movq %r11,%rbx # Restore rbx
- movl $1, %eax
- testl $(1 << 28), %ecx
+ xorl %eax, %eax
+ // AVX and XSAVE supported?
+ andl $((1 << 28) | (1 << 27)), %ecx
+ cmpl $((1 << 28) | (1 << 27)), %ecx
        jne 2f
- negl %eax
-2: movl %eax, L(have_avx)(%rip)
+ xorl %ecx, %ecx
+ // Get XFEATURE_ENABLED_MASK
+ xgetbv
+ andl $0x6, %eax
+ cmpl $0x6, %eax
+ // Nonzero if SSE and AVX state saving is enabled.
+ sete %al
+2: leal -1(%eax,%eax), %eax
+ movl %eax, L(have_avx)(%rip)
        cmpl $0, %eax
 
 1: js L(no_avx5)
--- dl-trampoline.h.old 2012-05-10 07:22:16.000000000 -0500
+++ dl-trampoline.h 2012-05-10 07:22:56.000000000 -0500
@@ -1,6 +1,6 @@
 /* Partial PLT profile trampoline to save and restore x86-64 vector
    registers.
- Copyright (C) 2009 Free Software Foundation, Inc.
+ Copyright (C) 2009, 2011 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
 
    The GNU C Library is free software; you can redistribute it and/or
@@ -14,9 +14,8 @@
    Lesser General Public License for more details.
 
    You should have received a copy of the GNU Lesser General Public
- License along with the GNU C Library; if not, write to the Free
- Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
- 02111-1307 USA. */
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
 
 #ifdef RESTORE_AVX
        /* This is to support AVX audit modules. */
@@ -195,14 +194,14 @@
           _dl_call_pltexit. The La_x86_64_regs is being pointed by rsp now,
           so we just need to allocate the sizeof(La_x86_64_retval) space on
           the stack, since the alignment has already been taken care of. */
-# ifdef RESTORE_AVX
+#ifdef RESTORE_AVX
        /* sizeof(La_x86_64_retval). Need extra space for 2 SSE
           registers to detect if xmm0/xmm1 registers are changed
           by audit module. */
        subq $(LRV_SIZE + XMM_SIZE*2), %rsp
-# else
+#else
        subq $LRV_SIZE, %rsp # sizeof(La_x86_64_retval)
-# endif
+#endif
        movq %rsp, %rcx # La_x86_64_retval argument to %rcx.
 
        /* Fill in the La_x86_64_retval structure. */
@@ -212,7 +211,7 @@
        movaps %xmm0, LRV_XMM0_OFFSET(%rcx)
        movaps %xmm1, LRV_XMM1_OFFSET(%rcx)
 
-# ifdef RESTORE_AVX
+#ifdef RESTORE_AVX
        /* This is to support AVX audit modules. */
        vmovdqu %ymm0, LRV_VECTOR0_OFFSET(%rcx)
        vmovdqu %ymm1, LRV_VECTOR1_OFFSET(%rcx)
@@ -221,14 +220,14 @@
           by audit module. */
        vmovdqa %xmm0, (LRV_SIZE)(%rcx)
        vmovdqa %xmm1, (LRV_SIZE + XMM_SIZE)(%rcx)
-# endif
+#endif
 
        fstpt LRV_ST0_OFFSET(%rcx)
        fstpt LRV_ST1_OFFSET(%rcx)
 
        movq 24(%rbx), %rdx # La_x86_64_regs argument to %rdx.
        movq 40(%rbx), %rsi # Copy args pushed by PLT in register.
- movq 32(%rbx), %rdi # %rdi: link_map, %rsi: reloc_index
+ movq 32(%rbx), %rdi # %rdi: link_map, %rsi: reloc_index
        call _dl_call_pltexit
 
        /* Restore return registers. */
@@ -238,7 +237,7 @@
        movaps LRV_XMM0_OFFSET(%rsp), %xmm0
        movaps LRV_XMM1_OFFSET(%rsp), %xmm1
 
-# ifdef RESTORE_AVX
+#ifdef RESTORE_AVX
        /* Check if xmm0/xmm1 registers are changed by audit module. */
        vpcmpeqq (LRV_SIZE)(%rsp), %xmm0, %xmm2
        vpmovmskb %xmm2, %esi
@@ -253,7 +252,7 @@
        vmovdqu LRV_VECTOR1_OFFSET(%rsp), %ymm1
 
 1:
-# endif
+#endif
 
        fldt LRV_ST1_OFFSET(%rsp)
        fldt LRV_ST0_OFFSET(%rsp)
@@ -267,3 +266,10 @@
                                # (eats the reloc index and link_map)
        cfi_adjust_cfa_offset(-48)
        retq
+
+#ifdef MORE_CODE
+ cfi_adjust_cfa_offset(48)
+ cfi_rel_offset(%rbx, 0)
+ cfi_def_cfa_register(%rbx)
+# undef MORE_CODE
+#endif


Steps To ReproduceHave a sandy bridge machine with AVX. (You could get a cloud machine from softlayer with CentOS 6.2 and run the following:

http://vps.cpgarys.info/test-code-case-52461.zip

Once you unzip that, 'make' then run ./exe, the 2nd one should crash with SIGILL. Problem code is Xen doesn't seem to allow xsave to be enabled correctly which is needed with AVX, so if AVX is working and xsave() isn't, sigills happen.


TagsNo tags attached.

Activities

gstanley

gstanley

2012-05-23 18:16

reporter   ~0015140

Here's some more information.

- On virtualized environments, AVX is being reported as enabled to the containers, but xsave is not. AVX requires XSAVE support in order to function.

- GCC on virtualized environments that has AVX processors will generate bogus code with -march=native because while cpuid reports the processor having AVX support, the container isn't supporting it so you'll get SIGILL's on AVX instructions.
b.kunz

b.kunz

2012-06-29 13:25

reporter   ~0015349

This bug is currently stopping me from setting up a server, because it occurs quite frequently in the perl and java binaries. Thank you for your excellent report.

Now, who is responsible for fixing this? Xen, to report the correct cpuid flags (not to set AVX) or gcc to generate correct code (not to use AVX if XSAVE isn't available)? Surely this is not CentOS specific?

Or, in simpler terms, when and where will i be able to pull an update from that fixes this issue =)

host
XenServer 6.0.2-53456p
Linux xenserver-0 2.6.32.12-0.7.1.xs6.0.2.542.170665xen #1 SMP Tue Jan 17 15:14:24 EST 2012 i686 i686 i386 GNU/Linux

guest
CentOS release 6.2 (Final)
Linux allen-dev01.allen.de 2.6.32-220.23.1.el6.x86_64 #1 SMP Mon Jun 18 18:58:52 BST 2012 x86_64 x86_64 x86_64 GNU/Linux

Issue History

Date Modified Username Field Change
2012-05-10 14:02 gstanley New Issue
2012-05-23 18:16 gstanley Note Added: 0015140
2012-06-29 13:25 b.kunz Note Added: 0015349