View Issue Details

IDProjectCategoryView StatusLast Update
0010930CentOS-6curlpublic2016-07-23 04:54
Reporterpko 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0010930: curl https://www.google.com -> Illegal Instruction after centos 6.8 update
DescriptionAfter upgrading to centos 6.8, all https curl requests are failing:

% curl https://www.google.com
Illegal Instruction
% echo $?
132

This is affecting some servers, but apparently not 100%. We've seen it so far on two separate servers. Downgrading curl, libcurl, and libcurl-devel to the versions provided in CentOS 6.7 fixes the problem.
Tags6.8, nss

Activities

bogdan.sh

bogdan.sh

2016-05-26 18:06

reporter   ~0026700

We were able to identify that AVX cpu flag is in charge here:

# cat /proc/cpuinfo | grep -i avx
#

It all started with PK11_Encrypt() function from /usr/lib64/libnss3.so which belongs to nss-3.21.0-8.el6.x86_64 .

Some debug information:

Program received signal SIGILL, Illegal instruction.
0x00007ffff1100d60 in ?? () from /usr/lib64/libfreeblpriv3.so
(gdb) bt
#0 0x00007ffff1100d60 in ?? () from /usr/lib64/libfreeblpriv3.so
#1 0x00007ffff10fd998 in ?? () from /usr/lib64/libfreeblpriv3.so
#2 0x00007ffff10c6b36 in ?? () from /usr/lib64/libfreeblpriv3.so
#3 0x00007ffff10c70f3 in ?? () from /usr/lib64/libfreeblpriv3.so
#4 0x00007ffff15cfaef in ?? () from /usr/lib64/libsoftokn3.so
#5 0x00007ffff15d0551 in ?? () from /usr/lib64/libsoftokn3.so
#6 0x00000031a3847fee in PK11_Encrypt () from /usr/lib64/libnss3.so
#7 0x00000031a24156e9 in ?? () from /usr/lib64/libssl3.so
#8 0x00000031a240f9b9 in ?? () from /usr/lib64/libssl3.so
#9 0x00000031a240fea2 in ?? () from /usr/lib64/libssl3.so
#10 0x00000031a2410267 in ?? () from /usr/lib64/libssl3.so
#11 0x00000031a2414c5c in ?? () from /usr/lib64/libssl3.so
#12 0x00000031a2415d7a in ?? () from /usr/lib64/libssl3.so
#13 0x00000031a2419bea in ?? () from /usr/lib64/libssl3.so
#14 0x00000031a241afd2 in ?? () from /usr/lib64/libssl3.so
#15 0x00000031a241bebf in ?? () from /usr/lib64/libssl3.so
#16 0x00000031a241e832 in ?? () from /usr/lib64/libssl3.so
#17 0x00000031a2425b75 in ?? () from /usr/lib64/libssl3.so
#18 0x00000031a24273af in SSL_ForceHandshake () from /usr/lib64/libssl3.so
#19 0x00007fffece4b868 in ?? () from /usr/lib64/libcurl.so.4
#20 0x00007fffece431c5 in Curl_ssl_connect () from /usr/lib64/libcurl.so.4
#21 0x00007fffece21b4b in Curl_http_connect () from /usr/lib64/libcurl.so.4
#22 0x00007fffece282f2 in Curl_protocol_connect () from /usr/lib64/libcurl.so.4
#23 0x00007fffece2e885 in Curl_connect () from /usr/lib64/libcurl.so.4
#24 0x00007fffece368c0 in Curl_perform () from /usr/lib64/libcurl.so.4

(gdb) x /i 0x00007ffff1100d60
=> 0x7ffff1100d60: vmovdqu (%rsi),%xmm0
(gdb)



The quick workaround is:
export NSS_DISABLE_HW_AES=1
pko

pko

2016-05-26 20:32

reporter   ~0026702

fwiw this also seems to fix it for the session:

export NSS_DISABLE_HW_GCM=1

We're up to 5 servers experience this problem over the last 48 hours and it seems to be accelerating.
kaz

kaz

2016-05-27 07:58

reporter   ~0026705

Seems to be the same issue as the recent problem occurring between RHEL 6.8 and Xen Hypervisor:
https://access.redhat.com/solutions/2313911
(article available for RHEL subscribes only)

This fix works for me:
# NSS_DISABLE_HW_AES=1
# yum downgrade nss nss-util nss-tools nss-sysinit
# yum install yum-plugin-versionlock
# yum versionlock add! nss-3.21.0-0.3.el6_7.x86_64 nss-sysinit-3.21.0-0.3.el6_7.x86_64 nss-tools-3.21.0-0.3.el6_7.x86_64 nss-util-3.21.0-0.3.el6_7.x86_64
razyr

razyr

2016-05-28 06:19

reporter   ~0026717

We had similar CURL libfreeblpriv3.so problems. We additionally had problems with silent failures in WordPress admin dashboard pages - no logged errors and null output.

We are a tenant in an OpenStack hosting environment which is using KVM. Our upstream provider moved our instance to a newer hypervisor array and all of our problems disappeared.

The instance in question has been incrementally upgraded through multiple 6.x releases for the past two years on the old hypervisor without incident. There is definitely something in 6.8 which is "sensitive" perhaps to discrepancies between advertised CPU capabilities and what the VM is actually presented with.
lpcollier

lpcollier

2016-05-28 10:05

reporter   ~0026718

I'm using 6.8 on a 1&1 virtual machine. I had similar errors with WordPress admin pages, and https Wordpress pages.

I couldn't get yum to downgrade curl, so the fix I used:

wget http://mirror.simwood.com/centos/6.7/os/x86_64/Packages/curl-7.19.7-46.el6.x86_64.rpm
wget http://mirror.simwood.com/centos/6.7/os/x86_64/Packages/libcurl-7.19.7-46.el6.x86_64.rpm
rpm -Uvh --oldpackage ./curl-7.19.7-46.el6.x86_64.rpm ./libcurl-7.19.7-46.el6.x86_64.rpm
yum install yum-plugin-versionlock
yum versionlock curl
yum versionlock libcurl
jissereitsma

jissereitsma

2016-05-28 20:08

reporter   ~0026719

I had a similar issue with PHP-FPM crashing (dumping cores) whenever PHP was using CURL to setup connections.

A workaround was to set all CURL connections to use SSLv3 (instead of TLSv1 which seems to be causing issues: curl_setopt($this->_getResource(), CURLOPT_SSLVERSION, 3);

Also a simple command-line "curl" crashed, but was fixed by setting NSS_DISABLE_HW_GCM=1.

Both solutions were not acceptable for me. Instead, I downgraded the following 5 packages from the mirror.simwood.com folder mentioned by @Ipcollier and this worked to get all PHP applications with CURL usage working again: nss, nspr, nss-sysinit, nss-tools, nss-util
support@viviotech.net

support@viviotech.net

2016-05-28 21:27

reporter   ~0026720

This appears to be effecting openjdk as well. Anyone have any crafty suggestions on how to temporarily address this (exporting the suggested variables didn't seem to have an effect (a shot in the dark I know but, it's pretty critical for me to get this working))? I'm seeing this manifested with this log entry in tomcat.

28-May-2016 13:49:41.427 SEVERE [http-nio-443-exec-12] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
 java.lang.RuntimeException: Could not generate DH keypair
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1429)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:350)
        at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:208)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1496)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Could not generate DH keypair
        at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:68)
        at sun.security.ssl.ServerHandshaker.setupEphemeralECDHKeys(ServerHandshaker.java:1432)
        at sun.security.ssl.ServerHandshaker.trySetCipherSuite(ServerHandshaker.java:1219)
        at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1023)
        at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:738)
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
        at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:300)
        at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:358)
        ... 7 more
Caused by: java.security.InvalidAlgorithmParameterException: Unknown curve name: 1.3.132.0.39
        at sun.security.ec.ECKeyPairGenerator.initialize(ECKeyPairGenerator.java:100)
        at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:674)
        at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:63)
        ... 19 more
tru

tru

2016-05-28 21:53

administrator   ~0026721

I don't have the issue on bare metal without avx support, upgraded from 6.7.

/proc/cpuinfo:
...
model name : AMD Turion(tm) II Neo N54L Dual-Core Processor
...
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp
 lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc extd_apicid pni monitor cx1
6 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowpr
efetch osvw ibs skinit wdt nodeid_msr npt lbrv svm_lock nrip_save
...

[tru@n54l ~]$ rpm -q kernel nss nss-util nss-tools nss-sysinit
kernel-2.6.32-573.26.1.el6.x86_64
kernel-2.6.32-642.el6.x86_64
nss-3.21.0-8.el6.x86_64
nss-util-3.21.0-2.el6.x86_64
nss-tools-3.21.0-8.el6.x86_64
nss-sysinit-3.21.0-8.el6.x86_64
[tru@n54l ~]$ uname -a
Linux n54l.home 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[tru@n54l ~]$ curl https://www.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
https://www.google.fr/?gfe_rd=cr&ei=rBJKV5SSB8rj8wfD0J-oAw.
</BODY></HTML>

[tru@n54l ~]$ grep -i avx /proc/cpuinfo
[tru@n54l ~]$ echo $?
1
barritel-gblades

barritel-gblades

2016-05-31 13:50

reporter   ~0026730

We had the same issue after a 'yum update --disablerepo=epel' and then trying to do a regular update.

# yum update
Loaded plugins: fastestmirror
Setting up Update Process
Loading mirror speeds from cached hostfile
Illegal instruction

# tail /var/log/messages
May 31 11:08:03 smtpout2 kernel: yum[1514] trap invalid opcode ip:7ff1742afd60 sp:7ffd78372ed8 error:0 in libfreeblpriv3.so[7ff17425d000+72000]

The machine is centos6 running as a virtual machine within xenserver. /proc/cpuinfo does not show avx support.

I setup a test box and was able to fully update it but excluded 3 packages so I still had these old versions :-
nss-3.21.0-0.3.el6_7.x86_64
nss-sysinit-3.21.0-0.3.el6_7.x86_64
nss-tools-3.21.0-0.3.el6_7.x86_64
'yum makecache' worked fine. I used makecache as it forces the https connected to the epel repo which was the one causing the issue.
As soon as I updated 'nss' with the other two updating as required dependencies I had the same problem.

I fixed my broken box by downloading the old versions of nss manually and using 'rpm -Uvh nss*' to downgrade.
lixdeg

lixdeg

2016-05-31 19:36

reporter   ~0026735

Redhat hadn't post an update since friday.
https://access.redhat.com/solutions/2313911

Looks like a harder issue or a seldom issue.

Is it the AVX or the AES flag.
I had a few machines on XEN with a provider. And nss had the issue when AES flag was available. No AVX flags on any Xen host available to the virtual machines.
locojohn

locojohn

2016-05-31 20:23

reporter   ~0026737

I too fixed it by using the older versions of nss*. Clearly, a bug on the CentOS/RedHat side.
support@viviotech.net

support@viviotech.net

2016-05-31 22:47

reporter   ~0026739

We were able to solve our issue by recovering "/usr/lib64/libssl3.so" from a backup that we had from before the nss update and overwriting the current file. The system still thinks that it is running the latest version of nss as well so yum will not try to update nss until there is a newer version at which point the new version may have the fix we need.

On one of our servers we did not have a backup of "/usr/lib64/libssl3.so" so we used the file from a different server. Not sure if this is the best practice but it fixed our issue and we have not seen any other issue arise.
gavin-markup

gavin-markup

2016-06-01 10:31

reporter   ~0026740

We had this exact issue with a Rackspace Cloud Server, I did the following steps to downgrade my NSS to the EPEL 6.7 RElease

Also export NSS_DISABLE_HW_GCM=1 did work nicely

cd /root/
wget http://mirror.rackspace.com/centos/6.7/os/x86_64/Packages/nss-sysinit-3.18.0-5.3.el6_6.x86_64.rpm
wget http://mirror.rackspace.com/centos/6.7/os/x86_64/Packages/nss-tools-3.18.0-5.3.el6_6.x86_64.rpm
wget http://mirror.rackspace.com/centos/6.7/os/x86_64/Packages/nss-3.18.0-5.3.el6_6.x86_64.rpm
rpm -Uvh --force nss*

yum install yum-plugin-versionlock
yum versionlock add! nss nss-sysinit nss-tools
applematt84

applematt84

2016-06-02 18:45

reporter   ~0026758

I, too, can confirm that using any binary that makes a call to NSS segfaults with a message like the following:

git-remote-http[2882] trap invalid opcode ip:7f35c5784d60 sp:7ffe90e41a38 error:0 in libfreeblpriv3.so[7f35c5732000+72000]

I am running CentOS 6.8 on a Xen hypervisor. I was able to resolve the issue by performing the following steps:

1) cd ~/
2) mkdir RPM-Downgrade; cd RPM-Downgrade
3) wget http://vault.centos.org/6.7/updates/x86_64/Packages/nss-3.21.0-0.3.el6_7.x86_64.rpm
4) wget http://vault.centos.org/6.7/updates/x86_64/Packages/nss-sysinit-3.21.0-0.3.el6_7.x86_64.rpm
5) wget http://vault.centos.org/6.7/updates/x86_64/Packages/nss-tools-3.21.0-0.3.el6_7.x86_64.rpm
6) wget http://vault.centos.org/6.7/updates/x86_64/Packages/nss-util-3.21.0-0.3.el6_7.x86_64.rpm
7) yum downgrade ./*.rpm

I verified services and applications are running as expected with the downgraded NSS, post-reboot.
applematt84

applematt84

2016-06-02 18:51

reporter   ~0026759

I forgot to state the following step in my notes above:

0) export NSS_DISABLE_HW_AES=1
[...] Proceed with steps 1 - 7
akuusela

akuusela

2016-06-06 09:24

reporter   ~0026795

We too have encountered an issue similar to that reported by support@viviotech.net. We run Wildfly 10 application server (http://wildfly.org) on top of openjdk.

2016-05-26 15:15:48,363 ERROR [org.xnio.nio] (default I/O-1) XNIO000011: Task io.undertow.protocols.ssl.SslConduit$4$1@a54c7d5 failed with an exception: java.lang.RuntimeException: Could not generate DH keypair
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1429)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:705)
        at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:608)
        at io.undertow.protocols.ssl.SslConduit.access$600(SslConduit.java:63)
        at io.undertow.protocols.ssl.SslConduit$4$1.run(SslConduit.java:982)
        at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:580)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:464)
Caused by: java.lang.RuntimeException: Could not generate DH keypair
        at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:68)
        at sun.security.ssl.ServerHandshaker.setupEphemeralECDHKeys(ServerHandshaker.java:1432)
        at sun.security.ssl.ServerHandshaker.trySetCipherSuite(ServerHandshaker.java:1219)
        at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1023)
        at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:738)
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:221)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
        at io.undertow.protocols.ssl.SslConduit$4.run(SslConduit.java:970)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.InvalidAlgorithmParameterException: Unknown curve name: 1.3.132.0.39
        at sun.security.ec.ECKeyPairGenerator.initialize(ECKeyPairGenerator.java:100)
        at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:674)
        at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:63)
        ... 14 more

Using https://ssllabs.com/ssltest on an affected server shows "Protocol or cipher suite mismatch" at handshake simulations for following clients

Android 4.0.4
Android 4.1.1
Android 4.2.2
Android 4.3
BingPreview Jan 2015
Googlebot Feb 2015
OpenSSL 1.0.1l
YandexBot Jan 2015

Changing libssl3.so to an earlier version does not resolve the issue but downgrading openjdk back to 1.8.0.91-0.b14.el6_7 does work.

yum --releasever=6.7 downgrade java-1.8.0-openjdk java-1.8.0-openjdk-headless
yum install yum-plugin-versionlock
yum versionlock java-1.8.0-openjdk java-1.8.0-openjdk-headless

This issue seems to be similar to one fixed in Fedora:

https://bugzilla.redhat.com/show_bug.cgi?id=1329342
kstange

kstange

2016-06-10 16:20

reporter   ~0026844

***Absolutely no warranty/support on any of this***

I've pushed this nss-softokn RPMs into a repo for internal use based on the original Mozilla fix pushed back in 2013. (https://bugzilla.mozilla.org/show_bug.cgi?id=940794)

https://mirror.steadfast.net/centos-steadfast/special/nssfix/

If you use this repo with yum-plugin-priorities and set it higher priority than base and updates, you'll get nss-softokn updates from here and not have them reverted by CentOS updates. However, I will discontinue this repo if/when RH pushes a fix upstream, so priorities may mask those package updates once that occurs.
TimL

TimL

2016-06-24 12:49

reporter   ~0026964

not only did these work;
    export NSS_DISABLE_HW_AES=1;
    export NSS_DISABLE_HW_GCM=1;
    export NSS_DISABLE_HW_AVX=1;

but also this works;
    export NSS_DISABLE_HW_GCM=0;
troyengel

troyengel

2016-07-12 18:56

reporter   ~0027049

Red Hat has publicly released the fix to this issue.

https://access.redhat.com/errata/RHBA-2016:1397
TimL

TimL

2016-07-13 21:31

reporter   ~0027054

I think we can close this now.
    export NSS_DISABLE_HW_GCM=0;yum upgrade
installed
    nss-softokn-3.14.3-23.3.el6_8.x86_64.rpm
which seems to have fixed the issue.
devhen

devhen

2016-07-15 21:34

reporter   ~0027066

Its fixed for me as well, with the latest nss updates.
info@ssntpl.com

info@ssntpl.com

2016-07-23 04:54

reporter   ~0027111

People who are still facing this issue, just run "NSS_DISABLE_HW_GCM=1 yum update", this will updated the NSS package to latest version in which the issue is already fixed.

Issue History

Date Modified Username Field Change
2016-05-26 15:10 pko New Issue
2016-05-26 18:06 bogdan.sh Note Added: 0026700
2016-05-26 20:32 pko Note Added: 0026702
2016-05-27 07:58 kaz Note Added: 0026705
2016-05-28 06:19 razyr Note Added: 0026717
2016-05-28 10:05 lpcollier Note Added: 0026718
2016-05-28 20:08 jissereitsma Note Added: 0026719
2016-05-28 21:27 support@viviotech.net Note Added: 0026720
2016-05-28 21:53 tru Note Added: 0026721
2016-05-31 13:50 barritel-gblades Note Added: 0026730
2016-05-31 19:36 lixdeg Note Added: 0026735
2016-05-31 20:23 locojohn Note Added: 0026737
2016-05-31 22:47 support@viviotech.net Note Added: 0026739
2016-06-01 10:31 gavin-markup Note Added: 0026740
2016-06-02 18:45 applematt84 Note Added: 0026758
2016-06-02 18:51 applematt84 Note Added: 0026759
2016-06-06 09:24 akuusela Note Added: 0026795
2016-06-10 16:20 kstange Note Added: 0026844
2016-06-24 12:49 TimL Note Added: 0026964
2016-07-12 18:56 troyengel Note Added: 0027049
2016-07-13 21:31 TimL Note Added: 0027054
2016-07-15 21:34 devhen Note Added: 0027066
2016-07-20 16:24 sandalle Tag Attached: 6.8
2016-07-20 16:24 sandalle Tag Attached: nss
2016-07-23 04:54 info@ssntpl.com Note Added: 0027111