0003479: Upon upgrade to CentOS 5.3 OpenSSH fails.

Notes
(0009021) tru (administrator) 2009-04-02 18:32	you don't seem to be running a genuine centos version, there was never a "OpenSSL 0.9.8e-fips-rhel5 " version.... from CentOS [tru@centos5-i386-vm ~]$ rpm -qa openssl\* openssh\* openssh-4.3p2-29.el5.i386 openssh-askpass-4.3p2-29.el5.i386 openssh-server-4.3p2-29.el5.i386 openssl-0.9.8e-7.el5.i686 openssl-devel-0.9.8e-7.el5.i386 openssh-clients-4.3p2-29.el5.i386

(0009022) range (administrator) 2009-04-02 18:36 edited on: 2009-04-02 18:38	I cannot reproduce that: [angenenr@shutdown ~]$ssh -V OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 [angenenr@shutdown ~]$rpm -q openssl openssl-0.9.8e-7.el5.x86_64 openssl-0.9.8e-7.el5.i686 [angenenr@shutdown ~]$rpm -q openssh-server openssh-server-4.3p2-29.el5.x86_64 [angenenr@shutdown ~]$ssh localhost Last login: Thu Apr 2 20:35:43 2009 from localhost.localdomain [...] What does "cat /proc/sys/crypto/fips_enabled" say on your system? Tru: There is that version now :)

(0009023) SirVic (reporter) 2009-04-02 18:42	@ tru : the CentOS is genuine, i installed the 5.1 DVD (downloaded from centos website) myself, and the yum repos were also the default ones. @ range : # cat /proc/sys/crypto/fips_enabled 0 however, i did the same on a server where the upgrade was successful (as in, ssh works fine) and it's still 0 so far, i have upgraded 6 servers, 3 of them work fine, 3 don't work, no idea why.

(0009030) 3inetworks (reporter) 2009-04-03 07:40	I have the same issue with one of my servers. I am using CentOS 64 bits. When using version 5.2, it's working fine. But after upgrading, could not ssh to the box, with the same error above

(0009036) range (administrator) 2009-04-03 08:34	Okay, which openssl version is on those boxes?

(0009040) stnoonan (reporter) 2009-04-03 13:13	I saw this on two dozen servers this morning. They were all running 4.3p2-29.el5, though some were x86_64 and others were i386. I've rolled all of the servers with failures back to 4.3p2-26.el5_2.1. Of our servers running CentOS, only four were unaffected. All systems are running openssl-0.9.8e-7.el5, however we have a package installing 0.9.8b-8.3.2.<our tag> into /usr/local/. For some reason, this ssh is linking against that! [root@hostname ~]# ldd /usr/sbin/sshd \| grep crypt libcrypto.so.6 => /usr/local/lib/libcrypto.so.6 (0x00249000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x0012b000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00fd3000) After I remove our packages, it links against the correct libcrypto and ssh works. [root@gsbackup ~]# ldd /usr/sbin/sshd \| grep crypt libcrypto.so.6 => /lib/libcrypto.so.6 (0x00193000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00533000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x0073d000) So I suppose in my case this ends up being a 'non-standard' setup. I fixed it by adding /lib{,64} and /usr/lib{,64} to /etc/ld.so.conf and running ldconfig. That being said, if you're going to change the library requirements, there should have been a jump in the SO version.

(0009041) range (administrator) 2009-04-03 13:48	Upstream ported FIPS certification (or the possibility to do so, I'm not sure about that) into the already existing openssl library - so a SO bump is not really possible there, as the version stays the same as before, only with "more capabilities".

(0009042) SirVic (reporter) 2009-04-03 13:56	my systems were all i386 so far, and i don't have openssl installed in /usr/local. # rpm -qa openssh\* openssl\* openssl-devel-0.9.8e-7.el5 openssh-clients-4.3p2-29.el5 openssh-4.3p2-29.el5 openssl-0.9.8e-7.el5 openssh-askpass-4.3p2-29.el5 openssh-server-4.3p2-29.el5 # ssh -vvv localhost OpenSSH_4.3p2, OpenSSL 0.9.8b 04 May 2006 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to localhost [127.0.0.1] port 22. debug1: Connection established. ssh: symbol lookup error: ssh: undefined symbol: FIPS_mode # ldd /usr/sbin/sshd \|grep crypt libcrypto.so.6 => /lib/libcrypto.so.6 (0x00110000) libcrypt.so.1 => /lib/libcrypt.so.1 (0x00896000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00335000) # uname -ir 2.6.18-128.1.1.el5 i386 from what i see, there is no difference in the output of ldd /usr/sbin/sshd in a working system and a non-working one. i tried diff`ing the rpm packages installed on a working system and those installed on a non-working system but there are many differences, i can't figure out what's causing it.

(0009043) range (administrator) 2009-04-03 14:37	Ummm. From your note: # ssh -vvv localhost OpenSSH_4.3p2, OpenSSL 0.9.8b 04 May 2006 On my host: [angenenr@shutdown ~]$ssh -vvv localhost OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 Can you spot the difference? Where does 0.9.8b come from? looks like that comes first in your path.

(0009048) SirVic (reporter) 2009-04-03 18:06	well, i got it to work, apparently, when the openssl is upgraded for some reason libcrypto.so.4 and libssl.so.4 are not replaced with libcrypto.so.0.9.8e and libssl.so.0.9.8e, thus the openssl is broken, i replaced the old libssl.so.4 with a symlink pointing to lib*.so.0.9.8e and now everything is working fine, problem is, since i've made symbolic links from the old so.4 and so.6 what will happen when a new version of openssl will come out, like 0.9.8f ? something is wrong here.

(0009059) range (administrator) 2009-04-04 11:44 edited on: 2009-04-04 11:46	libcrypto.so.4 and libssl.so.4 are from the openssl-0.9.7(!) compat package. And sshd is linked against libcrypto.so.6(!). So this still doesn't explain why your ssh picked up 0.9.8b (which was in earlier CentOS 5 versions, but also contained libcrypto.so.6).

(0009066) SirVic (reporter) 2009-04-04 16:15	sshd picked up 0.9.8b because that was the version of libssl.so.4 and libcrypto.so.4 which were not symbolic links (before or after the upgrade). although openssh uses libssl.so.6, /lib/libssl.so.6 was a symlink which pointed to libssl.so.4 which had the version 0.9.8b, thus by removing libssl.so.4 and correcting the symlink to point to libssl.so.0.9.8e everything was fixed, i'm pissed because i didn't see that openssh was using the bad libs, and ssl was among the first things i've checked. My guess is that when the openssl got upgraded somehow the symlinks were not updated and were still pointing to the old openssl libs which were also not removed like they should have, i had to manually delete libcrypto.so.4 and libssl.so.4 and correct the symlink. i guess the bug is in the upgrade of openssl and the fact that openssh stopped working was just an ugly side effect, as to why this only happens on some boxes and it doesn't happen on others, i have no ideea. regards, and thanks range for helping me fix this.

(0009071) range (administrator) 2009-04-04 19:33	But openssl 0.9.8* does not even contain libssl.so.4 and libcrypto.so.4, so this must be something else not caused by CentOS upgrades. libssl.so.4 and libcrypto.so.4 are in the openssl097a package. But there are no links from .6 to .4 there either.

(0009072) range (administrator) 2009-04-04 19:33	Does seem to be a local issue.

Issue History
Date Modified	Username	Field	Change
2009-04-02 18:27	SirVic	New Issue
2009-04-02 18:32	tru	Note Added: 0009021
2009-04-02 18:35	tru	Status	new => feedback
2009-04-02 18:36	range	Note Added: 0009022
2009-04-02 18:38	range	Note Edited: 0009022
2009-04-02 18:42	SirVic	Note Added: 0009023
2009-04-03 07:40	3inetworks	Note Added: 0009030
2009-04-03 08:34	range	Note Added: 0009036
2009-04-03 13:13	stnoonan	Note Added: 0009040
2009-04-03 13:48	range	Note Added: 0009041
2009-04-03 13:56	SirVic	Note Added: 0009042
2009-04-03 14:37	range	Note Added: 0009043
2009-04-03 18:06	SirVic	Note Added: 0009048
2009-04-04 11:44	range	Note Added: 0009059
2009-04-04 11:46	range	Note Edited: 0009059
2009-04-04 16:15	SirVic	Note Added: 0009066
2009-04-04 19:33	range	Note Added: 0009071
2009-04-04 19:33	range	Note Added: 0009072
2009-04-04 19:33	range	Status	feedback => closed
2009-04-04 19:33	range	Resolution	open => no change required

Relationships