CentOS Bug Tracker
CentOS Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0003884CentOS-5glibcpublic2009-10-03 17:562011-07-06 01:31
ReporterJohnnyHughes 
PrioritynormalSeveritycrashReproducibilityalways
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version5.6 
Summary0003884: CentOS 5.4 glibc causes crash of VMWare vmware-hostd process in VMWare 2.0.0 and 2.0.1
DescriptionCentOS 5.4 glibc causes crash of VMWare vmware-hostd process in VMWare 2.0.0 and 2.0.1. This is documented on these 2 VM threads:

http://communities.vmware.com/message/1364852 [^]

and

http://communities.vmware.com/thread/230842 [^]

Two ways to fix this are called out in the above threads. The easiest one is to just exclude glibc from CentOS-5.4 in the yum.conf file if you are upgrading a 5.3 machine, using this line:

exclude=*2.5-42*

Then do this to verify you will not get the new files in the upgrade:

yum list glibc*

The output should not show new glibc files.

===================
If you have already installed (or updated) to the new glibc files, you can manually download and install the older version of glibc* and/or ncsd from the 5.3 tree.

For a short time after 5.4 is released, this tree will be on the main centos mirrors:

http://mirror.centos.org/centos/5.3/ [^]

Later, you will still be able to get them from the CentOS Vault:

http://vault.centos.org/ [^]
TagsNo tags attached.
Attached Files

- Relationships
related to 0003987closedtoracat Vmware Guests crash since upgrades that included glib 2.5-42 & Kernel 2.6.18-164.2.1 

-  Notes
(0010045)
Charlie Brady (reporter)
2009-10-08 19:46
edited on: 2010-02-10 17:12

Or do this before upgrading (adapted from http://communities.vmware.com/message/1364852 [^] ) :

    * Create the directory /usr/lib/vmware/lib/libc.so.6
    * Grab /lib64/libc-2.5.so (or
      /lib/libc-2.5.so in case you're running an 32 Bit host) and copy it to
      into /usr/lib/vmware/lib/libc.so.6
    * Rename the file libc-2.5.so within /usr/lib/vmware/lib/libc.so.6 to libc.so.6
    * Open /usr/sbin/vmware-hostd and add /usr/lib/vmware/lib/libc.so.6 to the LD_LIBRARY_PATH. I just added an "export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH" before the last line.
    * Restart your vmware services (our just the host)

(0010046)
Charlie Brady (reporter)
2009-10-08 19:47

You should also complain to VMWare. Considering that RHEL5 is their recommended platform, they support it poorly, e.g. they expect selinux to be disabled.
(0010159)
RevRagnarok (reporter)
2009-10-25 02:59
edited on: 2009-10-26 20:17

If you've already updated glibc like I have, a hybrid version of note 1 above is to do this:

mkdir /usr/lib/vmware/lib/libc.so.6
cd /tmp
mkdir tmp
cd tmp
wget http://mirror.centos.org/centos/5.3/os/x86_64/CentOS/glibc-2.5-34.x86_64.rpm [^]
rpm2cpio glibc-2.5-34.x86_64.rpm | cpio -ivd
mv lib64/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6

Edit /usr/sbin/vmware-hostd as above.

Wish me luck... I'm about to reboot.

(0010160)
RevRagnarok (reporter)
2009-10-25 03:00
edited on: 2009-10-26 20:19

OK, I accidentally linked to another bug report. Also, the rpm2cpio shouldn't have the ../ leading the RPM file.

[EDIT] It was not your error. Mantis did that. I removed the "#" to prevent it. Also removed the ../ for people who get there through searches or from outside links. -toracat

(0010195)
LinuxETC (reporter)
2009-10-28 15:37

VMware just release v2.0.2 as of 26 Oct 2009. Does this resolve the original bug report with glibc v2.5-42 by chance?
(0010203)
prgarcia (reporter)
2009-10-29 16:57

I am still having the same issue with VMware-server-2.0.2-203138.x86_64.
(0010213)
nogin (reporter)
2009-10-30 21:52

While I do not see much vmware-hostd crashes, after the 5.3->5.4 upgrade, I keep having the vmware-vmx (the virtual machine itself) crash, which is much more unpleasant that "just" a hostd crash. Not sure if this is a related problem, but it also did not go away after the 2.0.1->2.0.2 upgrade of VMWare and installing the latest kernel update.
(0010214)
RevRagnarok (reporter)
2009-10-30 22:23

nogin, I have had 100% uptime since the 5.4 upgrade with three VMs running on consumer grade hardware. First thing I would do is run memtest86+ or some other program that will beat on your system. If that's not the problem, you should open a new bug because this one is tracking the specific issue of glibc not working with vmware-hostd...
(0010358)
jhaig (reporter)
2009-11-12 20:46

I don't know if I am having the same problem. I upgraded to 5.4 (x86_64) some time ago and have been running a couple of CentOS vms since, one 5.4 and the other 4.8 (both i386). I had no problems until yesterday, when I installed and set up munin on them. Since doing this, the 5.4 guest has crashed for no apparent reason twice in a day.

The only major change I had to make was installing the rpmforge repositories on the host, to get the munin packages. Is there some easy way I can see if this bug is the problem I am seeing?

Thanks.
(0010376)
drennalls (reporter)
2009-11-18 08:07

@jhaig
You might try adding "ulimit -c unlimited" just before the last line of /usr/sbin/vmware-hostd to enable core file generation. When/if a crash happens again you can examine the core file by doing...

gdb /usr/lib/vmware/bin/vmware-hostd /path/to/core_file

..once in gdb do "thread apply all bt" to generate backtraces for all the threads. I've found the signature of the crash is "Program terminated with signal 6, Aborted." where one of the backtraces shows an abort() from freeing memory, e.g.
Thread 1 (process 12674):
#0 0xb7fd3402 in __kernel_vsyscall ()
#1 0x00909df0 in raise () from /lib/libc.so.6
#2 0x0090b701 in abort () from /lib/libc.so.6
#3 0x0094ddc7 in free_check () from /lib/libc.so.6
#4 0x0094aa5b in free () from /lib/libc.so.6
#5 0xb76db211 in operator delete () from /usr/lib/vmware/lib/libstdc++.so.6/libstdc++.so.6
#6 0xb7db9cb1 in Vmomi::PropertyCollectorImpl::FilterImpl::AfterComputeUpdate () from /usr/lib/vmware/vmacore/libvmomi.so.1.0
..snip..

HTH
(0010377)
jhaig (reporter)
2009-11-18 08:42

@drennalls
Thanks. I have already followed the steps given by RevRagnarok to fix the problem and I haven't had any more crashes (yet).
(0010391)
ndelong (reporter)
2009-11-21 19:33

I can also confirm on fully updated CentOS 5.4 + VMware Server 2.0.2 rpm, using @Charlie Brady's export line in /usr/sbin/vmware-hostd and @RevRagnarok's steps seems to work.

I can now access my VM's (and their consoles) without having to restart vmware-mgmt.

Thanks gentlemen!
(0010436)
PBrimacombe (reporter)
2009-12-01 19:34

RevRagnarok: "Edit /usr/sbin/vmware-hostd as above."

what do you mean?

I opened vmwre-hostd with vi - looks like I could do alot of damage

Peter B.
(0010443)
PBrimacombe (reporter)
2009-12-03 15:13

"Edit /usr/sbin/vmware-hostd as above."

what do you mean?

you mean to edit it according to Charlie Brady whose post is indeed above

I did but I got the same result, a blank screen!

hostd.log still has messages about Proxysvc, SSL handshake

anyway, it's nice to know that this is a serious problem
(0010444)
PBrimacombe (reporter)
2009-12-03 18:43

I got another box with MS Windows. I'll catch up later once VMware and CentOS sort this out.
(0010455)
berrydejager (reporter)
2009-12-04 09:22

Thanks for the support on this ticket... i have followed the instructions from RevRagnarok and Charlie Brady and the system has been stable since.

:-) again...
(0010553)
MerrimackBob (reporter)
2009-12-19 03:34

The @CharlieBrady and @RevRagnarok suggestions do not change anything for me. From a clean build using CentOS 5.4 + VMware Server 2.0.2 rpm, the problem still is that secure service (e.g. https://<ip>:8333/ [^]) hangs and the unsecure service (e.g. http://<ip>:8222/ [^]) works great. Again, workaround provides no change
(0010555)
MerrimackBob (reporter)
2009-12-19 20:51

Okay. I did a complete new clean build from CentOS 5.3 + VMware Server 2.0.2 rpm. Everything works fine. Performed the "exclude=*2.5-42*" in the yum.conf before running updates. After 16 hours of updating, the updates were complete.

Same problem occurs with CentOS 5.3 with updates and 2.5-42 modules blocked. The secure service hangs and the unsecure service works great. I don't believe that the glibc is the problem, in this particular case. It appears to be a problem with getting past the CA certificate. - Bob
(0010556)
toracat (developer)
2009-12-19 22:14

Bob,

Could you try one (or both) of the following?

(1) In your browser, delete certificate for VMWare and any reference to your server. (in firefox, Edit -> Preferences -> Advanced -> View Certificates and look at both 'Servers' and 'Authorities' tabs). *Restart the browser*

(2) On your server running vmware, find the process running hostd (ps ax | grep hostd) and kill it. Then run 'service vmware-mgmt restart'.
(0010557)
MerrimackBob (reporter)
2009-12-19 23:29

Toracat,

No change by deleting certificates for both 'Servers' and "Authorities". Unsecured still works, secured does not.
(0010558)
toracat (developer)
2009-12-19 23:42

How about item (2)? Have you tried that one?

Also, when you say "does not work", what exactly happens when you try to connect from the browser? Do you see "Loading..." but the page is blank? Do you get any errors logged somewhere in /var/log/vmware ? Please give us more details.
(0010559)
MerrimackBob (reporter)
2009-12-19 23:59

I did both: (1) deleted both the 'Servers' and 'Authorities' certificates and (2) killed the hostd process (ps ax|grep hostd | kill -9 ####), then ran the 'service vmware-mgmt restart'.

One the secured web access, when it 'no longer works', it says loading, there are no errors in /var/log/vmware, and it sits there forever. Again, on the secured access only.

On the unsecured, http://myipaddress:8222/, [^] the login screen pops up almost immediately, and I can see my three container files.

I have now blown that instance away, and am restarting with another clean build from CentOS 5.3 + VMware Server 2.0.2 rpm. I will exclude the 2.5-34 glibc updates and see if it is as early as that release.

NOTE: One thing I did notice after the super-long updates from the 5.3 ISOs is that on the first time running VMware *after* the upgrade, it forced me to re-run the vmware-config.pl file. Odd. I will try stopping at other checkpoints along the way. I need a clean build of this for a brand new client, and cannot afford the screwup on the first visit to their office. - MerrimackBob
(0010568)
drennalls (reporter)
2009-12-21 15:51

Re: the Loading... issue when using the web interface on the secure port (8333)

This is a bit off-topic, no ? It's not due to the vmware-hostd crash which this bug report is for.

FWIW I've run into is as well (intermittently) and from what digging I did it appeared to be a timing problem, basically it takes too long to load things up in the backend at times. When you access the web interface the traffic goes through vmware-hostd (either 8222 or 8333) to the 'ui' tomcat webapp. When I traced things it appeared that at times vmware-hostd would give up(timeout) when proxying the content from the webapp. You end up with 'Loading..' in the browser because the javascript libraries (jslib.js and wbc.js IIRC) are pretty huge and aren't fully downloaded when the timeout occurs. My guess was that there's more overhead with the secure port.. I ended up just using the unsecure port as well. There might be a config option (under /etc/vmware/hostd/*) to extend the timeout.
(0010689)
herrold (reporter)
2010-01-06 14:40

NOTICE: A person reading reading this instructions in 2010, post the release of the stabilized CentOS 5.4 respin and updates has encountered problems with these instructions.

They may or not still be applicable; they may eat your machine. The person asking 'manually downgrade[d] glibc' which according to their report, broke lots of package dependencies, including most of the security madel for authentication

As this is a VMware support issue, and as VMware has not made their sources available and freely reproducible, please use the VMware support channels.

-- Russ herrold
(0010791)
bill_mcgonigle (reporter)
2010-01-18 21:50

Has anybody determined the root cause of this? Usually minor number bumps don't break ABI.
(0010877)
bwlinux (reporter)
2010-01-28 16:36

If you update your 32 bit system before finding this thread you'll need to do the following

mkdir /usr/lib/vmware/lib/libc.so.6
cd /tmp
mkdir tmp
cd tmp
wget http://mirror.centos.org/centos/5.3/os/i386/CentOS/glibc-2.5-34.i686.rpm [^]
rpm2cpio glibc-2.5-34.x86_64.rpm | cpio -ivd
mv lib/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6

Edit /usr/sbin/vmware-hostd,
add an "export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH" before the last line.

Then restart the vmware-mgmt with /etc/init.d/vmware-mgmt restart, of just restart the host.

The i686 rpm is important, the i386 rpm won't work.
(0010878)
Charlie Brady (reporter)
2010-01-28 16:46

> The i686 rpm is important, the i386 rpm won't work.

Presumably neither will x86_64, which is what you are unpacking in your howto. I suggest that you repost it, with correction.

Thanks.
(0010879)
Charlie Brady (reporter)
2010-01-28 16:48

> Has anybody determined the root cause of this?

VMWare is the party which must do that.
(0010880)
bwlinux (reporter)
2010-01-28 16:50

Thanks Charlie Brady, I was using the lines from RevRagnarok. Here's the corrected post

If you update your 32 bit system before finding this thread you'll need to do the following

mkdir /usr/lib/vmware/lib/libc.so.6
cd /tmp
mkdir tmp
cd tmp
wget http://mirror.centos.org/centos/5.3/os/i386/CentOS/glibc-2.5-34.i686.rpm [^]
rpm2cpio glibc-2.5-34.i686.rpm | cpio -ivd
mv lib/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6

Edit /usr/sbin/vmware-hostd,
add an "export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH" before the last line.

Then restart the vmware-mgmt with /etc/init.d/vmware-mgmt restart, of just restart the host.

The i686 rpm is important, the i386 rpm won't work.
(0010964)
jameswilson (reporter)
2010-02-10 14:58

Thanks for the above but im still having problems with 32 bit CENTOS 5.4 and vmware 2.0.2.

log from last crash
Feb 10 06:13:27.378: vmx| DISKLIB-LIB : numIOs = 1550000 numMergedIOs = 0 numSplitIOs = 0
Feb 10 08:39:00.827: vmx| DISKLIB-LIB : numIOs = 1600000 numMergedIOs = 0 numSplitIOs = 0
Feb 10 10:12:44.999: vmx| DISKLIB-LIB : numIOs = 1650000 numMergedIOs = 0 numSplitIOs = 0
Feb 10 11:43:24.188: vmx| DISKLIB-LIB : numIOs = 1700000 numMergedIOs = 0 numSplitIOs = 0
Feb 10 12:36:46.504: vcpu-0| Not caching pages 4691527 through 4691528 because of write on page 4691527 on write.
Feb 10 12:56:34.597: vmx| DISKLIB-LIB : numIOs = 1750000 numMergedIOs = 0 numSplitIOs = 0
Feb 10 12:57:06.326: Worker#0| Caught signal 6 -- tid 7685
Feb 10 12:57:06.326: Worker#0| SIGNAL: eip 0x117410 esp 0xb6553014 ebp 0xb655302c
Feb 10 12:57:06.326: Worker#0| SIGNAL: eax 0x0 ebx 0x1e04 ecx 0x1e05 edx 0x6 esi 0xb65530cc edi 0x259ff4
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553014 : 0xb655302c 0x00000006 0x00001e05 0x00140df0
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553024 : 0x00259ff4 0xb6553b90 0xb6553158 0x00142701
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553034 : 0x00000006 0xb65530cc 0x00000000 0x00000005
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553044 : 0xb7fcf380 0x0023ea65 0x00000000 0x0023ea65
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553054 : 0x0025b140 0x00000019 0x0025b164 0x0025b164
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553064 : 0x00000000 0x0025b170 0x00000038 0x0025b164
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553074 : 0x00000000 0x00000003 0x00029000 0x0017f7eb
Feb 10 12:57:06.326: Worker#0| SIGNAL: stack B6553084 : 0x00259ff4 0x00009000 0x00009000 0xb655315c
Feb 10 12:57:06.326: Worker#0| Backtrace:
Feb 10 12:57:06.326: Worker#0| Backtrace[0] 0xb6552bc8 eip 0x8052690
Feb 10 12:57:06.326: Worker#0| Backtrace[1] 0xb6552c88 eip 0x80a71e0
Feb 10 12:57:06.326: Worker#0| Backtrace[2] 0xb655302c eip 0x117440
Feb 10 12:57:06.326: Worker#0| Backtrace[3] 0xb6553158 eip 0x142701
Feb 10 12:57:06.326: Worker#0| Backtrace[4] 0xb6553190 eip 0x184dc7
Feb 10 12:57:06.326: Worker#0| Backtrace[5] 0xb65531c8 eip 0x181a5b
Feb 10 12:57:06.326: Worker#0| Backtrace[6] 0xb6553228 eip 0x80bd1d4
Feb 10 12:57:06.326: Worker#0| Backtrace[7] 0xb6553278 eip 0x819c408
Feb 10 12:57:06.326: Worker#0| Backtrace[8] 0xb65532b8 eip 0x8139cff
Feb 10 12:57:06.326: Worker#0| Backtrace[9] 0xb65532e8 eip 0x8139f07
Feb 10 12:57:06.326: Worker#0| Backtrace[10] 0xb65533c8 eip 0x80c338d
Feb 10 12:57:06.326: Worker#0| Backtrace[11] 0xb65534b8 eip 0xcf573b
Feb 10 12:57:06.326: Worker#0| Backtrace[12] 00000000 eip 0x1e9cfe
Feb 10 12:57:06.326: Worker#0| SymBacktrace[0] 0xb6552bc8 eip 0x8052690 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.326: Worker#0| SymBacktrace[1] 0xb6552c88 eip 0x80a71e0 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.326: Worker#0| SymBacktrace[2] 0xb655302c eip 0x117440 in function __kernel_rt_sigreturn in object loaded at 0x117000
Feb 10 12:57:06.326: Worker#0| SymBacktrace[3] 0xb6553158 eip 0x142701 in function abort in object /lib/libc.so.6 loaded at 0x118000
Feb 10 12:57:06.326: Worker#0| SymBacktrace[4] 0xb6553190 eip 0x184dc7 in function (null) in object /lib/libc.so.6 loaded at 0x118000
Feb 10 12:57:06.326: Worker#0| SymBacktrace[5] 0xb65531c8 eip 0x181a5b in function cfree in object /lib/libc.so.6 loaded at 0x118000
Feb 10 12:57:06.326: Worker#0| SymBacktrace[6] 0xb6553228 eip 0x80bd1d4 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.327: Worker#0| SymBacktrace[7] 0xb6553278 eip 0x819c408 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.327: Worker#0| SymBacktrace[8] 0xb65532b8 eip 0x8139cff in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.327: Worker#0| SymBacktrace[9] 0xb65532e8 eip 0x8139f07 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.327: Worker#0| SymBacktrace[10] 0xb65533c8 eip 0x80c338d in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0x8048000
Feb 10 12:57:06.327: Worker#0| SymBacktrace[11] 0xb65534b8 eip 0xcf573b in function (null) in object /lib/libpthread.so.0 loaded at 0xcf0000
Feb 10 12:57:06.327: Worker#0| SymBacktrace[12] 00000000 eip 0x1e9cfe in function clone in object /lib/libc.so.6 loaded at 0x118000
Feb 10 12:57:06.327: Worker#0| Unexpected signal: 6.
Feb 10 12:57:06.327: Worker#0| Core dump limit is 0 KB.
Feb 10 12:57:06.332: Worker#0| Child process 24539 failed to dump core (status 0x6).
Feb 10 12:57:06.332: Worker#0| Backtrace:
Feb 10 12:57:06.332: Worker#0| Backtrace[0] 0xb6552798 eip 0x8052690

guest is SME server wich is based on CentOS 4.x

Any further suggestions? A rollback on glibc isnt really an option as the host is in a hosted rack and im fearful of it not coming backup with something so major.

Many Thanks
(0010965)
Charlie Brady (reporter)
2010-02-10 16:31

> log from last crash

James, your issue is different. Please do not hijack this bug report.

> A rollback on glibc isnt really an option as the host is in a hosted
> rack and im fearful of it not coming backup with something so major.

The suggestion here isn't to roll back glibc anyway - just to install a special version of glibc just for vmware-hostd.
(0010966)
jameswilson (reporter)
2010-02-10 17:03

Charlie.
Im not trying to hijack anything. My problem is the same and the suggested fix doesnt solve it. It seems to help though.
As the log shows above im getting problems from libc.so.6, so why is my issue different. I know via contribs you have said ive got a hardware issue, but i have run memtest for 6 hours, with no errors. Nor do i have any issues on the host. Only with vm's.
If my issue is not related to this (and you may be correct), then what would you suggest i do when it appears that it is related to this.
(0010967)
toracat (developer)
2010-02-10 17:21

@jameswilson

Your issue may be more relevant in the related bug #3987 .
(0010974)
rul3z (reporter)
2010-02-11 19:51

I have exactly the same issue here (using CentOS 5.4 + vmware-server 2.0.2). I tried all the solutions the guys provide and still didn't help !!!

The problem is that it seems that VMWARE doesn't seem to care about their product, specially when its a free one !!! I doubt this would of happened to any of their other paid products!

Wish I manage to solve this once and for all, because its annoying to keep restarting the vmware-mgmt to be able to login again !!!!
(0010976)
rul3z (reporter)
2010-02-11 20:02

I forgot to say that I am using the 64bit version of both CentOS 5.4 and Vmware-server
(0011012)
crefeld (reporter)
2010-02-17 11:59

As CentOS 5.3 has been removed from the mirrors the workaround described above doesn't work any more. Hopefully in near future someone will put the subtree into http://vault.centos.org/5.3/ [^]
(0011017)
SJUTECH (reporter)
2010-02-18 16:51

Here is how I fixed my issues: http://www.davidmarkley.com/vmware/vmware-server-2-on-centos-5-4 [^]


VMware Server 2 on CentOS 5.4
November 20, 2009 – 9:47 am Well, I’m sure that there are many people that are running VMware Server 2 on CentOS 5. After all, it’s one of the major Host OSes that VMware recognizes. Popularity notwithstanding, there is a major bug that can bring your VM screamer to a hault.

CentOS 5.4 has a new glibc package that essentially breaks VMware Server’s hostd process. There are many posts out there regarding the issues, and various means of fixing them. However, I am just going to summarize info I’ve found out there on the net, and hopefully you should be able to follow very easily and get your VMs back up and running.

PROBLEM:
VMware Server 2 (hostd) crashes on CentOS 5 after upgrading to the latest releases of glibc and glibc-common

NOTE ON SOLUTIONS: There are two methods to solve this. The first requires downgrading the libraries system-wide. This should be fine if you only use the CentOS host as a VMware Server Host and nothing else. However, if you are in doubt whether your other applications, etc. on that host will run on a slightly older version of glibc, please use SOLUTION METHOD 2 as it will only affect VMware, and essentially tell VMware Server where to look for the correct libraries it needs.

Disclaimer: I am not responsible for you rendering your server useless. When in doubt, don’t upgrade to CentOS 5.4… Although, you probably already did that, and this is why you’re here.


SOLUTION METHOD 1:
Step 1: Go to /etc/yum.repos.d and copy the file CentOS-Base.repo to CentOS53-Base.repo

Step 2: In CentOS53-Base.repo, rename all the packages to reflect the 5.3 version. So, change:

[base] --> [base53]
[updates] --> [updates53]
[addons] --> [addons53]
[extras] --> [extras53]
[centosplus] --> [centosplus53]
[contrib] --> [contrib53]Step 3: In CentOS53-Base.repo, replace all instances of release=$releasever with release=5.3

Step 4: Now, downgrade the glibc and glibc-common libraries by running the following commands:

yum downgrade glibc glibc-commonStep 5: To avoid any problems with future upgrades/updates, it would be best to exclude them from the list of available updates on yum. Add the following to your /etc/yum.conf file:


exclude=glibc glibc-common glibc-devel glibc-headers glibc-utils nscdStep 6: Reboot the server, and now re-run /usr/bin/vmware-config.pl to reconfigure with downgraded glibc libraries.

/usr/bin/vmware-config.plStep 7: (Optional) Run the following command to make sure future upgrades/updates will not download the updated glibc* libraries.

yum list glibc*Now, if you have other applications installed on CentOS, and you don’t want to worry about any issues with future use of glibc on your server, please follow the next method to manually link VMware hostd process to use the older glibc libraries.

SOLUTION METHOD 2:
(Make sure you are logged in as root for these steps)

Step 1: Install the latest 2.0.2 VMware Server package and run the configuration. It will crash, but just ignore this for now.

Step 2: Run the following command, and make note of the response.

arch

Step3: Run the following commands, and replace any instance of {ARCH} with the result of Step 2:

mkdir ~/vmwareglibc
cd ~/vmwareglibc
wget http://mirror.centos.org/centos/5.3/os/{ARCH}/CentOS/glibc-2.5-34.{ARCH}.rpm [^]
rpm2cpio glibc-2.5-34.{ARCH}.rpm | cpio -ivd
mkdir /usr/lib/vmware/lib/libc.so.6
mv lib64/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6Step 4: Open the VMware hostd process script for editing.

vim /usr/sbin/vmware-hostdStep 5: At line 372, before the program is called, insert two empty lines and add the following:

export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATHBefore Example:

if [ ! "@@VMWARE_NO_MALLOC_CHECK@@" = 1 ]; then
     export MALLOC_CHECK_=2
fi
 
eval exec "$DEBUG_CMD" "$binary" "$@"After Example:

if [ ! "@@VMWARE_NO_MALLOC_CHECK@@" = 1 ]; then
     export MALLOC_CHECK_=2
fi
 
export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH
 
eval exec "$DEBUG_CMD" "$binary" "$@"NOTE: In Step 3, the wget command may in the future be this as 5.3 repositories are taken off of the main CentOS Mirrors:

wget http://vault.centos.org/5.3/os/{ARCH}/CentOS/glibc-2.5-34.{ARCH}.rpmEither [^] method you use should get you the desired end result: VMware Server 2 running on CentOS 5.4

Be sure to comment if this helped you. Thanks.


This saved the day for me.... I used Method 1 as I ahd already instlled and ran the YUM updates.
(0011039)
toracat (developer)
2010-02-27 04:34

> As CentOS 5.3 has been removed from the mirrors the workaround described above
> doesn't work any more. Hopefully in near future someone will put the subtree
> into http://vault.centos.org/5.3/ [^]

The 5.3 tree is now available at vault.centos.org.
(0011129)
nqdave (reporter)
2010-04-07 00:22

Followed METHOD 2 posted by SJUTECH and it works perfectly.

My env:
Centos 5.4 x86_64
VMware-server-2.0.2-203138.x86_64

As a last step had to re-run /usr/bin/vmware-config.pl and selected all defaults. No changes were made to the VMWare config files (the NEW_xxxx.xml files were identical to the originals). I rebooted (this probably wasn't necessary) and VMWare is working as it should.

Thanks for your posts. They've save me a LOT of time.
(0011157)
tomerb (reporter)
2010-04-16 06:45

I just applied "SOLUTION METHOD 2" above and it does _not_ work for me. :-(

CentOS release 5.4 (Final) i686
VMware Server 2.0.0 build-122956
glibc-2.5-34.i686.rpm installed as described

I have not (yet) upgraded to VMware Server 2.0.2 as this version seemed to be even more unstable (before applying the patch). Is there a chance that upgrade _and_ patch might solve the issue?
(0011174)
tru (administrator)
2010-04-27 15:54

just a reminder, http://www.vmware.com/support/policies/lifecycle/general/ [^]

VMware Server General Availability End of General Support
                (YYYY/MM/DD) (YYYY/MM/DD)
Version 2.x 2008/09/23 2011/06/30
Version 1.x 2006/07/12 2010/03/23

(I guess I can drop my vmware server 1.x and start looking at esxi)
(0011261)
airoctive (reporter)
2010-05-15 14:20

Hey Folks,

Prior to applying updates yesterday to my centos 5.4 os I had a working centos 5.4/VMware Server 2.02 system. Yes, I did experience the intermittent vmware-hostd crashes but the "well known" fix below solved the issue:

----------------
rpm2cpio glibc-2.5-34.XXX.rpm | cpio -ivd
mkdir /usr/lib/vmware/lib/libc.so.6
mv libXX/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6

Now we need to edit the vmware script
vim /usr/sbin/vmware-hostd
At line 372 insert a couple of empty lines and add the following before the program execution
export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH
------------------

However after applying all updates to the system(about 55 new packages including new glibc libs) yesterday, vmware-hostd now crashes during service startup. I figured I'd try to downgrade the glibc and blic-common libs back to the centos 5.4 originals 2.5.42 to make it work. I did that but the same issue, wont start at all, below is the kernel (dmesg) error log:


vmware-hostd[3868]: segfault at 0000000000000038 rip 00000000006e3bd1 rsp 00007fff44e667c0 error 4
(0011262)
Charlie Brady (reporter)
2010-05-15 15:01

> However after applying all updates to the system(about 55 new packages
> including new glibc libs) yesterday, vmware-hostd now crashes during
> service startup.

This is a separate problem, which should be reported separately.

The problem should also be reported to VMWare - it is their problem, and they should fix it.
(0011365)
Phil Schaffner (reporter)
2010-05-28 15:15

A recent forum post has workarounds for both issues reported here:

https://www.centos.org/modules/newbb/viewtopic.php?post_id=108087&topic_id=26431&forum=37 [^]
(0011367)
Charlie Brady (reporter)
2010-05-28 15:58

> A recent forum post has workarounds for both issues reported here:

Let's call it "METHOD 3e", and transcribe to here.



There is another option which does not involve maintaining the old glibc libraries which I found here (solution 2):
http://webalution.com/techshare/2009/11/16/vmware-server-2-web-access-connection-loss-vmware-hostd-crash-workarounds/ [^]
You modify the vmware startup script (/etc/init.d/vmware) to avoid the dynamic loading of the library and start the vmware-hostd process directly: The modifications to the script are shown below:

# Start host agent
vmware_start_hostd() {
    export LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
   vmware_bg_exec "`vmware_product_name` Host Agent" \
      "$vmdb_answer_LIBDIR/vmware-hostd" -a -d -u "$vmware_etc_dir/hostd/config.xml"
# "$vmdb_answer_SBINDIR/vmware-hostd" -a -d -u "$vmware_etc_dir/hostd/config.xml"
}


This fix has given me a stable interface to the vmware system and my virtual machines for over a week. Your mileage may vary.
(0012042)
jasonwu (reporter)
2010-11-12 03:47

I use vmware server 2.02 in cent os 5.4?I meet the same bug.
[root@localhost ~]# mkdir /usr/lib/vmware/lib/libc.so.6
[root@localhost ~]# cd /tmp
[root@localhost tmp]# mkdir tmp
[root@localhost tmp]# cd tmp
[root@localhost tmp]# wgethttp://vault.centos.org/5.3/os/x86_64/CentOS/glibc-2.5-34.x86_64.rpm [^]
[root@localhost tmp]# rpm2cpio glibc-2.5-34.x86_64.rpm | cpio -ivd
[root@localhost tmp]# mv lib64/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6
[root@localhost tmp]# vi /usr/sbin/vmware-hostd
and add export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH to the vmware-hostd

The problem is still exist.
(0012043)
jasonwu (reporter)
2010-11-12 03:49

PS ?I use cent os 5.4 -x64
(0012904)
RevRagnarok (reporter)
2011-07-06 01:23

Looks like CentOS 5.6 fixes the problem... in fact, the release notes at http://x.co/Y6cz [^] specifically say to undo the hack. I've been running for a few days now with no issues.
(0012905)
toracat (developer)
2011-07-06 01:31

Closing as 'fixed in CentOS 5.6'.

- Issue History
Date Modified Username Field Change
2009-10-03 17:56 JohnnyHughes New Issue
2009-10-08 19:46 Charlie Brady Note Added: 0010045
2009-10-08 19:47 Charlie Brady Note Added: 0010046
2009-10-25 02:59 RevRagnarok Note Added: 0010159
2009-10-25 03:00 RevRagnarok Note Added: 0010160
2009-10-26 20:17 toracat Note Edited: 0010159
2009-10-26 20:19 toracat Note Edited: 0010160
2009-10-28 15:37 LinuxETC Note Added: 0010195
2009-10-29 16:57 prgarcia Note Added: 0010203
2009-10-30 21:52 nogin Note Added: 0010213
2009-10-30 22:23 RevRagnarok Note Added: 0010214
2009-11-12 20:46 jhaig Note Added: 0010358
2009-11-13 15:25 toracat Relationship added has duplicate 0003987
2009-11-18 08:07 drennalls Note Added: 0010376
2009-11-18 08:42 jhaig Note Added: 0010377
2009-11-21 19:33 ndelong Note Added: 0010391
2009-12-01 19:34 PBrimacombe Note Added: 0010436
2009-12-03 15:13 PBrimacombe Note Added: 0010443
2009-12-03 18:43 PBrimacombe Note Added: 0010444
2009-12-04 09:22 berrydejager Note Added: 0010455
2009-12-04 12:31 toracat Status new => assigned
2009-12-19 03:35 MerrimackBob Note Added: 0010553
2009-12-19 20:51 MerrimackBob Note Added: 0010555
2009-12-19 22:14 toracat Note Added: 0010556
2009-12-19 23:29 MerrimackBob Note Added: 0010557
2009-12-19 23:42 toracat Note Added: 0010558
2009-12-19 23:59 MerrimackBob Note Added: 0010559
2009-12-21 15:51 drennalls Note Added: 0010568
2010-01-06 14:40 herrold Note Added: 0010689
2010-01-18 21:50 bill_mcgonigle Note Added: 0010791
2010-01-28 16:36 bwlinux Note Added: 0010877
2010-01-28 16:46 Charlie Brady Note Added: 0010878
2010-01-28 16:48 Charlie Brady Note Added: 0010879
2010-01-28 16:50 bwlinux Note Added: 0010880
2010-02-09 22:45 toracat Relationship replaced related to 0003987
2010-02-10 14:58 jameswilson Note Added: 0010964
2010-02-10 16:31 Charlie Brady Note Added: 0010965
2010-02-10 17:03 jameswilson Note Added: 0010966
2010-02-10 17:12 toracat Note Edited: 0010045
2010-02-10 17:12 toracat Note Edited: 0010045
2010-02-10 17:21 toracat Note Added: 0010967
2010-02-11 19:51 rul3z Note Added: 0010974
2010-02-11 20:02 rul3z Note Added: 0010976
2010-02-17 11:59 crefeld Note Added: 0011012
2010-02-18 16:51 SJUTECH Note Added: 0011017
2010-02-27 04:34 toracat Note Added: 0011039
2010-04-07 00:22 nqdave Note Added: 0011129
2010-04-16 06:45 tomerb Note Added: 0011157
2010-04-27 15:54 tru Note Added: 0011174
2010-05-15 14:20 airoctive Note Added: 0011261
2010-05-15 15:01 Charlie Brady Note Added: 0011262
2010-05-28 15:15 Phil Schaffner Note Added: 0011365
2010-05-28 15:58 Charlie Brady Note Added: 0011367
2010-11-12 03:47 jasonwu Note Added: 0012042
2010-11-12 03:49 jasonwu Note Added: 0012043
2011-07-06 01:23 RevRagnarok Note Added: 0012904
2011-07-06 01:31 toracat Note Added: 0012905
2011-07-06 01:31 toracat Status assigned => resolved
2011-07-06 01:31 toracat Resolution open => fixed
2011-07-06 01:31 toracat Fixed in Version => 5.6


Copyright © 2000 - 2016 MantisBT Team
Powered by Mantis Bugtracker