0003884: CentOS 5.4 glibc causes crash of VMWare vmware-hostd process in VMWare 2.0.0 and 2.0.1

Notes
(0010045) Charlie Brady (reporter) 2009-10-08 19:46	Or do this before upgrading (adapted from http://communities.vmware.com/message/1364852): [^] * Create the directory /usr/lib/vmware/lib/libc.so.6 * Grab /lib64/libc-2.5.so (or /lib/libc-2.5.so in case you're running an 32 Bit host) and copy it to into /usr/lib/vmware/lib/libc.so.6 * Rename the file libc-2.5.so within /usr/lib/vmware/lib/libc.so.6 to libc.so.6 * Open /usr/sbin/vmware-hostd and add /usr/lib/vmware/lib/libc.so.6 to the LD_LIBRARY_PATH. I just added an "export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH" before the last line. * Restart your vmware services (our just the host)

(0010046) Charlie Brady (reporter) 2009-10-08 19:47	You should also complain to VMWare. Considering that RHEL5 is their recommended platform, they support it poorly, e.g. they expect selinux to be disabled.

(0010159) RevRagnarok (reporter) 2009-10-25 02:59 edited on: 2009-10-26 20:17	If you've already updated glibc like I have, a hybrid version of note 1 above is to do this: mkdir /usr/lib/vmware/lib/libc.so.6 cd /tmp mkdir tmp cd tmp wget http://mirror.centos.org/centos/5.3/os/x86_64/CentOS/glibc-2.5-34.x86_64.rpm [^] rpm2cpio glibc-2.5-34.x86_64.rpm \| cpio -ivd mv lib64/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6 Edit /usr/sbin/vmware-hostd as above. Wish me luck... I'm about to reboot.

(0010160) RevRagnarok (reporter) 2009-10-25 03:00 edited on: 2009-10-26 20:19	OK, I accidentally linked to another bug report. Also, the rpm2cpio shouldn't have the ../ leading the RPM file. [EDIT] It was not your error. Mantis did that. I removed the "#" to prevent it. Also removed the ../ for people who get there through searches or from outside links. -toracat

(0010195) LinuxETC (reporter) 2009-10-28 15:37	VMware just release v2.0.2 as of 26 Oct 2009. Does this resolve the original bug report with glibc v2.5-42 by chance?

(0010203) prgarcia (reporter) 2009-10-29 16:57	I am still having the same issue with VMware-server-2.0.2-203138.x86_64.

(0010213) nogin (reporter) 2009-10-30 21:52	While I do not see much vmware-hostd crashes, after the 5.3->5.4 upgrade, I keep having the vmware-vmx (the virtual machine itself) crash, which is much more unpleasant that "just" a hostd crash. Not sure if this is a related problem, but it also did not go away after the 2.0.1->2.0.2 upgrade of VMWare and installing the latest kernel update.

(0010214) RevRagnarok (reporter) 2009-10-30 22:23	nogin, I have had 100% uptime since the 5.4 upgrade with three VMs running on consumer grade hardware. First thing I would do is run memtest86+ or some other program that will beat on your system. If that's not the problem, you should open a new bug because this one is tracking the specific issue of glibc not working with vmware-hostd...

(0010358) jhaig (reporter) 2009-11-12 20:46	I don't know if I am having the same problem. I upgraded to 5.4 (x86_64) some time ago and have been running a couple of CentOS vms since, one 5.4 and the other 4.8 (both i386). I had no problems until yesterday, when I installed and set up munin on them. Since doing this, the 5.4 guest has crashed for no apparent reason twice in a day. The only major change I had to make was installing the rpmforge repositories on the host, to get the munin packages. Is there some easy way I can see if this bug is the problem I am seeing? Thanks.

(0010376) drennalls (reporter) 2009-11-18 08:07	@jhaig You might try adding "ulimit -c unlimited" just before the last line of /usr/sbin/vmware-hostd to enable core file generation. When/if a crash happens again you can examine the core file by doing... gdb /usr/lib/vmware/bin/vmware-hostd /path/to/core_file ..once in gdb do "thread apply all bt" to generate backtraces for all the threads. I've found the signature of the crash is "Program terminated with signal 6, Aborted." where one of the backtraces shows an abort() from freeing memory, e.g. Thread 1 (process 12674): #0 0xb7fd3402 in __kernel_vsyscall () 0000001 0x00909df0 in raise () from /lib/libc.so.6 0000002 0x0090b701 in abort () from /lib/libc.so.6 #3 0x0094ddc7 in free_check () from /lib/libc.so.6 #4 0x0094aa5b in free () from /lib/libc.so.6 #5 0xb76db211 in operator delete () from /usr/lib/vmware/lib/libstdc++.so.6/libstdc++.so.6 #6 0xb7db9cb1 in Vmomi::PropertyCollectorImpl::FilterImpl::AfterComputeUpdate () from /usr/lib/vmware/vmacore/libvmomi.so.1.0 ..snip.. HTH

(0010377) jhaig (reporter) 2009-11-18 08:42	@drennalls Thanks. I have already followed the steps given by RevRagnarok to fix the problem and I haven't had any more crashes (yet).

(0010391) ndelong (reporter) 2009-11-21 19:33	I can also confirm on fully updated CentOS 5.4 + VMware Server 2.0.2 rpm, using @Charlie Brady's export line in /usr/sbin/vmware-hostd and @RevRagnarok's steps seems to work. I can now access my VM's (and their consoles) without having to restart vmware-mgmt. Thanks gentlemen!

(0010436) PBrimacombe (reporter) 2009-12-01 19:34	RevRagnarok: "Edit /usr/sbin/vmware-hostd as above." what do you mean? I opened vmwre-hostd with vi - looks like I could do alot of damage Peter B.

(0010443) PBrimacombe (reporter) 2009-12-03 15:13	"Edit /usr/sbin/vmware-hostd as above." what do you mean? you mean to edit it according to Charlie Brady whose post is indeed above I did but I got the same result, a blank screen! hostd.log still has messages about Proxysvc, SSL handshake anyway, it's nice to know that this is a serious problem

(0010444) PBrimacombe (reporter) 2009-12-03 18:43	I got another box with MS Windows. I'll catch up later once VMware and CentOS sort this out.

(0010455) berrydejager (reporter) 2009-12-04 09:22	Thanks for the support on this ticket... i have followed the instructions from RevRagnarok and Charlie Brady and the system has been stable since. :-) again...

(0010553) MerrimackBob (reporter) 2009-12-19 03:34	The @CharlieBrady and @RevRagnarok suggestions do not change anything for me. From a clean build using CentOS 5.4 + VMware Server 2.0.2 rpm, the problem still is that secure service (e.g. https://<ip>:8333/) [^] hangs and the unsecure service (e.g. http://<ip>:8222/) [^] works great. Again, workaround provides no change

(0010555) MerrimackBob (reporter) 2009-12-19 20:51	Okay. I did a complete new clean build from CentOS 5.3 + VMware Server 2.0.2 rpm. Everything works fine. Performed the "exclude=2.5-42" in the yum.conf before running updates. After 16 hours of updating, the updates were complete. Same problem occurs with CentOS 5.3 with updates and 2.5-42 modules blocked. The secure service hangs and the unsecure service works great. I don't believe that the glibc is the problem, in this particular case. It appears to be a problem with getting past the CA certificate. - Bob

(0010556) toracat (developer) 2009-12-19 22:14	Bob, Could you try one (or both) of the following? (1) In your browser, delete certificate for VMWare and any reference to your server. (in firefox, Edit -> Preferences -> Advanced -> View Certificates and look at both 'Servers' and 'Authorities' tabs). Restart the browser (2) On your server running vmware, find the process running hostd (ps ax \| grep hostd) and kill it. Then run 'service vmware-mgmt restart'.

(0010557) MerrimackBob (reporter) 2009-12-19 23:29	Toracat, No change by deleting certificates for both 'Servers' and "Authorities". Unsecured still works, secured does not.

(0010558) toracat (developer) 2009-12-19 23:42	How about item (2)? Have you tried that one? Also, when you say "does not work", what exactly happens when you try to connect from the browser? Do you see "Loading..." but the page is blank? Do you get any errors logged somewhere in /var/log/vmware ? Please give us more details.

(0010559) MerrimackBob (reporter) 2009-12-19 23:59	I did both: (1) deleted both the 'Servers' and 'Authorities' certificates and (2) killed the hostd process (ps ax\|grep hostd \| kill -9 ####), then ran the 'service vmware-mgmt restart'. One the secured web access, when it 'no longer works', it says loading, there are no errors in /var/log/vmware, and it sits there forever. Again, on the secured access only. On the unsecured, http://myipaddress:8222/, [^] the login screen pops up almost immediately, and I can see my three container files. I have now blown that instance away, and am restarting with another clean build from CentOS 5.3 + VMware Server 2.0.2 rpm. I will exclude the 2.5-34 glibc updates and see if it is as early as that release. NOTE: One thing I did notice after the super-long updates from the 5.3 ISOs is that on the first time running VMware after the upgrade, it forced me to re-run the vmware-config.pl file. Odd. I will try stopping at other checkpoints along the way. I need a clean build of this for a brand new client, and cannot afford the screwup on the first visit to their office. - MerrimackBob

(0010568) drennalls (reporter) 2009-12-21 15:51	Re: the Loading... issue when using the web interface on the secure port (8333) This is a bit off-topic, no ? It's not due to the vmware-hostd crash which this bug report is for. FWIW I've run into is as well (intermittently) and from what digging I did it appeared to be a timing problem, basically it takes too long to load things up in the backend at times. When you access the web interface the traffic goes through vmware-hostd (either 8222 or 8333) to the 'ui' tomcat webapp. When I traced things it appeared that at times vmware-hostd would give up(timeout) when proxying the content from the webapp. You end up with 'Loading..' in the browser because the javascript libraries (jslib.js and wbc.js IIRC) are pretty huge and aren't fully downloaded when the timeout occurs. My guess was that there's more overhead with the secure port.. I ended up just using the unsecure port as well. There might be a config option (under /etc/vmware/hostd/*) to extend the timeout.

(0010689) herrold (administrator) 2010-01-06 14:40	NOTICE: A person reading reading this instructions in 2010, post the release of the stabilized CentOS 5.4 respin and updates has encountered problems with these instructions. They may or not still be applicable; they may eat your machine. The person asking 'manually downgrade[d] glibc' which according to their report, broke lots of package dependencies, including most of the security madel for authentication As this is a VMware support issue, and as VMware has not made their sources available and freely reproducible, please use the VMware support channels. -- Russ herrold

(0010791) bill_mcgonigle (reporter) 2010-01-18 21:50	Has anybody determined the root cause of this? Usually minor number bumps don't break ABI.

(0010877) bwlinux (reporter) 2010-01-28 16:36	If you update your 32 bit system before finding this thread you'll need to do the following mkdir /usr/lib/vmware/lib/libc.so.6 cd /tmp mkdir tmp cd tmp wget http://mirror.centos.org/centos/5.3/os/i386/CentOS/glibc-2.5-34.i686.rpm [^] rpm2cpio glibc-2.5-34.x86_64.rpm \| cpio -ivd mv lib/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6 Edit /usr/sbin/vmware-hostd, add an "export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH" before the last line. Then restart the vmware-mgmt with /etc/init.d/vmware-mgmt restart, of just restart the host. The i686 rpm is important, the i386 rpm won't work.

(0010878) Charlie Brady (reporter) 2010-01-28 16:46	> The i686 rpm is important, the i386 rpm won't work. Presumably neither will x86_64, which is what you are unpacking in your howto. I suggest that you repost it, with correction. Thanks.

(0010879) Charlie Brady (reporter) 2010-01-28 16:48	> Has anybody determined the root cause of this? VMWare is the party which must do that.

(0010880) bwlinux (reporter) 2010-01-28 16:50	Thanks Charlie Brady, I was using the lines from RevRagnarok. Here's the corrected post If you update your 32 bit system before finding this thread you'll need to do the following mkdir /usr/lib/vmware/lib/libc.so.6 cd /tmp mkdir tmp cd tmp wget http://mirror.centos.org/centos/5.3/os/i386/CentOS/glibc-2.5-34.i686.rpm [^] rpm2cpio glibc-2.5-34.i686.rpm \| cpio -ivd mv lib/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6 Edit /usr/sbin/vmware-hostd, add an "export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH" before the last line. Then restart the vmware-mgmt with /etc/init.d/vmware-mgmt restart, of just restart the host. The i686 rpm is important, the i386 rpm won't work.

Issue History
Date Modified	Username	Field	Change
2009-10-03 17:56	jhughes@hughesjr.com	New Issue
2009-10-03 17:56	jhughes@hughesjr.com	Assigned To	=> kbsingh@karan.org
2009-10-08 19:46	Charlie Brady	Note Added: 0010045
2009-10-08 19:47	Charlie Brady	Note Added: 0010046
2009-10-22 06:31	NETim	Issue Monitored: NETim
2009-10-22 14:29	villainy	Issue Monitored: villainy
2009-10-25 02:59	RevRagnarok	Note Added: 0010159
2009-10-25 03:00	RevRagnarok	Note Added: 0010160
2009-10-26 20:17	toracat	Note Edited: 0010159
2009-10-26 20:19	toracat	Note Edited: 0010160
2009-10-28 15:37	LinuxETC	Note Added: 0010195
2009-10-28 20:24	tljohnsn	Issue Monitored: tljohnsn
2009-10-29 16:57	prgarcia	Note Added: 0010203
2009-10-30 21:49	nogin	Issue Monitored: nogin
2009-10-30 21:52	nogin	Note Added: 0010213
2009-10-30 22:23	RevRagnarok	Note Added: 0010214
2009-11-12 02:58	kwanlowe	Issue Monitored: kwanlowe
2009-11-12 20:46	jhaig	Note Added: 0010358
2009-11-13 15:25	toracat	Relationship added	has duplicate 0003987
2009-11-15 09:44	jhaig	Issue Monitored: jhaig
2009-11-16 12:56	mfalb	Issue Monitored: mfalb
2009-11-18 08:07	drennalls	Note Added: 0010376
2009-11-18 08:42	jhaig	Note Added: 0010377
2009-11-20 11:43	wste	Issue Monitored: wste
2009-11-21 19:33	ndelong	Note Added: 0010391
2009-12-01 19:34	PBrimacombe	Note Added: 0010436
2009-12-03 15:13	PBrimacombe	Note Added: 0010443
2009-12-03 18:43	PBrimacombe	Note Added: 0010444
2009-12-04 09:22	berrydejager	Note Added: 0010455
2009-12-04 12:31	toracat	Status	new => assigned
2009-12-17 19:57	pjwelsh	Issue Monitored: pjwelsh
2009-12-19 02:43	MerrimackBob	Issue Monitored: MerrimackBob
2009-12-19 03:35	MerrimackBob	Note Added: 0010553
2009-12-19 20:51	MerrimackBob	Note Added: 0010555
2009-12-19 22:14	toracat	Note Added: 0010556
2009-12-19 23:29	MerrimackBob	Note Added: 0010557
2009-12-19 23:42	toracat	Note Added: 0010558
2009-12-19 23:59	MerrimackBob	Note Added: 0010559
2009-12-21 15:51	drennalls	Note Added: 0010568
2010-01-06 14:40	herrold	Note Added: 0010689
2010-01-18 21:50	bill_mcgonigle	Note Added: 0010791
2010-01-19 14:26	rogeliodh	Issue Monitored: rogeliodh
2010-01-28 16:36	bwlinux	Note Added: 0010877
2010-01-28 16:46	Charlie Brady	Note Added: 0010878
2010-01-28 16:48	Charlie Brady	Note Added: 0010879
2010-01-28 16:50	bwlinux	Note Added: 0010880
2010-02-01 14:22	tofo	Issue Monitored: tofo