0002317: CentOS 5 domU randomly fails to start - CentOS Bug Tracker

ID	Project	Category	View Status	Date Submitted	Last Update

0002317	CentOS-5	xen	public	2007-09-05 17:49	2007-10-16 15:49

Reporter	colo4jax	Assigned To
Priority	normal	Severity	minor	Reproducibility	random
Status	assigned	Resolution	open
Product Version	5.0 - i386

Summary	0002317: CentOS 5 domU randomly fails to start
Description	On occasion when starting a CentOS 5 domU, the start fails due to a kernel panic as follows: Linux version 2.6.18-8.1.8.el5xen (mockbuild@builder4.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Tue Jul 10 08:51:27 EDT 2007 BIOS-provided physical RAM map: Xen: 0000000000000000 - 000000001fb00000 (usable) 0MB HIGHMEM available. 507MB LOWMEM available. NX (Execute Disable) protection: active ACPI in unprivileged domain disabled Built 1 zonelists. Total pages: 129792 Kernel command line: ro root=LABEL=/ console=xvc0 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 CPU 0 irqstacks, hard=c071b000 soft=c06fb000 PID hash table entries: 2048 (order: 11, 8192 bytes) Xen reported: 2659.996 MHz processor. Console: colour dummy device 80x25 Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Software IO TLB disabled vmalloc area: e0800000-f4ffe000, maxmem 2d7fe000 Memory: 497664k/519168k available (2017k kernel code, 12908k reserved, 824k data, 172k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay using timer specific routine.. 6657.62 BogoMIPS (lpj=13315256) Security Framework v1.0.0 initialized SELinux: Initializing. SELinux: Starting in permissive mode selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 512 CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 2048K Checking 'hlt' instruction... OK. SMP alternatives: switching to UP code Freeing SMP alternatives: 16k freed Brought up 1 CPUs checking if image is initramfs... it is Freeing initrd memory: 3071k freed Grant table initialized NET: Registered protocol family 16 ACPI Exception (utmutex-0262): AE_BAD_PARAMETER, Thread C06CAAA0 could not acquire Mutex [2] [20060707] Brought up 1 CPUs PCI: setting up Xen PCI frontend stub ACPI: Interpreter disabled. Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI: disabled xen_mem: Initialising balloon driver. usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: System does not support PCI PCI: System does not support PCI NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default NET: Registered protocol family 2 IP route cache hash table entries: 4096 (order: 2, 16384 bytes) TCP established hash table entries: 16384 (order: 5, 131072 bytes) TCP bind hash table entries: 8192 (order: 4, 65536 bytes) TCP: Hash tables configured (established 16384 bind 8192) TCP reno registered IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com> audit: initializing netlink socket (disabled) audit(1189014170.106:1): initialized VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) SELinux: Registering netfilter hooks Initializing Cryptographic API ksign: Installing public key data Loading keyring - Added public key 3EFBCAAC52BC4CBF - User ID: CentOS (Kernel Module GPG key) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 rtc: IRQ 8 is not free. Non-volatile memory driver v1.2 Linux agpgart interface v0.101 (c) Dave Jones RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize Xen virtual console successfully installed as xvc0 Linux version 2.6.18-8.1.8.el5xen (mockbuild@builder4.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Tue Jul 10 08:51:27 EDT 2007 BIOS-provided physical RAM map: Xen: 0000000000000000 - 000000001fb00000 (usable) 0MB HIGHMEM available. 507MB LOWMEM available. NX (Execute Disable) protection: active ACPI in unprivileged domain disabled Built 1 zonelists. Total pages: 129792 Kernel command line: ro root=LABEL=/ console=xvc0 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 CPU 0 irqstacks, hard=c071b000 soft=c06fb000 PID hash table entries: 2048 (order: 11, 8192 bytes) Xen reported: 2659.996 MHz processor. Console: colour dummy device 80x25 Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Software IO TLB disabled vmalloc area: e0800000-f4ffe000, maxmem 2d7fe000 Memory: 497664k/519168k available (2017k kernel code, 12908k reserved, 824k data, 172k init, 0k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay using timer specific routine.. 6657.62 BogoMIPS (lpj=13315256) Security Framework v1.0.0 initialized SELinux: Initializing. SELinux: Starting in permissive mode selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 512 CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 2048K Checking 'hlt' instruction... OK. SMP alternatives: switching to UP code Freeing SMP alternatives: 16k freed Brought up 1 CPUs checking if image is initramfs... it is Freeing initrd memory: 3071k freed Grant table initialized NET: Registered protocol family 16 ACPI Exception (utmutex-0262): AE_BAD_PARAMETER, Thread C06CAAA0 could not acquire Mutex [2] [20060707] Brought up 1 CPUs PCI: setting up Xen PCI frontend stub ACPI: Interpreter disabled. Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI: disabled xen_mem: Initialising balloon driver. usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: System does not support PCI PCI: System does not support PCI NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default NET: Registered protocol family 2 IP route cache hash table entries: 4096 (order: 2, 16384 bytes) TCP established hash table entries: 16384 (order: 5, 131072 bytes) TCP bind hash table entries: 8192 (order: 4, 65536 bytes) TCP: Hash tables configured (established 16384 bind 8192) TCP reno registered IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com> audit: initializing netlink socket (disabled) audit(1189014170.106:1): initialized VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) SELinux: Registering netfilter hooks Initializing Cryptographic API ksign: Installing public key data Loading keyring - Added public key 3EFBCAAC52BC4CBF - User ID: CentOS (Kernel Module GPG key) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 rtc: IRQ 8 is not free. Non-volatile memory driver v1.2 Linux agpgart interface v0.101 (c) Dave Jones RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize Xen virtual console successfully installed as xvc0 Event-channel device installed. Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx ide-floppy driver 0.99.newide usbcore: registered new driver hiddev usbcore: registered new driver usbhid drivers/usb/input/hid-core.c: v2.6:USB HID core driver PNP: No PS/2 controller found. Probing ports directly. i8042.c: No controller found. mice: PS/2 mouse device common for all mice md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 md: bitmap version 4.39 TCP bic registered Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 17 Using IPI No-Shortcut mode XENBUS: Device with no driver: device/vbd/51712 XENBUS: Device with no driver: device/vif/0 Freeing unused kernel memory: 172k freed Write protecting the kernel read-only data: 355k Red Hat nash version 5.1.19.6 starting Mounting proc filesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes Setting up hotplug. Creating block device nodes. Loading uhci-hcd.ko module USB Universal Host Controller Interface driver v3.0 Loading ohci-hcd.ko module Loading ehci-hcd.ko module Loading jbd.ko module Loading ext3.ko module Loading xenblk.ko module Registering block device major 202 xvda:Creating root device. xvda1 xvda2 Mounting root filesystem. mount: could not find filesystem '/dev/root' Setting up other filesystems. Setting up new root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Kernel panic - not syncing: Attempted to kill init!
Additional Information	This is random; difficult to reproduce, seems to happen one in 20 times or so.
Tags	No tags attached.

colo4jax 2007-09-05 17:50 reporter ~0005961	domU config: name = "srwhite" memory = "499" vif = [ 'mac=00:16:3e:3d:a0:22, vifname=srwhite, script=vif-srwhite'] disk = [ 'tap:aio:/home/vps/srwhite.img,xvda,w' ] on_reboot = 'restart' on_crash = 'restart' vcpus=1 bootloader="/usr/bin/pygrub"

colo4jax 2007-09-05 17:52 reporter ~0005962	Following are the last few relevant lines of a good boot when it works (same domU, same config, etc).: Mounting proc filesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes Setting up hotplug. Creating block device nodes. Loading uhci-hcd.ko module USB Universal Host Controller Interface driver v3.0 Loading ohci-hcd.ko module Loading ehci-hcd.ko module Loading jbd.ko module Loading ext3.ko module Loading xenblk.ko module Registering block device major 202 xvda: xvda1 xvda2 Creating root device. Mounting root filesystem. EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. Setting up other filesystems. Setting up new root fs no fstab.sys, mounting internal defaults Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys SELinux: Disabled at runtime. SELinux: Unregistering netfilter hooks audit(1189014645.697:2): selinux=0 auid=4294967295 INIT: version 2.86 booting

colo4jax 2007-10-16 15:49 reporter ~0006132	After talking this over with a colleague, he reports the same issue under CentOS 5 in a similar configuration. He uses LVM only, no tap:aio, but symptoms are the same. It seems to happen under high IO. I can also report that this seems to happen very prevalently when I reboot a Xen dom0 and /etc/init.d/xendomains runs to start all of the guests. While numerous guests are starting, about half of them fail, presumably because they can't attach or associate the disk with a device in their kernel. I can also report that I've seen this happen on a Debian guest at least once. Due to the lack of enthusiasm about addressing this, which is probably a broader issue with Xen, i'll take this up with xen-users and then with xen-dev if necessary.

Date Modified	Username	Field	Change
2007-09-05 17:49	colo4jax	New Issue
2007-09-05 17:49	colo4jax	Status	new => assigned
2007-09-05 17:50	colo4jax	Note Added: 0005961
2007-09-05 17:52	colo4jax	Note Added: 0005962
2007-10-16 15:49	colo4jax	Note Added: 0006132