View Issue Details

IDProjectCategoryView StatusLast Update
0002317CentOS-5xenpublic2007-10-16 15:49
Reportercolo4jax Assigned To 
PrioritynormalSeverityminorReproducibilityrandom
Status assignedResolutionopen 
Product Version5.0 - i386 
Summary0002317: CentOS 5 domU randomly fails to start
DescriptionOn occasion when starting a CentOS 5 domU, the start fails due to a kernel panic as follows:

Linux version 2.6.18-8.1.8.el5xen (mockbuild@builder4.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Tue Jul 10 08:51:27 EDT 2007
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000001fb00000 (usable)
0MB HIGHMEM available.
507MB LOWMEM available.
NX (Execute Disable) protection: active
ACPI in unprivileged domain disabled
Built 1 zonelists. Total pages: 129792
Kernel command line: ro root=LABEL=/ console=xvc0
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c071b000 soft=c06fb000
PID hash table entries: 2048 (order: 11, 8192 bytes)
Xen reported: 2659.996 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Software IO TLB disabled
vmalloc area: e0800000-f4ffe000, maxmem 2d7fe000
Memory: 497664k/519168k available (2017k kernel code, 12908k reserved, 824k data, 172k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 6657.62 BogoMIPS (lpj=13315256)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 16k freed
Brought up 1 CPUs
checking if image is initramfs... it is
Freeing initrd memory: 3071k freed
Grant table initialized
NET: Registered protocol family 16
ACPI Exception (utmutex-0262): AE_BAD_PARAMETER, Thread C06CAAA0 could not acquire Mutex [2] [20060707]
Brought up 1 CPUs
PCI: setting up Xen PCI frontend stub
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
xen_mem: Initialising balloon driver.
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 8192)
TCP reno registered
IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com>
audit: initializing netlink socket (disabled)
audit(1189014170.106:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 3EFBCAAC52BC4CBF
- User ID: CentOS (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rtc: IRQ 8 is not free.
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Xen virtual console successfully installed as xvc0
Linux version 2.6.18-8.1.8.el5xen (mockbuild@builder4.centos.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SMP Tue Jul 10 08:51:27 EDT 2007
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 000000001fb00000 (usable)
0MB HIGHMEM available.
507MB LOWMEM available.
NX (Execute Disable) protection: active
ACPI in unprivileged domain disabled
Built 1 zonelists. Total pages: 129792
Kernel command line: ro root=LABEL=/ console=xvc0
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c071b000 soft=c06fb000
PID hash table entries: 2048 (order: 11, 8192 bytes)
Xen reported: 2659.996 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Software IO TLB disabled
vmalloc area: e0800000-f4ffe000, maxmem 2d7fe000
Memory: 497664k/519168k available (2017k kernel code, 12908k reserved, 824k data, 172k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 6657.62 BogoMIPS (lpj=13315256)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 16k freed
Brought up 1 CPUs
checking if image is initramfs... it is
Freeing initrd memory: 3071k freed
Grant table initialized
NET: Registered protocol family 16
ACPI Exception (utmutex-0262): AE_BAD_PARAMETER, Thread C06CAAA0 could not acquire Mutex [2] [20060707]
Brought up 1 CPUs
PCI: setting up Xen PCI frontend stub
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
xen_mem: Initialising balloon driver.
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 8192)
TCP reno registered
IA-32 Microcode Update Driver: v1.14-xen <tigran@veritas.com>
audit: initializing netlink socket (disabled)
audit(1189014170.106:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 3EFBCAAC52BC4CBF
- User ID: CentOS (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rtc: IRQ 8 is not free.
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Xen virtual console successfully installed as xvc0
Event-channel device installed.
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vif/0
Freeing unused kernel memory: 172k freed
Write protecting the kernel read-only data: 355k
Red Hat nash version 5.1.19.6 starting
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
Creating block device nodes.
Loading uhci-hcd.ko module
USB Universal Host Controller Interface driver v3.0
Loading ohci-hcd.ko module
Loading ehci-hcd.ko module
Loading jbd.ko module
Loading ext3.ko module
Loading xenblk.ko module
Registering block device major 202
 xvda:Creating root device.
 xvda1 xvda2
Mounting root filesystem.
mount: could not find filesystem '/dev/root'
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
Additional InformationThis is random; difficult to reproduce, seems to happen one in 20 times or so.
TagsNo tags attached.

Activities

colo4jax

colo4jax

2007-09-05 17:50

reporter   ~0005961

domU config:

name = "srwhite"
memory = "499"
vif = [ 'mac=00:16:3e:3d:a0:22, vifname=srwhite, script=vif-srwhite']
disk = [ 'tap:aio:/home/vps/srwhite.img,xvda,w' ]
on_reboot = 'restart'
on_crash = 'restart'
vcpus=1
bootloader="/usr/bin/pygrub"
colo4jax

colo4jax

2007-09-05 17:52

reporter   ~0005962

Following are the last few relevant lines of a good boot when it works (same domU, same config, etc).:

Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
Creating block device nodes.
Loading uhci-hcd.ko module
USB Universal Host Controller Interface driver v3.0
Loading ohci-hcd.ko module
Loading ehci-hcd.ko module
Loading jbd.ko module
Loading ext3.ko module
Loading xenblk.ko module
Registering block device major 202
 xvda: xvda1 xvda2
Creating root device.
Mounting root filesystem.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
audit(1189014645.697:2): selinux=0 auid=4294967295
INIT: version 2.86 booting
colo4jax

colo4jax

2007-10-16 15:49

reporter   ~0006132

After talking this over with a colleague, he reports the same issue under CentOS 5 in a similar configuration. He uses LVM only, no tap:aio, but symptoms are the same. It seems to happen under high IO. I can also report that this seems to happen *very* prevalently when I reboot a Xen dom0 and /etc/init.d/xendomains runs to start all of the guests. While numerous guests are starting, about half of them fail, presumably because they can't attach or associate the disk with a device in their kernel. I can also report that I've seen this happen on a Debian guest at least once.

Due to the lack of enthusiasm about addressing this, which is probably a broader issue with Xen, i'll take this up with xen-users and then with xen-dev if necessary.

Issue History

Date Modified Username Field Change
2007-09-05 17:49 colo4jax New Issue
2007-09-05 17:49 colo4jax Status new => assigned
2007-09-05 17:50 colo4jax Note Added: 0005961
2007-09-05 17:52 colo4jax Note Added: 0005962
2007-10-16 15:49 colo4jax Note Added: 0006132