|
|
2.6.32-754.2.1.el6-panic.txt (16,137 bytes)
# virsh console fwa36ax46jpbja
Connected to domain fwa36ax46jpbja
Escape character is ^]
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-754.2.1.el6.x86_64 (mockbuild@x86-01.bsys.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) ) #1 SMP Fri Jul 13 12:50:12 UTC 2018
Command line: ro root=/dev/xvda1 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto console=tty0
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000000a0000 (usable)
Xen: 00000000000a0000 - 0000000000100000 (reserved)
Xen: 0000000000100000 - 0000000100000000 (usable)
DMI not present or invalid.
last_pfn = 0x100000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000100000000
RAMDISK: 02071000 - 05daa000
ACPI: Deleted _OSI(Windows 2012)
ACPI: Deleted _OSI(Windows 2013)
No NUMA configuration found
Faking a node at 0000000000000000-0000000100000000
Bootmem setup node 0 0000000000000000-0000000100000000
NODE_DATA [0000000000008000 - 000000000003bfff]
bootmap [000000000003c000 - 000000000005bfff] pages 20
(7 early reservations) ==> bootmem [0000000000 - 0100000000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [00065ad000 - 00065e4000] XEN PAGETABLES ==> [00065ad000 - 00065e4000]
#2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#3 [0001000000 - 0002050a64] TEXT DATA BSS ==> [0001000000 - 0002050a64]
#4 [0002071000 - 0005daa000] RAMDISK ==> [0002071000 - 0005daa000]
#5 [0005daa000 - 00065ad000] XEN START INFO ==> [0005daa000 - 00065ad000]
#6 [0000100000 - 00008cb000] PGTABLE ==> [0000100000 - 00008cb000]
Found 129MB of memory at 112MB for crashkernel auto
Reserving 129MB of memory at 112MB for crashkernel (System RAM: 4096MB)
Zone PFN ranges:
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000001 -> 0x000000a0
0: 0x00000100 -> 0x00100000
SFI: Simple Firmware Interface v0.7 http://simplefirmware.org
SMP: Allowing 32 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
PCI: Warning: Cannot find a gap in the 32bit address range
PCI: Unassigned devices with 32bit resource registers may break!
Allocating PCI resources starting at 100100000 (gap: 100100000:400000)
Booting paravirtualized kernel on Xen
Xen version: 4.8.3-5.el7 (preserve-AD)
NR_CPUS:4096 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:1
PERCPU: Embedded 33 pages/cpu @ffff880012303000 s104088 r8192 d22888 u135168
pcpu-alloc: s104088 r8192 d22888 u135168 alloc=33*4096
pcpu-alloc: [0] 00 [0] 01 [0] 02 [0] 03 [0] 04 [0] 05 [0] 06 [0] 07
pcpu-alloc: [0] 08 [0] 09 [0] 10 [0] 11 [0] 12 [0] 13 [0] 14 [0] 15
pcpu-alloc: [0] 16 [0] 17 [0] 18 [0] 19 [0] 20 [0] 21 [0] 22 [0] 23
pcpu-alloc: [0] 24 [0] 25 [0] 26 [0] 27 [0] 28 [0] 29 [0] 30 [0] 31
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on. Total pages: 1032145
Policy zone: DMA32
Kernel command line: ro root=/dev/xvda1 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=129M@112M console=tty0
PID hash table entries: 4096 (order: 3, 32768 bytes)
x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
xsave: enabled eager FPU xstate_bv 0x7, cntxt size 0x340
Memory: 3903828k/4194304k available (5525k kernel code, 388k absent, 290088k reserved, 6904k data, 1340k init)
Kernel/User page tables isolation: disabled on XEN PV.
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:528
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
allocated 16777216 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
installing Xen timer for CPU 0
TSC: cpu family 6 model 45, tsc initial value = 6fc598cdcc1144
Detected 2000.000 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 4000.00 BogoMIPS (lpj=2000000)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux: Initializing.
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Speculative Store Bypass: Vulnerable
FEATURE SPEC_CTRL Not Present
FEATURE IBPB_SUPPORT Not Present
Spectre V2 : Mitigation: Full retpoline
alternatives: switching to unfair spinlock
SMP alternatives: switching to UP code
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 22056 entries in 87 pages
Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
installing Xen timer for CPU 1
SMP alternatives: switching to SMP code
installing Xen timer for CPU 2
installing Xen timer for CPU 3
installing Xen timer for CPU 4
installing Xen timer for CPU 5
installing Xen timer for CPU 6
installing Xen timer for CPU 7
installing Xen timer for CPU 8
installing Xen timer for CPU 9
installing Xen timer for CPU 10
installing Xen timer for CPU 11
installing Xen timer for CPU 12
installing Xen timer for CPU 13
installing Xen timer for CPU 14
installing Xen timer for CPU 15
installing Xen timer for CPU 16
installing Xen timer for CPU 17
installing Xen timer for CPU 18
installing Xen timer for CPU 19
installing Xen timer for CPU 20
installing Xen timer for CPU 21
installing Xen timer for CPU 22
installing Xen timer for CPU 23
installing Xen timer for CPU 24
installing Xen timer for CPU 25
installing Xen timer for CPU 26
installing Xen timer for CPU 27
installing Xen timer for CPU 28
installing Xen timer for CPU 29
installing Xen timer for CPU 30
installing Xen timer for CPU 31
Brought up 32 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
NET: Registered protocol family 16
Cannot set affinity for irq 368
Broke affinity for irq 503
Broke affinity for irq 504
Broke affinity for irq 505
Broke affinity for irq 506
Broke affinity for irq 507
Broke affinity for irq 498
Broke affinity for irq 499
Broke affinity for irq 500
Broke affinity for irq 501
Broke affinity for irq 502
Broke affinity for irq 493
Broke affinity for irq 494
Broke affinity for irq 495
Broke affinity for irq 496
Broke affinity for irq 497
Broke affinity for irq 488
Broke affinity for irq 489
Broke affinity for irq 490
Broke affinity for irq 491
Broke affinity for irq 492
Broke affinity for irq 483
Broke affinity for irq 484
Broke affinity for irq 485
Broke affinity for irq 486
Broke affinity for irq 487
Broke affinity for irq 478
Broke affinity for irq 479
Broke affinity for irq 480
Broke affinity for irq 481
Broke affinity for irq 482
Broke affinity for irq 473
Broke affinity for irq 474
Broke affinity for irq 475
Broke affinity for irq 476
Broke affinity for irq 477
Broke affinity for irq 468
Broke affinity for irq 469
Broke affinity for irq 470
Broke affinity for irq 471
Broke affinity for irq 472
Broke affinity for irq 463
Broke affinity for irq 464
Broke affinity for irq 465
Broke affinity for irq 466
Broke affinity for irq 467
Broke affinity for irq 458
Broke affinity for irq 459
Broke affinity for irq 460
Broke affinity for irq 461
Broke affinity for irq 462
Broke affinity for irq 453
Broke affinity for irq 454
Broke affinity for irq 455
Broke affinity for irq 456
Broke affinity for irq 457
Broke affinity for irq 448
Broke affinity for irq 449
Broke affinity for irq 450
Broke affinity for irq 451
Broke affinity for irq 452
Broke affinity for irq 443
Broke affinity for irq 444
Broke affinity for irq 445
Broke affinity for irq 446
Broke affinity for irq 447
Broke affinity for irq 438
Broke affinity for irq 439
Broke affinity for irq 440
Broke affinity for irq 441
Broke affinity for irq 442
Broke affinity for irq 433
Broke affinity for irq 434
Broke affinity for irq 435
Broke affinity for irq 436
Broke affinity for irq 437
Broke affinity for irq 428
Broke affinity for irq 429
Broke affinity for irq 430
Broke affinity for irq 431
Broke affinity for irq 432
Broke affinity for irq 423
Broke affinity for irq 424
Broke affinity for irq 425
Broke affinity for irq 426
Broke affinity for irq 427
Broke affinity for irq 418
Broke affinity for irq 419
Broke affinity for irq 420
Broke affinity for irq 421
Broke affinity for irq 422
Broke affinity for irq 413
Broke affinity for irq 414
Broke affinity for irq 415
Broke affinity for irq 416
Broke affinity for irq 417
Broke affinity for irq 408
Broke affinity for irq 409
Broke affinity for irq 410
Broke affinity for irq 411
Broke affinity for irq 412
Broke affinity for irq 403
Broke affinity for irq 404
Broke affinity for irq 405
Broke affinity for irq 406
Broke affinity for irq 407
Broke affinity for irq 398
Broke affinity for irq 399
Broke affinity for irq 400
Broke affinity for irq 401
Broke affinity for irq 402
Broke affinity for irq 393
Broke affinity for irq 394
Broke affinity for irq 395
Broke affinity for irq 396
Broke affinity for irq 397
Broke affinity for irq 388
Broke affinity for irq 389
Broke affinity for irq 390
Broke affinity for irq 391
Broke affinity for irq 392
Broke affinity for irq 383
Broke affinity for irq 384
Broke affinity for irq 385
Broke affinity for irq 386
Broke affinity for irq 387
Broke affinity for irq 378
Broke affinity for irq 379
Broke affinity for irq 380
Broke affinity for irq 381
Broke affinity for irq 382
Broke affinity for irq 373
Broke affinity for irq 374
Broke affinity for irq 375
Broke affinity for irq 376
Broke affinity for irq 377
Broke affinity for irq 368
Broke affinity for irq 369
Broke affinity for irq 370
Broke affinity for irq 371
Broke affinity for irq 372
PCI: Fatal: No config space access function found
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x100000 max_arch_pfn = 0x400000000
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
Switching to clocksource xen
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 62692k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
sha256_ssse3: Using AVX optimized SHA-256 implementation
futex hash table entries: 8192 (order: 7, 524288 bytes)
audit: initializing netlink socket (disabled)
type=2000 audit(1531768148.187:1): initialized
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
hugetlbfs: disabling because there are no supported hugepage sizes
msgmni has been set to 7747
ksign: Installing public key data
Loading keyring
- Added public key C77E37DA8191DEBD
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Console: switching to colour frame buffer device 100x37
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input0
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
input: Xen Virtual Keyboard as /devices/virtual/input/input1
input: Xen Virtual Pointer as /devices/virtual/input/input2
rtc_cmos: probe of rtc_cmos failed with error -16
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
GRE over IPv4 demultiplexor driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/51713
XENBUS: Device with no driver: device/vbd/51714
XENBUS: Device with no driver: device/vif/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 1340k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 600k freed
Freeing unused kernel memory: 1524k freed
alignment check: 0000 [#1] SMP
last sysfs file:
CPU 3
Modules linked in:
Pid: 325, comm: init Not tainted 2.6.32-754.2.1.el6.x86_64 #1
RIP: e030:[<ffffffff81560f2e>] [<ffffffff81560f2e>] retint_userspace_restore_args+0x56/0xd3
RSP: e02b:ffff880005b7bfd0 EFLAGS: 00250086
RAX: 0000000000000000 RBX: 00007ffc55bfd730 RCX: ffffffffffffffff
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 00007ffc55bfd770 R08: 00007f16afdde700 R09: 0000000000000000
R10: 00007f16afdde9d0 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 000000000061be28 R15: 0000000000000000
FS: 00007f16afdde700(0000) GS:ffff880012366000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f16afdde9d0 CR3: 0000000005b7e000 CR4: 0000000000042660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process init (pid: 325, threadinfo ffff880005b78000, task ffff8800fde27520)
Stack:
0000000000000000 0000003cd1eacd9d 0000000000000033 0000000000000246
<d> 00007ffc55bfd730 000000000000002b
Call Trace:
Code: 54 24 08 4c 8b 4c 24 10 4c 8b 44 24 18 48 8b 44 24 20 48 8b 4c 24 28 48 8b 54 24 30 48 8b 74 24 38 48 8b 7c 24 40 48 83 c4 50 50 <65> 48 8b 04 25 04 40 00 00 48 8d 40 d0 8f 00 8f 40 08 8f 40 10
RIP [<ffffffff81560f2e>] retint_userspace_restore_args+0x56/0xd3
RSP <ffff880005b7bfd0>
---[ end trace 12eb13408aece901 ]---
Kernel panic - not syncing: Fatal exception
Pid: 325, comm: init Tainted: G D -- ------------ 2.6.32-754.2.1.el6.x86_64 #1
Call Trace:
[<ffffffff8155407d>] ? panic+0xa7/0x18b
[<ffffffff81557e0c>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8155a304>] ? oops_end+0xe4/0x100
[<ffffffff8100f94b>] ? die+0x5b/0x90
[<ffffffff81559b24>] ? do_trap+0xc4/0x160
[<ffffffff8100b4b2>] ? do_alignment_check+0x92/0xb0
[<ffffffff8155961a>] ? error_exit+0x2a/0x60
[<ffffffff81560fd6>] ? retint_restore_args+0x5/0x36
[<ffffffff81561685>] ? alignment_check+0x25/0x30
[<ffffffff81560f2e>] ? retint_userspace_restore_args+0x56/0xd3
|
|
|
|
You can disable eagerfpu from grub kernel line using eagerfpu=off. Once you have done so the Xen guests will boot. The kernel has eagerfpu on by default and doesn't look lke there is a way to disable it without code modifications or using the grub line. |
|
|
|
My testing suggests that kernel 2.6.32-754.3.5 - which is now the latest - works? |
|
|
|
My testing does not confirm that observation. 754.3.5 still fails with the same panic.
RIP [<ffffffff81564f2e>] retint_userspace_restore_args+0x56/0xd3
RSP <ffff880005b83fd0>
---[ end trace 5d852052cd1a913c ]---
Kernel panic - not syncing: Fatal exception
Pid: 325, comm: init Tainted: G D -- ------------ 2.6.32-754.3.5.el6.x86_64 #1
Call Trace:
[<ffffffff81558237>] ? panic+0xa7/0x18b
[<ffffffff8155bfcc>] ? _spin_unlock_irqrestore+0x1c/0x20
[<ffffffff8155e304>] ? oops_end+0xe4/0x100
[<ffffffff8100f95b>] ? die+0x5b/0x90
[<ffffffff8155db24>] ? do_trap+0xc4/0x160
[<ffffffff8100b4c2>] ? do_alignment_check+0x92/0xb0
[<ffffffff8155d61a>] ? error_exit+0x2a/0x60
[<ffffffff81564fd6>] ? retint_restore_args+0x5/0x36
[<ffffffff81565685>] ? alignment_check+0x25/0x30
[<ffffffff81564f2e>] ? retint_userspace_restore_args+0x56/0xd3 |
|
|
|
Indeed - I used to be able to produce this on XenServer, however my current set up does not reproduce this even with kernel 754.2.1; therefore there must be something different in my environment which changed the behaviour.
For reference, my command line doesn't contain eagerfpu and is very simply:
ro root=/dev/mapper/VolGroup-lv_root console=hvc0 rd_LVM_LV=VolGroup/lv_swap rd_LVM_LV=VolGroup/lv_root
I can boot CentOS 6.9 with kernels 2.6.32-754.2.1.el6.x86_64 and 2.6.32-754.3.5.el6.x86_64 |
|
|
|
For completeness, I also updated to CentOS 6.10 and that boots as above (I didn't imagine updating the rest of the distro would affect this boot failure, but thought I'd check! |
|
|
|
I've now reproduced this again: the bug requires xsave support to be present in the guest. I was obviously running on older hardware for some of my tests. Hiding xsave from the VM permits it to boot. My understanding is the guest can still protect itself with eagerfpu even without xsave support |
|
|
|
I wanted to provide an update from Red Hat. Unfortunately it's not a helpful one. The upstream bug has been closed as WONTFIX without comment, so I can only conclude that they have decided it is not worth fixing or falls outside the scope of what they care to support. The options remain to leave eagerfpu=off on the command line, use CentOS 7, or use another kernel like kernel-lt from elrepo. |
|
|
|
@kstange
Thanks for the update. Too bad RH did not care. On the other hand it is understandable. EL6 has only 1.5 yrs until EOL. Maybe it's time to update to 7. |
|
|
|
Closing. Feel free to reopen if/when there is any status change. |
|
