View Issue Details

IDProjectCategoryView StatusLast Update
0014347CentOS-7kernelpublic2018-05-11 22:23
Reporterawelbers 
PriorityurgentSeveritycrashReproducibilityalways
Status resolvedResolutionfixed 
Product Version7.4.1708 
Target VersionFixed in Version 
Summary0014347: kernel-plus-3.10.0-693.11.6.el7.centos.plus under Xen4 runs in endless loop
DescriptionAfter installing and rebooting the latest "kernel-plus-3.10.0-693.11.6.el7.centos.plus" under Xen4 guests it ends up in an endless loop.

I had to reboot the older working version kernel-plus-3.10.0-693.11.1.el7.centos.plus.x86_64" and to exclude the new version in the updates.

I'm running the latest CentOS7 and Xen4 (xen-4.6.6-8.el7.x86_64) on the host machine and also the latest CentOS7 on all guests.
Steps To Reproduce- Install "kernel-plus-3.10.0-693.11.6.el7.centos.plus" on Xen4 guest
- Reboot
- See the endless loop

It's every time reproducable. Had to fix it on 5 Xen-Guests.
Additional InformationThe output from "xl create -c guestname" is attached as file

TagsNo tags attached.
abrt_hash
URL

Relationships

related to 0014336 resolvedtoracat CentOS-6 EL6 VM fails to boot after updating to kernel-2.6.32-696.18.7.el6.x86_64 
related to 0013763 resolvedJohnnyHughes CentOS-7 CentOS 7.4 kernel (3.10.0-693*) fails to boot as Xen PV guest 

Activities

awelbers

awelbers

2018-01-09 02:02

reporter  

xen-centos-kernel-plus-endless-loop-20170109.txt (12,424 bytes)
Message from "xl create -c hostname"
====================================

After the message the VM seems to hang in den endless boot loop

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018
[    0.000000] Command line: root=/dev/xvda1 ro console=hvc0 selinux=1 vconsole.keymap=de-latin1-nodeadkeys vconsole.font=latarcyrheb-sun16 LANG=de_DE.UTF-8
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x00000001ffffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] e820: last_pfn = 0x200000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0x100000 max_arch_pfn = 0x400000000
[    0.000000] RAMDISK: [mem 0x0243e000-0x0349efff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x00000001ffffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x7effc000-0x7f022fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0x1ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x1ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x00001000-0x1ffffffff]
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.000000] e820: cannot find a gap in the 32bit address range
[    0.000000] e820: PCI devices with unassigned 32bit BARs may break!
[    0.000000] e820: [mem 0x200100000-0x2004fffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.6.6-8.el7 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff88007ec00000 s104600 r8192 d30568 u1048576
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2064266
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=/dev/xvda1 ro console=hvc0 selinux=1 vconsole.keymap=de-latin1-nodeadkeys vconsole.font=latarcyrheb-sun16 LANG=de_DE.UTF-8
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form
[    0.000000] Memory: 1910728k/8388608k available (6984k kernel code, 388k absent, 6477492k reserved, 4581k data, 1800k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] x86/pti: Xen PV detected, disabling PTI protection
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=2.
[    0.000000] NR_IRQS:327936 nr_irqs:32 0
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] allocated 33554432 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2599.874 MHz processor
[    0.002000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5200.12 BogoMIPS (lpj=2600064)
[    0.002000] pid_max: default: 32768 minimum: 301
[    0.002000] Security Framework initialized
[    0.002000] SELinux:  Initializing.
[    0.002000] Yama: becoming mindful.
[    0.002355] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.004606] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.005471] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.005499] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes)
[    0.005805] Initializing cgroup subsys memory
[    0.005826] Initializing cgroup subsys devices
[    0.005832] Initializing cgroup subsys freezer
[    0.005838] Initializing cgroup subsys net_cls
[    0.005844] Initializing cgroup subsys blkio
[    0.005849] Initializing cgroup subsys perf_event
[    0.005854] Initializing cgroup subsys hugetlb
[    0.005859] Initializing cgroup subsys pids
[    0.005866] Initializing cgroup subsys net_prio
[    0.005927] FEATURE SPEC_CTRL Not Present
[    0.005933] FEATURE IBPB_SUPPORT Not Present
[    0.005938] CPU: Physical Processor ID: 0
[    0.005942] CPU: Processor Core ID: 0
[    0.006740] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.006746] Last level dTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.006750] tlb_flushall_shift: 6
[    0.037040] ftrace: allocating 26850 entries in 105 pages
[    0.043080] cpu 0 spinlock event irq 17
[    0.043091] smpboot: Max logical packages: 1
[    0.043122] Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only.
[    0.044458] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.044466] NMI watchdog: Shutting down hard lockup detector on all cpus
[    0.044540] installing Xen timer for CPU 1
[    0.044554] cpu 1 spinlock event irq 24
[    0.044632] SMP alternatives: switching to SMP code
[    0.002000] [Firmware Bug]: CPU1: APIC id mismatch. Firmware: ffff APIC: 27
[    0.069581] Brought up 2 CPUs
[    0.076171] node 0 initialised, 0 pages in 7ms
[    0.076232] devtmpfs: initialized
[    0.079293] EVM: security.selinux
[    0.079299] EVM: security.ima
[    0.079303] EVM: security.capability
[    0.080180] atomic64 test passed for x86-64 platform with CX8 and with SSE
[    0.080190] pinctrl core: initialized pinctrl subsystem
[    0.080247] xen:grant_table: Grant tables using version 2 layout
[    0.080278] Grant table initialized
[    0.100263] RTC time: 165:165:165, date: 165/165/65
[    0.100367] NET: Registered protocol family 16
[    0.101041] PCI: setting up Xen PCI frontend stub
[    0.107068] ACPI: Interpreter disabled.
[    0.107077] xen:balloon: Initialising balloon driver
[    0.134035] random: fast init done
[    0.135020] xen_balloon: Initialising balloon driver
[    0.135043] vgaarb: loaded
[    0.135106] SCSI subsystem initialized
[    0.135149] usbcore: registered new interface driver usbfs
[    0.135163] usbcore: registered new interface driver hub
[    0.135187] usbcore: registered new device driver usb
[    0.136032] dmi: Firmware registration failed.
[    0.136056] PCI: System does not support PCI
[    0.136061] PCI: System does not support PCI
[    0.136176] NetLabel: Initializing
[    0.136181] NetLabel:  domain hash size = 128
[    0.136184] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.136201] NetLabel:  unlabeled traffic allowed by default
[    0.136244] amd_nb: Cannot enumerate AMD northbridges
[    0.136270] Switched to clocksource xen
[    0.145186] pnp: PnP ACPI: disabled
[    0.147664] NET: Registered protocol family 2
[    0.147915] TCP established hash table entries: 65536 (order: 7, 524288 bytes)
[    0.148340] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.148590] TCP: Hash tables configured (established 65536 bind 65536)
[    0.148660] TCP: reno registered
[    0.148713] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[    0.148797] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[    0.148927] NET: Registered protocol family 1
[    0.149070] Unpacking initramfs...
[    0.386815] Freeing initrd memory: 16772k freed
[    0.392466] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer
[    0.392478] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules
[    0.392483] RAPL PMU: hw unit of domain package 2^-16 Joules
[    0.392487] RAPL PMU: hw unit of domain dram 2^-16 Joules
[    0.392594] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.392985] sha1_ssse3: Using AVX optimized SHA-1 implementation
[    0.393078] sha256_ssse3: Using AVX optimized SHA-256 implementation
[    0.393401] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.393423] Initialise system trusted keyring
[    0.393453] audit: initializing netlink socket (disabled)
[    0.393473] type=2000 audit(1515460754.588:1): initialized
[    0.425273] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.426123] zpool: loaded
[    0.426132] zbud: loaded
[    0.426294] VFS: Disk quotas dquot_6.5.2
[    0.426323] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.426470] msgmni has been set to 3764
[    0.426524] Key type big_key registered
[    0.427505] NET: Registered protocol family 38
[    0.427518] Key type asymmetric registered
[    0.427525] Asymmetric key parser 'x509' registered
[    0.427565] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    0.427615] io scheduler noop registered
[    0.427625] io scheduler deadline registered (default)
[    0.427669] io scheduler cfq registered
[    0.427772] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.427783] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.428283] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.428695] Non-volatile memory driver v1.3
[    0.428737] Linux agpgart interface v0.103
[    0.428810] crash memory driver: version 1.1
[    0.428854] rdac: device handler registered
[    0.428904] hp_sw: device handler registered
[    0.428911] emc: device handler registered
[    0.428916] alua: device handler registered
[    0.428955] libphy: Fixed MDIO Bus: probed
[    0.429019] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.429030] ehci-pci: EHCI PCI platform driver
[    0.429042] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.429049] ohci-pci: OHCI PCI platform driver
[    0.429059] uhci_hcd: USB Universal Host Controller Interface driver
[    0.429103] usbcore: registered new interface driver usbserial
[    0.429114] usbcore: registered new interface driver usbserial_generic
[    0.429123] usbserial: USB Serial support registered for generic
[    0.429144] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    1.439148] i8042: No controller found
[    1.439226] mousedev: PS/2 mouse device common for all mice
[    1.499512] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    1.499567] rtc_cmos: probe of rtc_cmos failed with error -38
[    1.499622] dmi-sysfs: dmi entry is absent.
[    1.499644] hidraw: raw HID events driver (C) Jiri Kosina
[    1.499743] usbcore: registered new interface driver usbhid
[    1.499750] usbhid: USB HID core driver
[    1.499773] drop_monitor: Initializing network drop monitor service
[    1.499841] TCP: cubic registered
[    1.499851] Initializing XFRM netlink socket
[    1.499945] NET: Registered protocol family 10
[    1.500160] NET: Registered protocol family 17
[    1.500174] mce: Unable to init device /dev/mcelog (rc: -5)
[    1.500241] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x416
[    1.500259] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x416
[    1.500316] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    1.500452] Loading compiled-in X.509 certificates
[    1.500481] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[    1.500497] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[    1.500992] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'
[    1.501030] registered taskstats version 1
bill_mcgonigle

bill_mcgonigle

2018-01-09 03:48

reporter   ~0030884

Confirming. I have a dozen C6/7 VM's running under Fedora 27 (Xen 4.9.1) that didn't come up on last night's reboot for the patched kernel and we had to back out. The F27 and Debian VM's came up fine; C6 would die at DomU creation with:

"libxl: error: libxl_dom.c:38:libxl__domain_type: unable to get domain type for domid=189
xl: unable to exec console client: No such file or directory
libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: console child [18778] exited with error status 1"

With C7, sometimes I got "invalid image" right from Xen's bootloader; sometimes I got progress like awelbers's with a stacktrace (continuing after his, below).

"rpm -V" verifies that both kernel-plus-3.10.0-693.11.6.el7.centos.plus.x86_64 and kernel-2.6.32-696.18.7.el6.x86_64 are installed correctly but neither will boot as Xen PV DomU. We're relying on kernel-plus from CentOS as the RHEL 7.4 kernel doesn't work at all under Xen PV. Back on kernel-plus-3.10.0-693.11.1.el7.centos.plus.x86_64 and kernel-2.6.32-696.16.1.el6.x86_64 we're running fine but would rather have the Meltdown/Spectre patches (and stay up to date with updates).

---

[ 1.664338] registered taskstats version 1
[ 1.667379] alignment check: 0000 [#1] SMP
[ 1.667494] Modules linked in:
[ 1.667575] CPU: 8 PID: 0 Comm: swapper/8 Not tainted 3.10.0-693.11.6.el7.centos.plus.x86_64 #1
[ 1.667742] task: ffff880086508fd0 ti: ffff880086514000 task.ti: ffff880086514000
[ 1.667879] RIP: e030:[<ffffffff8134444d>] [<ffffffff8134444d>] memcpy+0xd/0x110
[ 1.668007] RSP: e02b:ffff880086517c98 EFLAGS: 00250002
[ 1.668007] RAX: ffffffff81db4de8 RBX: 0000000000000000 RCX: 000000000000000b
[ 1.668007] RDX: 0000000000000004 RSI: ffffffff81db0542 RDI: ffffffff81db4de8
[ 1.668007] RBP: ffff880086517cf8 R08: ffffffff81db0dfc R09: 000000000000005c
[ 1.668007] R10: 00000000000fc024 R11: 0000000000100000 R12: 000000000000005c
[ 1.668007] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff81db4dd8
[ 1.668007] FS: 00007f35d8311740(0000) GS:ffff8800bb800000(0000) knlGS:0000000000000000
[ 1.668007] CS: e033 DS: 002b ES: 002b CR0: 0000000080050033
[ 1.668007] CR2: 00007f35d77a14f0 CR3: 0000000001a12000 CR4: 0000000000042660
[ 1.668007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1.668007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1.668007] Call Trace:
[ 1.668007] [<ffffffff81089a43>] ? log_store+0x103/0x200
[ 1.668007] [<ffffffff8108c039>] vprintk_emit+0x199/0x510
[ 1.668007] [<ffffffff8108c5d9>] vprintk_default+0x29/0x40
[ 1.668007] [<ffffffff816b0eb9>] printk+0x5d/0x74
[ 1.668007] [<ffffffff816b0d1e>] panic+0xcf/0x20d
[ 1.668007] [<ffffffff816bc62d>] ? __schedule+0x95d/0x9b0
[ 1.668007] [<ffffffff810ca4d0>] ? switched_to_idle+0x10/0x10
[ 1.668007] [<ffffffff810891bb>] __stack_chk_fail+0x1b/0x20
[ 1.668007] [<ffffffff816bc62d>] __schedule+0x95d/0x9b0
[ 1.668007] [<ffffffff816c4801>] ? __do_page_fault+0x171/0x450
[ 1.668007] [<ffffffff816c4b15>] ? do_page_fault+0x35/0x90
[ 1.668007] [<ffffffff816c08f8>] ? page_fault+0x28/0x30
[ 1.668007] [<ffffffff816c0dda>] ? error_exit+0x2a/0x60
[ 1.668007] [<ffffffff816c053a>] ? retint_swapgs+0x5/0x6
[ 1.689895] Key type trusted registered
[ 1.687055] Code: d4 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b
toracat

toracat

2018-01-09 06:57

manager   ~0030886

See https://lists.centos.org/pipermail/centos-virt/2018-January/005716.html for a report on the centos-virt mailing list. In that thread, a possible solution is provided by @JohnnyHughes:

https://lists.centos.org/pipermail/centos-virt/2018-January/005721.html
bill_mcgonigle

bill_mcgonigle

2018-01-09 20:15

reporter   ~0030896

I see there's a 4.9 kernel available there, which could be a workaround for some users who don't need to be on 3.10.
bill_mcgonigle

bill_mcgonigle

2018-01-13 17:00

reporter   ~0030929

from https://bugs.centos.org/view.php?id=13763:

# diff 11.1/boot/config-3.10.0-693.11.1.el7.centos.plus.x86_64 11.6/boot/config-3.10.0-693.11.6.el7.centos.plus.x86_64
3c3
< # Linux/x86_64 3.10.0-693.11.1.el7.centos.plus.x86_64 Kernel Configuration
---
> # Linux/x86_64 3.10.0-693.11.6.el7.centos.plus.x86_64 Kernel Configuration
5735a5736
> CONFIG_KAISER=y
adrian27

adrian27

2018-01-13 19:43

reporter   ~0030931

I had the same issue booting a CentOS 7 PV instance-store AMI in AWS EC2 using both latest kernels from CentOS 7 (kernel-3.10.0-693.11.6) and CentOS Plus (kernel-plus-3.10.0-693.11.6).

Basically the boot process stops after the following messages are displayed on the console, then the VM reboots:

(...)
[ 3.570512] usbcore: registered new interface driver usbhid
[ 3.581218] usbhid: USB HID core driver
[ 3.588925] drop_monitor: Initializing network drop monitor service
[ 3.601194] TCP: cubic registered
[ 3.608001] Initializing XFRM netlink socket
[ 3.616800] NET: Registered protocol family 10
[ 3.625903] NET: Registered protocol family 17
[ 3.634501] mce: Unable to init device /dev/mcelog (rc: -5)
[ 3.645168] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712
[ 3.656568] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[ 3.674563] Loading compiled-in X.509 certificates
[ 3.683010] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[ 3.700538] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[ 3.720636] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'
[ 3.738829] registered taskstats version 1

The standard CentOS 7.4 kernels (kernel-3.10.0-693*) did not work with PV instance-store based AMIs in AWS EC2 even before the latest CentOS 7.4 kernel 3.10.0-693.11.6 was released, however the CentOS Plus kernels did work from version 3.10.0-693.2.1 up to version 3.10.0-693.11.1. Now, the latest CentOS Plus kernel 3.10.0-693.11.6 does not work with PV instance-store based AMIs in AWS EC2 anymore.

The full CentOS Plus kernel version 3.10.0-693.11.6 boot log as retrieved from AWS EC2 is attached.

aws-kernel-plus-3.10.0-693.11.6.el7.centos.plus.x86_64.log.txt (65,581 bytes)
i-0071d198fc56b75db	e: registered new interface driver hub

[    0.128069] usbcore: registered new device driver usb

[    0.129126] dmi: Firmware registration failed.

[    0.130037] PCI: System does not support PCI

[    0.131009] PCI: System does not support PCI

[    0.132169] NetLabel: Initializing

[    0.133024] NetLabel:  domain hash size = 128

[    0.134017] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.135055] NetLabel:  unlabeled traffic allowed by default

[    0.136050] amd_nb: Cannot enumerate AMD northbridges

[    0.137037] Switched to clocksource xen

[    0.150363] pnp: PnP ACPI: disabled

[    0.158911] NET: Registered protocol family 2

[    0.166778] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.178769] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.189316] TCP: Hash tables configured (established 16384 bind 16384)

[    0.199539] TCP: reno registered

[    0.204854] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.214132] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.224257] NET: Registered protocol family 1

[    0.231702] Unpacking initramfs...

[    0.322421] random: fast init done

[    0.795814] Freeing initrd memory: 19404k freed

[    0.836023] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.849038] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.858265] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.868441] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.877679] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.889792] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.899877] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.910971] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.921383] Initialise system trusted keyring

[    0.929467] audit: initializing netlink socket (disabled)

[    0.938389] type=2000 audit(1515851068.643:1): initialized

[    0.981758] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    0.994775] zpool: loaded

[    0.999321] zbud: loaded

[    1.003769] VFS: Disk quotas dquot_6.5.2

[    1.010182] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.020523] msgmni has been set to 3031

[    1.027239] Key type big_key registered

[    1.034631] NET: Registered protocol family 38

[    1.041808] Key type asymmetric registered

[    1.048082] Asymmetric key parser 'x509' registered

[    1.055638] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.067494] io scheduler noop registered

[    1.073972] io scheduler deadline registered (default)

[    1.082193] io scheduler cfq registered

[    1.088598] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.097006] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.108019] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.118109] Non-volatile memory driver v1.3

[    1.124671] Linux agpgart interface v0.103

[    1.131759] crash memory driver: version 1.1

[    1.138607] rdac: device handler registered

[    1.145365] hp_sw: device handler registered

[    1.152263] emc: device handler registered

[    1.158974] alua: device handler registered

[    1.166059] libphy: Fixed MDIO Bus: probed

[    1.173435] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.184515] ehci-pci: EHCI PCI platform driver

[    1.191747] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.201627] ohci-pci: OHCI PCI platform driver

[    1.209097] uhci_hcd: USB Universal Host Controller Interface driver

[    1.219336] usbcore: registered new interface driver usbserial

[    1.228610] usbcore: registered new interface driver usbserial_generic

[    1.239959] usbserial: USB Serial support registered for generic

[    1.249822] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.268318] i8042: No controller found

[    3.274232] mousedev: PS/2 mouse device common for all mice

[    3.391486] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.426704] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.436801] dmi-sysfs: dmi entry is absent.

[    3.444127] hidraw: raw HID events driver (C) Jiri Kosina

[    3.452806] usbcore: registered new interface driver usbhid

[    3.461781] usbhid: USB HID core driver

[    3.469664] drop_monitor: Initializing network drop monitor service

[    3.481349] TCP: cubic registered

[    3.488241] Initializing XFRM netlink socket

[    3.497567] NET: Registered protocol family 10

[    3.507254] NET: Registered protocol family 17

[    3.515980] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.528628] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.539843] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.556197] Loading compiled-in X.509 certificates

[    3.565330] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.583448] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.603163] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.622367] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21479/2049

Failed to read /local/domain/0/backend/vbd/21479/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21479/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21479/2050

Failed to read /local/domain/0/backend/vbd/21479/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21479/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21479/2051

Failed to read /local/domain/0/backend/vbd/21479/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21479/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21479/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21479/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21479/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.712 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795672)

[    0.004019] pid_max: default: 32768 minimum: 301

[    0.005080] Security Framework initialized

[    0.006027] SELinux:  Initializing.

[    0.007047] Yama: becoming mindful.

[    0.008631] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.009826] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.010285] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011046] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.012401] Initializing cgroup subsys memory

[    0.013032] Initializing cgroup subsys devices

[    0.014015] Initializing cgroup subsys freezer

[    0.015022] Initializing cgroup subsys net_cls

[    0.016015] Initializing cgroup subsys blkio

[    0.017021] Initializing cgroup subsys perf_event

[    0.018022] Initializing cgroup subsys hugetlb

[    0.019015] Initializing cgroup subsys pids

[    0.020022] Initializing cgroup subsys net_prio

[    0.021131] FEATURE SPEC_CTRL Not Present

[    0.022017] FEATURE IBPB_SUPPORT Not Present

[    0.023011] CPU: Physical Processor ID: 0

[    0.024017] CPU: Processor Core ID: 0

[    0.025025] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.026010] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.027018] tlb_flushall_shift: 6

[    0.073170] Freeing SMP alternatives: 24k freed

[    0.080600] ftrace: allocating 26850 entries in 105 pages

[    0.087145] cpu 0 spinlock event irq 17

[    0.088017] smpboot: Max logical packages: 1

[    0.089072] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.093354] Brought up 1 CPUs

[    0.094159] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.095009] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.096161] devtmpfs: initialized

[    0.099151] EVM: security.selinux

[    0.100017] EVM: security.ima

[    0.101017] EVM: security.capability

[    0.103000] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.103032] pinctrl core: initialized pinctrl subsystem

[    0.104072] xen:grant_table: Grant tables using version 1 layout

[    0.105064] Grant table initialized

[    0.119000] RTC time: 165:165:165, date: 165/165/65

[    0.119167] NET: Registered protocol family 16

[    0.121019] PCI: setting up Xen PCI frontend stub

[    0.123321] ACPI: Interpreter disabled.

[    0.124054] xen:balloon: Initialising balloon driver

[    0.127085] xen_balloon: Initialising balloon driver

[    0.128239] vgaarb: loaded

[    0.129123] SCSI subsystem initialized

[    0.130082] usbcore: registered new interface driver usbfs

[    0.131052] usbcore: registered new interface driver hub

[    0.132077] usbcore: registered new device driver usb

[    0.133130] dmi: Firmware registration failed.

[    0.134043] PCI: System does not support PCI

[    0.135009] PCI: System does not support PCI

[    0.136183] NetLabel: Initializing

[    0.137018] NetLabel:  domain hash size = 128

[    0.138008] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.139043] NetLabel:  unlabeled traffic allowed by default

[    0.140058] amd_nb: Cannot enumerate AMD northbridges

[    0.141033] Switched to clocksource xen

[    0.155171] pnp: PnP ACPI: disabled

[    0.164175] NET: Registered protocol family 2

[    0.173230] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.187132] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.199634] TCP: Hash tables configured (established 16384 bind 16384)

[    0.211944] TCP: reno registered

[    0.218365] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.230016] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.242567] NET: Registered protocol family 1

[    0.251336] Unpacking initramfs...

[    0.331396] random: fast init done

[    0.811863] Freeing initrd memory: 19404k freed

[    0.828857] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.844194] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.855245] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.866217] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.876589] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.891732] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.903982] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.917256] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.929113] Initialise system trusted keyring

[    0.937869] audit: initializing netlink socket (disabled)

[    0.948101] type=2000 audit(1515851106.545:1): initialized

[    0.992975] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    1.006345] zpool: loaded

[    1.011753] zbud: loaded

[    1.016974] VFS: Disk quotas dquot_6.5.2

[    1.024513] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.037019] msgmni has been set to 3031

[    1.044514] Key type big_key registered

[    1.053089] NET: Registered protocol family 38

[    1.061772] Key type asymmetric registered

[    1.069658] Asymmetric key parser 'x509' registered

[    1.079075] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.093266] io scheduler noop registered

[    1.100910] io scheduler deadline registered (default)

[    1.110763] io scheduler cfq registered

[    1.118346] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.128918] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.141975] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.154649] Non-volatile memory driver v1.3

[    1.162813] Linux agpgart interface v0.103

[    1.170746] crash memory driver: version 1.1

[    1.179257] rdac: device handler registered

[    1.187305] hp_sw: device handler registered

[    1.195603] emc: device handler registered

[    1.203646] alua: device handler registered

[    1.211784] libphy: Fixed MDIO Bus: probed

[    1.219453] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.231762] ehci-pci: EHCI PCI platform driver

[    1.240466] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.252368] ohci-pci: OHCI PCI platform driver

[    1.261066] uhci_hcd: USB Universal Host Controller Interface driver

[    1.273539] usbcore: registered new interface driver usbserial

[    1.285321] usbcore: registered new interface driver usbserial_generic

[    1.297925] usbserial: USB Serial support registered for generic

[    1.309542] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.339419] i8042: No controller found

[    3.346952] mousedev: PS/2 mouse device common for all mice

[    3.465359] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.478094] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.489522] dmi-sysfs: dmi entry is absent.

[    3.497741] hidraw: raw HID events driver (C) Jiri Kosina

[    3.508257] usbcore: registered new interface driver usbhid

[    3.519108] usbhid: USB HID core driver

[    3.526549] drop_monitor: Initializing network drop monitor service

[    3.538685] TCP: cubic registered

[    3.545311] Initializing XFRM netlink socket

[    3.553795] NET: Registered protocol family 10

[    3.562884] NET: Registered protocol family 17

[    3.571651] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.582560] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.593957] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.611079] Loading compiled-in X.509 certificates

[    3.620654] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.638693] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.659264] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.677322] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21480/2049

Failed to read /local/domain/0/backend/vbd/21480/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21480/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21480/2050

Failed to read /local/domain/0/backend/vbd/21480/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21480/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21480/2051

Failed to read /local/domain/0/backend/vbd/21480/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21480/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21480/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21480/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21480/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.819 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795673)

[    0.004009] pid_max: default: 32768 minimum: 301

[    0.005088] Security Framework initialized

[    0.006027] SELinux:  Initializing.

[    0.007046] Yama: becoming mindful.

[    0.008643] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.009752] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.010278] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011040] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.012392] Initializing cgroup subsys memory

[    0.013038] Initializing cgroup subsys devices

[    0.014016] Initializing cgroup subsys freezer

[    0.015021] Initializing cgroup subsys net_cls

[    0.016013] Initializing cgroup subsys blkio

[    0.017017] Initializing cgroup subsys perf_event

[    0.018017] Initializing cgroup subsys hugetlb

[    0.019022] Initializing cgroup subsys pids

[    0.020014] Initializing cgroup subsys net_prio

[    0.021133] FEATURE SPEC_CTRL Not Present

[    0.022009] FEATURE IBPB_SUPPORT Not Present

[    0.023019] CPU: Physical Processor ID: 0

[    0.024008] CPU: Processor Core ID: 0

[    0.025028] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.026017] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.027010] tlb_flushall_shift: 6

[    0.067936] Freeing SMP alternatives: 24k freed

[    0.074423] ftrace: allocating 26850 entries in 105 pages

[    0.080150] cpu 0 spinlock event irq 17

[    0.081019] smpboot: Max logical packages: 1

[    0.082072] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.086512] Brought up 1 CPUs

[    0.087178] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.088019] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.089209] devtmpfs: initialized

[    0.092000] EVM: security.selinux

[    0.092015] EVM: security.ima

[    0.093017] EVM: security.capability

[    0.095000] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.095034] pinctrl core: initialized pinctrl subsystem

[    0.096067] xen:grant_table: Grant tables using version 1 layout

[    0.097070] Grant table initialized

[    0.109710] RTC time: 165:165:165, date: 165/165/65

[    0.110157] NET: Registered protocol family 16

[    0.111899] PCI: setting up Xen PCI frontend stub

[    0.113000] ACPI: Interpreter disabled.

[    0.113064] xen:balloon: Initialising balloon driver

[    0.116035] xen_balloon: Initialising balloon driver

[    0.117229] vgaarb: loaded

[    0.118112] SCSI subsystem initialized

[    0.119099] usbcore: registered new interface driver usbfs

[    0.120030] usbcore: registered new interface driver hub

[    0.121068] usbcore: registered new device driver usb

[    0.122104] dmi: Firmware registration failed.

[    0.123037] PCI: System does not support PCI

[    0.124012] PCI: System does not support PCI

[    0.125154] NetLabel: Initializing

[    0.126007] NetLabel:  domain hash size = 128

[    0.127007] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.128045] NetLabel:  unlabeled traffic allowed by default

[    0.129044] amd_nb: Cannot enumerate AMD northbridges

[    0.130032] Switched to clocksource xen

[    0.142672] pnp: PnP ACPI: disabled

[    0.149900] NET: Registered protocol family 2

[    0.159748] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.171361] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.181446] TCP: Hash tables configured (established 16384 bind 16384)

[    0.191429] TCP: reno registered

[    0.196457] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.205809] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.215831] NET: Registered protocol family 1

[    0.222682] Unpacking initramfs...

[    0.313537] random: fast init done

[    0.751449] Freeing initrd memory: 19404k freed

[    0.790937] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.803321] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.812277] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.821075] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.829506] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.841398] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.850936] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.861264] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.870747] Initialise system trusted keyring

[    0.877954] audit: initializing netlink socket (disabled)

[    0.886444] type=2000 audit(1515851144.172:1): initialized

[    0.928735] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    0.940142] zpool: loaded

[    0.944447] zbud: loaded

[    0.948811] VFS: Disk quotas dquot_6.5.2

[    0.955179] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    0.965481] msgmni has been set to 3031

[    0.971734] Key type big_key registered

[    0.978980] NET: Registered protocol family 38

[    0.985925] Key type asymmetric registered

[    0.992181] Asymmetric key parser 'x509' registered

[    0.999734] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.011351] io scheduler noop registered

[    1.017455] io scheduler deadline registered (default)

[    1.025385] io scheduler cfq registered

[    1.031529] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.040211] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.050422] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.060390] Non-volatile memory driver v1.3

[    1.066928] Linux agpgart interface v0.103

[    1.073336] crash memory driver: version 1.1

[    1.080100] rdac: device handler registered

[    1.086457] hp_sw: device handler registered

[    1.093264] emc: device handler registered

[    1.099722] alua: device handler registered

[    1.106137] libphy: Fixed MDIO Bus: probed

[    1.112447] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.122401] ehci-pci: EHCI PCI platform driver

[    1.129752] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.139547] ohci-pci: OHCI PCI platform driver

[    1.146220] uhci_hcd: USB Universal Host Controller Interface driver

[    1.156146] usbcore: registered new interface driver usbserial

[    1.165566] usbcore: registered new interface driver usbserial_generic

[    1.175533] usbserial: USB Serial support registered for generic

[    1.185025] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.175642] i8042: No controller found

[    3.181966] mousedev: PS/2 mouse device common for all mice

[    3.298903] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.332947] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.341847] dmi-sysfs: dmi entry is absent.

[    3.348309] hidraw: raw HID events driver (C) Jiri Kosina

[    3.356475] usbcore: registered new interface driver usbhid

[    3.365203] usbhid: USB HID core driver

[    3.371135] drop_monitor: Initializing network drop monitor service

[    3.380740] TCP: cubic registered

[    3.385987] Initializing XFRM netlink socket

[    3.392872] NET: Registered protocol family 10

[    3.400357] NET: Registered protocol family 17

[    3.407609] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.416494] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.426053] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.439861] Loading compiled-in X.509 certificates

[    3.447617] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.462289] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.478885] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.493902] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21481/2049

Failed to read /local/domain/0/backend/vbd/21481/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21481/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21481/2050

Failed to read /local/domain/0/backend/vbd/21481/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21481/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21481/2051

Failed to read /local/domain/0/backend/vbd/21481/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21481/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21481/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21481/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21481/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.719 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795673)

[    0.004017] pid_max: default: 32768 minimum: 301

[    0.005098] Security Framework initialized

[    0.006026] SELinux:  Initializing.

[    0.007050] Yama: becoming mindful.

[    0.008686] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.009828] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.010305] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011046] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.012439] Initializing cgroup subsys memory

[    0.013040] Initializing cgroup subsys devices

[    0.014023] Initializing cgroup subsys freezer

[    0.015023] Initializing cgroup subsys net_cls

[    0.016027] Initializing cgroup subsys blkio

[    0.017028] Initializing cgroup subsys perf_event

[    0.018030] Initializing cgroup subsys hugetlb

[    0.019020] Initializing cgroup subsys pids

[    0.020020] Initializing cgroup subsys net_prio

[    0.021153] FEATURE SPEC_CTRL Not Present

[    0.022021] FEATURE IBPB_SUPPORT Not Present

[    0.023014] CPU: Physical Processor ID: 0

[    0.024016] CPU: Processor Core ID: 0

[    0.025017] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.026024] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.027014] tlb_flushall_shift: 6

[    0.097031] Freeing SMP alternatives: 24k freed

[    0.108049] ftrace: allocating 26850 entries in 105 pages

[    0.118148] cpu 0 spinlock event irq 17

[    0.119024] smpboot: Max logical packages: 1

[    0.120081] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.124976] Brought up 1 CPUs

[    0.125181] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.126021] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.127247] devtmpfs: initialized

[    0.131000] EVM: security.selinux

[    0.132016] EVM: security.ima

[    0.133019] EVM: security.capability

[    0.135425] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.136038] pinctrl core: initialized pinctrl subsystem

[    0.137076] xen:grant_table: Grant tables using version 1 layout

[    0.138097] Grant table initialized

[    0.158860] RTC time: 165:165:165, date: 165/165/65

[    0.159193] NET: Registered protocol family 16

[    0.161024] PCI: setting up Xen PCI frontend stub

[    0.163416] ACPI: Interpreter disabled.

[    0.164094] xen:balloon: Initialising balloon driver

[    0.168063] xen_balloon: Initialising balloon driver

[    0.169239] vgaarb: loaded

[    0.170157] SCSI subsystem initialized

[    0.171131] usbcore: registered new interface driver usbfs

[    0.172071] usbcore: registered new interface driver hub

[    0.173126] usbcore: registered new device driver usb

[    0.174216] dmi: Firmware registration failed.

[    0.175063] PCI: System does not support PCI

[    0.176016] PCI: System does not support PCI

[    0.177325] NetLabel: Initializing

[    0.178020] NetLabel:  domain hash size = 128

[    0.179025] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.180073] NetLabel:  unlabeled traffic allowed by default

[    0.181114] amd_nb: Cannot enumerate AMD northbridges

[    0.182087] Switched to clocksource xen

[    0.199245] pnp: PnP ACPI: disabled

[    0.208597] NET: Registered protocol family 2

[    0.219064] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.234639] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.248032] TCP: Hash tables configured (established 16384 bind 16384)

[    0.261014] TCP: reno registered

[    0.267848] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.279841] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.292765] NET: Registered protocol family 1

[    0.301892] Unpacking initramfs...

[    0.315045] random: fast init done

[    0.864749] Freeing initrd memory: 19404k freed

[    0.882503] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.898833] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.910245] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.921758] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.932642] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.948836] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.961361] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.975966] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.988407] Initialise system trusted keyring

[    0.997143] audit: initializing netlink socket (disabled)

[    1.007983] type=2000 audit(1515851183.567:1): initialized

[    1.054484] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    1.068680] zpool: loaded

[    1.074123] zbud: loaded

[    1.079618] VFS: Disk quotas dquot_6.5.2

[    1.087497] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.100464] msgmni has been set to 3031

[    1.108301] Key type big_key registered

[    1.117363] NET: Registered protocol family 38

[    1.126362] Key type asymmetric registered

[    1.134694] Asymmetric key parser 'x509' registered

[    1.144411] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.159339] io scheduler noop registered

[    1.167597] io scheduler deadline registered (default)

[    1.177823] io scheduler cfq registered

[    1.185634] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.196439] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.210179] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.223251] Non-volatile memory driver v1.3

[    1.231632] Linux agpgart interface v0.103

[    1.240300] crash memory driver: version 1.1

[    1.249463] rdac: device handler registered

[    1.257886] hp_sw: device handler registered

[    1.266558] emc: device handler registered

[    1.274755] alua: device handler registered

[    1.283045] libphy: Fixed MDIO Bus: probed

[    1.291250] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.304124] ehci-pci: EHCI PCI platform driver

[    1.312871] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.325432] ohci-pci: OHCI PCI platform driver

[    1.334448] uhci_hcd: USB Universal Host Controller Interface driver

[    1.347205] usbcore: registered new interface driver usbserial

[    1.358851] usbcore: registered new interface driver usbserial_generic

[    1.371622] usbserial: USB Serial support registered for generic

[    1.383781] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.416560] i8042: No controller found

[    3.424758] mousedev: PS/2 mouse device common for all mice

[    3.543888] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.556864] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.568766] dmi-sysfs: dmi entry is absent.

[    3.577438] hidraw: raw HID events driver (C) Jiri Kosina

[    3.588561] usbcore: registered new interface driver usbhid

[    3.599564] usbhid: USB HID core driver

[    3.607238] drop_monitor: Initializing network drop monitor service

[    3.620381] TCP: cubic registered

[    3.627568] Initializing XFRM netlink socket

[    3.636822] NET: Registered protocol family 10

[    3.646443] NET: Registered protocol family 17

[    3.655642] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.666897] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.679130] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.697134] Loading compiled-in X.509 certificates

[    3.707843] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.726809] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.748045] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.767979] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21482/2049

Failed to read /local/domain/0/backend/vbd/21482/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21482/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21482/2050

Failed to read /local/domain/0/backend/vbd/21482/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21482/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21482/2051

Failed to read /local/domain/0/backend/vbd/21482/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21482/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21482/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21482/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21482/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.806 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795672)

[    0.004000] pid_max: default: 32768 minimum: 301

[    0.004097] Security Framework initialized

[    0.005028] SELinux:  Initializing.

[    0.006053] Yama: becoming mindful.

[    0.007650] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.008825] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.009272] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.010050] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011423] Initializing cgroup subsys memory

[    0.012037] Initializing cgroup subsys devices

[    0.013023] Initializing cgroup subsys freezer

[    0.014031] Initializing cgroup subsys net_cls

[    0.015034] Initializing cgroup subsys blkio

[    0.016020] Initializing cgroup subsys perf_event

[    0.017020] Initializing cgroup subsys hugetlb

[    0.018020] Initializing cgroup subsys pids

[    0.019028] Initializing cgroup subsys net_prio

[    0.020173] FEATURE SPEC_CTRL Not Present

[    0.021022] FEATURE IBPB_SUPPORT Not Present

[    0.022023] CPU: Physical Processor ID: 0

[    0.023019] CPU: Processor Core ID: 0

[    0.024031] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.025033] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.026016] tlb_flushall_shift: 6

[    0.064237] Freeing SMP alternatives: 24k freed

[    0.070363] ftrace: allocating 26850 entries in 105 pages

[    0.076160] cpu 0 spinlock event irq 17

[    0.077018] smpboot: Max logical packages: 1

[    0.078083] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.082401] Brought up 1 CPUs

[    0.083223] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.084013] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.085233] devtmpfs: initialized

[    0.088000] EVM: security.selinux

[    0.088026] EVM: security.ima

[    0.089018] EVM: security.capability

[    0.091000] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.091040] pinctrl core: initialized pinctrl subsystem

[    0.092123] xen:grant_table: Grant tables using version 1 layout

[    0.093114] Grant table initialized

[    0.105228] RTC time: 165:165:165, date: 165/165/65

[    0.106214] NET: Registered protocol family 16

[    0.107909] PCI: setting up Xen PCI frontend stub

[    0.109000] ACPI: Interpreter disabled.

[    0.109082] xen:balloon: Initialising balloon driver

[    0.112029] xen_balloon: Initialising balloon driver

[    0.113210] vgaarb: loaded

[    0.114140] SCSI subsystem initialized

[    0.115106] usbcore: registered new interface driver usbfs

[    0.116041] usbcore: registered new interface driver hub

[    0.117099] usbcore: registered new device driver usb

[    0.118153] dmi: Firmware registration failed.

[    0.119052] PCI: System does not support PCI

[    0.120016] PCI: System does not support PCI

[    0.121242] NetLabel: Initializing

[    0.122018] NetLabel:  domain hash size = 128

[    0.123009] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.124060] NetLabel:  unlabeled traffic allowed by default

[    0.125092] amd_nb: Cannot enumerate AMD northbridges

[    0.126043] Switched to clocksource xen

[    0.140743] pnp: PnP ACPI: disabled

[    0.149606] NET: Registered protocol family 2

[    0.158530] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.172639] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.185290] TCP: Hash tables configured (established 16384 bind 16384)

[    0.197702] TCP: reno registered

[    0.204071] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.216075] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.228633] NET: Registered protocol family 1

[    0.237750] Unpacking initramfs...

[    0.369479] random: fast init done

[    0.812227] Freeing initrd memory: 19404k freed

[    0.829207] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.847981] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.858970] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.870339] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.881408] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.896268] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.908284] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.921239] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.932918] Initialise system trusted keyring

[    0.941410] audit: initializing netlink socket (disabled)

[    0.951788] type=2000 audit(1515851217.858:1): initialized

[    1.021248] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    1.035196] zpool: loaded

[    1.035196] zbud: loaded

[    1.045980] VFS: Disk quotas dquot_6.5.2

[    1.053654] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.066576] msgmni has been set to 3031

[    1.067244] Key type big_key registered

[    1.082965] NET: Registered protocol family 38

[    1.091844] Key type asymmetric registered

[    1.099826] Asymmetric key parser 'x509' registered

[    1.109567] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.123911] io scheduler noop registered

[    1.131564] io scheduler deadline registered (default)

[    1.141809] io scheduler cfq registered

[    1.149393] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.160247] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.173417] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.186569] Non-volatile memory driver v1.3

[    1.194905] Linux agpgart interface v0.103

[    1.203037] crash memory driver: version 1.1

[    1.211751] rdac: device handler registered

[    1.220166] hp_sw: device handler registered

[    1.229060] emc: device handler registered

[    1.237808] alua: device handler registered

[    1.246499] libphy: Fixed MDIO Bus: probed

[    1.254925] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.268099] ehci-pci: EHCI PCI platform driver

[    1.269023] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.289595] ohci-pci: OHCI PCI platform driver

[    1.290550] uhci_hcd: USB Universal Host Controller Interface driver

[    1.311474] usbcore: registered new interface driver usbserial

[    1.322858] usbcore: registered new interface driver usbserial_generic

[    1.335999] usbserial: USB Serial support registered for generic

[    1.349056] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.400514] i8042: No controller found

[    3.408389] mousedev: PS/2 mouse device common for all mice

[    3.527531] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.540338] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.551716] dmi-sysfs: dmi entry is absent.

[    3.559914] hidraw: raw HID events driver (C) Jiri Kosina

[    3.570512] usbcore: registered new interface driver usbhid

[    3.581218] usbhid: USB HID core driver

[    3.588925] drop_monitor: Initializing network drop monitor service

[    3.601194] TCP: cubic registered

[    3.608001] Initializing XFRM netlink socket

[    3.616800] NET: Registered protocol family 10

[    3.625903] NET: Registered protocol family 17

[    3.634501] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.645168] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.656568] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.674563] Loading compiled-in X.509 certificates

[    3.683010] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.700538] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.720636] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.738829] registered taskstats version 1

	2018-01-13T13:47:04.000Z
kjohnson

kjohnson

2018-01-15 21:06

reporter   ~0030933

Is there a bug opened on RHEL bugzilla about this issue for both el6 and el7 kernels?
mlarkin

mlarkin

2018-01-15 23:53

reporter   ~0030936

Myself and a couple other engineers were able to track down the cause of the kernel corruption (which manifests itself like folks have described in this thread - either a crash/panic, or stuck at "registered taskstats version 1".

I opened a bug with RHEL bugzilla but it appears that it's taking a long time to wind its way through their system. No update from them yet (and the bug is still marked 'private' there for some reason).

I'm including the content of what I sent them below, for the sake of sharing more in-depth analysis.

We support customers who are not likely going to accept "upgrade to kernel 4.9", so we really need this fixed in 3.10, please :)

-----

We are seeing kernel stack corruption on Xen PV VMs with the following kernel version:

3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Dec 28 14:23:39 EST 2017

The previous kernel, 3.10.0-693.11.1.el7.x86_64 does not show this problem. Fully virtualized instances do not show this problem.

The problem manifests itself during the first guest userland trap (typically a #PF while reading in the first instruction of the first process). In this crash, the kernel stack is reported as being corrupt and the guest VM crashes with the following trace:

[ 1.816298] registered taskstats version 1
[ 2.132228] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffff816ab65d
[ 2.132228]
[ 2.132314] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W ------------ 3.10.0-693.11.6.el7.x86_64 #1
[ 2.132375] Call Trace:
[ 2.132405] [<ffffffff816a5ea1>] dump_stack+0x19/0x1b
[ 2.132439] [<ffffffff8169fd64>] panic+0xe8/0x20d
[ 2.132477] [<ffffffff816ab65d>] ? __schedule+0x95d/0x9b0
[ 2.132533] [<ffffffff810ca4d0>] ? switched_to_idle+0x10/0x10
[ 2.132575] [<ffffffff810891bb>] __stack_chk_fail+0x1b/0x20
[ 2.132608] [<ffffffff816ab65d>] __schedule+0x95d/0x9b0
[ 2.132645] [<ffffffff81020d9d>] ? xen_force_evtchn_callback+0xd/0x10
[ 2.132682] [<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[ 2.132717] [<ffffffff816ac5f9>] ? schedule_preempt_disabled+0x29/0x70
[ 2.132754] [<ffffffff816b3b15>] ? do_page_fault+0x35/0x90
[ 2.132789] [<ffffffff816af8f8>] ? page_fault+0x28/0x30
[ 2.132825] [<ffffffff816afdda>] ? error_exit+0x2a/0x60
[ 2.132860] [<ffffffff816af53a>] ? retint_swapgs+0x5/0x6
[ 2.132895] [<ffffffff81021780>] ? xen_sysret32_end+0x8/0x8


We tracked this down internally to a kernel %rsp set to 0. In Xen PV, the kernel stack pointer used to be set via a call through "load_sp0" in /arch/x86/kernel/cpu/common.c (which, in the older working version used a PV hypercall if needed), but that call was removed in 3.10.0-693.11.6.el7.x86_64 and replaced with different code that appears to use __this_cpu_write (and doesn't result in any hypercall being dispatched):

- load_sp0(t, &current->thread);
+ __this_cpu_write(init_tss.x86_tss.sp0,
+ (unsigned long) t + offsetofend(struct tss_struct,
+ stack));

Because of this, the PV-specific load_sp0 code is never called, the hypercall to Xen to set the kernel stack is never made, and kernel %rsp remains set to 0.

The above diff snippet was created as the result of diffing 3.10.0-693.11.1 source (working) vs 3.10.0-693.11.6 (not-working), and is in /arch/x86/kernel/cpu/common.c around line 1484 in the 3.10.0-693.11.6 tree. The change appears to be related to the recent KAISER/KPTI fixes.

As an experiment, I manually reverted just the load_sp0 part of the change (with all the associated baggage that involved), and the boot does proceed somewhat further but crashes in userland later with bizarre memory allocation failures. Some of the crashes at that point appear to be related to the direct 1:1 map being missing, but other OOPSes are reporting corrupt page table structures. I'm pretty sure there are other fixes required than just the load_sp0 bit; the diff between the two trees was nearly 8,000 lines of KAISER related stuff, and it's likely more PV code became a victim of that change.
bill_mcgonigle

bill_mcgonigle

2018-01-16 16:11

reporter   ~0030938

@mlarkin: have you seen :

https://patchwork.kernel.org/patch/10049093/

? PV isn't vulnerable to the attacks, per Intel, so it's OK to disable KAISER on PV (or use the previous kernel-plus for now).

PV not booting on RHEL kernel has been an issue since 7.4 but that was fixed by kernel-plus prior to KAISER. Unfortunately CentOS hasn't yet published the Source for 3.10.0-693.11.6.el7.x86_64, otherwise I would have built a test kernel with the KAISER/PV patch last weekend. :( I have a bug open for that problem as well.
mlarkin

mlarkin

2018-01-16 21:06

reporter   ~0030942

@bill_mcgonigle: yeah, saw that. cpu_init() does "ignore" KAISER when running on PV, but as I pointed out above, other areas of the KAISER diff don't appear to be as careful, and important PV initialization was removed (eg load_sp0() and other things causing later MMU-related crashes if the part of the diff that removed load_sp0() is manually added back).

As for "PV not booting on RHEL 7.4 kernel", we have not had that issue, it works fine for us.

As for the source code, the srpms are available for 3.10.0-693.11.6.el7.x86_64 in the centos vault, that's how I created the diff between the working 11.1 version and broken 11.6 version.

kernel-plus is not going to be an option for some/most of the customers we support, unfortunately.
kjohnson

kjohnson

2018-01-16 21:14

reporter   ~0030943

PV is indeed broken since 7.4 kernel (bug 13763) but a patch has been submitted to RHEL and it should be included the 7.5 kernel (already available in the -plus kernel). However, it's now broken again with the KAISER patch.
mlarkin

mlarkin

2018-01-16 22:11

reporter   ~0030944

Aside from the bug I filed at RH for this new issue, are there any other bugs filed? I'd like to be able to reference them if/when we get on the phone with their support ... Thanks!
awelbers

awelbers

2018-01-31 13:33

reporter   ~0031133

The new Centos Plus kernel doesn't fix this problem.
If I try vmlinuz-3.10.0-693.17.1.el7.centos.plus.x86_64 same results.

Has/Will Redhat or CentOS fix this problem?
Do we have to switch to the unofficial LTS-kernel from Elrepo?
kstange

kstange

2018-02-06 23:00

reporter   ~0031171

Since we have a fix now from CentOS 6 in kernel 696.20.1, I diffed out the changes and forward ported the modification to CentOS 7's kernel cpu_init() function. This function is almost identical between 2.6.32 and 3.10.0. It remains that way in 3.16.0 at least as well, which I compared for reference. Restoring load_sp0() as mentioned in comment ~0030936 seems not enough because RH's KAISER patch is somewhat different from mainline, and thus RH has done additional TSS work which is visible in the CentOS 6 fix. There are some other changes outside cpu_init() that seemed like they *could* also be pertinent, but it's hard to tell without any comments from RH.

I've attached my patch which I compiled into a test kernel and booted successfully. I've seen no signs of memory issues yet. I ran "memtester" inside the guest to put some memory pressure on it and nothing crashed, panicked, or produced errors. I also performed a successful hot migration over Xen 4.8.2-11 as well without incident.

xen-restore-load-sp0.patch (769 bytes)
--- a/arch/x86/kernel/cpu/common.c	2017-12-28 12:59:43.000000000 -0600
+++ b/arch/x86/kernel/cpu/common.c	2018-02-06 14:08:00.625010056 -0600
@@ -1484,9 +1484,16 @@
 	BUG_ON(me->mm);
 	enter_lazy_tlb(&init_mm, me);
 
-	__this_cpu_write(init_tss.x86_tss.sp0,
-			 (unsigned long) t + offsetofend(struct tss_struct,
-							 stack));
+	/*
+	 * load_sp0() is required for paravirt establishment of the vCPU sp0.
+	 */
+	v = current->thread.sp0;
+	current->thread.sp0 = (unsigned long)t +
+	                      offsetofend(struct tss_struct, stack);
+	load_sp0(t, &current->thread);
+	WRITE_ONCE(t->x86_tss.sp0, current->thread.sp0);
+	current->thread.sp0 = v;        /* Restore original value */
+
 	set_tss_desc(cpu, t);
 	load_TR_desc();
 	load_LDT(&init_mm.context);
toracat

toracat

2018-02-06 23:10

manager   ~0031172

Thanks. I will build a test kernel with your patch.
toracat

toracat

2018-02-07 02:10

manager   ~0031173

A patched set of the centosplus kernel is now available at:

https://people.centos.org/toracat/kernel/7/plus/bug14347/

Please test. Feedback from the affected users appreciated.
awelbers

awelbers

2018-02-07 09:33

reporter   ~0031177

Looks good so far!
I'm testing it on 2 system.
adrian27

adrian27

2018-02-07 11:12

reporter   ~0031178

I have tested this patched centosplus kernel in an Amazon PV AMI instance, and at least the VM is booting up successfully:

$ uname -a
Linux ip-10-74-164-143.eu-west-1.compute.internal 3.10.0-693.17.1.el7.bug14347.centos.plus.x86_64 #1 SMP Tue Feb 6 16:27:03 PST 2018 x86_64 x86_64 x86_64 GNU/Linux
cyberneticos

cyberneticos

2018-02-07 12:43

reporter   ~0031179

It works for me too! :)

# uname -r
3.10.0-693.17.1.el7.bug14347.centos.plus.x86_64
toracat

toracat

2018-02-07 14:20

manager   ~0031181

Excellent. The next plus kernel update will include the patch. @kstange, you did it again!
awelbers

awelbers

2018-02-07 15:35

reporter   ~0031184

Great. I also have to say many thx!
mlarkin

mlarkin

2018-02-07 19:28

reporter   ~0031186

Testing the diff now. Hey @kstange, the KAISER patch also removed another load_sp0 call in __switch_to (arch/x86/kernel/process_64.c). We're wondering here if that call is also needed? (Although from the reports, it seems that the load_sp0 call in cpu_init was the important one).
kstange

kstange

2018-02-07 20:10

reporter   ~0031187

@mlarkin My patch is based only on the solution that Red Hat put into CentOS 6 kernel 696.20.1. Looking at the differences in process_64.c on CentOS 6, the KAISER patch limits load_sp0() there to be called only on x86_32. That seems strange since I think x86_32 would never run code from process_64.c in the first place. Since the CentOS 7 kernel isn't intended for x86_32 they seem to have just deleted the call entirely instead. For the CentOS 6 PV fix, they didn't go back and undo the change made to process_64.c. I've looked around KPTI patches for various ML kernel versions and haven't found a similar change made in any of them regarding load_sp0(), so I'm not really sure why Red Hat was messing with this code. I checked 3.2, 3.16, and 4.9, and I also looked over https://github.com/IAIK/KAISER for clues.

The TL;DR version is that I'm assuming it isn't necessary, but I'm not really qualified to make the determination myself, and I'm trusting Red Hat did the right thing.
kstange

kstange

2018-02-07 20:21

reporter   ~0031188

Also: It's possible that this isn't KAISER/Meltdown but rather something related to Spectre that Red Hat worked on independently from upstream. Both patch sets are mashed together as far as we can see them, so it's difficult to see where one ends and the other begins. Someone with access to the RH kernel patch browser might be able to distill out the individual patches and their justifications further.
mlarkin

mlarkin

2018-02-07 22:48

reporter   ~0031191

Just following up that the patch worked here as well. Thanks!
toracat

toracat

2018-04-11 00:30

manager   ~0031594

RHEL 7.5 is out (kernel-3.10.0-862.el7). The patch in this bug report is now in this kernel, so will be removed from the centosplus kernel.
toracat

toracat

2018-05-11 17:04

manager   ~0031766

Could someone check to see if the problem persists in the 7.5 kernel (current is kernel-3.10.0-862.2.3.el7) ?
kstange

kstange

2018-05-11 18:39

reporter   ~0031771

In my testing, 3.10.0-862.el7 was working fine with no extra patches, but i have not tested 862.2.3 yet.
kjohnson

kjohnson

2018-05-11 19:43

reporter   ~0031772

Works for me:

[root@cpanel2 ~]# virt-what && cat /proc/version
xen
xen-domU
Linux version 3.10.0-862.2.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Wed May 9 18:05:47 UTC 2018
awelbers

awelbers

2018-05-11 21:53

reporter   ~0031774

For me it works also:

# virt-what && cat /proc/version
xen
xen-domU
Linux version 3.10.0-862.2.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Wed May 9 18:05:47 UTC 2018
toracat

toracat

2018-05-11 22:23

manager   ~0031775

Thank you all for your response. I'm closing this ticket as 'resolved'.

Issue History

Date Modified Username Field Change
2018-01-09 02:02 awelbers New Issue
2018-01-09 02:02 awelbers File Added: xen-centos-kernel-plus-endless-loop-20170109.txt
2018-01-09 03:48 bill_mcgonigle Note Added: 0030884
2018-01-09 06:57 toracat Status new => acknowledged
2018-01-09 06:57 toracat Note Added: 0030886
2018-01-09 07:01 toracat Relationship added related to 0014336
2018-01-09 20:15 bill_mcgonigle Note Added: 0030896
2018-01-13 17:00 bill_mcgonigle Note Added: 0030929
2018-01-13 19:43 adrian27 File Added: aws-kernel-plus-3.10.0-693.11.6.el7.centos.plus.x86_64.log.txt
2018-01-13 19:43 adrian27 Note Added: 0030931
2018-01-15 21:06 kjohnson Note Added: 0030933
2018-01-15 23:53 mlarkin Note Added: 0030936
2018-01-16 16:11 bill_mcgonigle Note Added: 0030938
2018-01-16 21:06 mlarkin Note Added: 0030942
2018-01-16 21:14 kjohnson Note Added: 0030943
2018-01-16 22:11 mlarkin Note Added: 0030944
2018-01-31 13:33 awelbers Note Added: 0031133
2018-02-06 23:00 kstange File Added: xen-restore-load-sp0.patch
2018-02-06 23:00 kstange Note Added: 0031171
2018-02-06 23:10 toracat Note Added: 0031172
2018-02-06 23:47 toracat Relationship added related to 0013763
2018-02-07 02:10 toracat Note Added: 0031173
2018-02-07 09:33 awelbers Note Added: 0031177
2018-02-07 11:12 adrian27 Note Added: 0031178
2018-02-07 12:43 cyberneticos Note Added: 0031179
2018-02-07 14:20 toracat Note Added: 0031181
2018-02-07 14:20 toracat Status acknowledged => assigned
2018-02-07 15:35 awelbers Note Added: 0031184
2018-02-07 19:28 mlarkin Note Added: 0031186
2018-02-07 20:10 kstange Note Added: 0031187
2018-02-07 20:21 kstange Note Added: 0031188
2018-02-07 22:48 mlarkin Note Added: 0031191
2018-04-11 00:30 toracat Note Added: 0031594
2018-05-11 17:04 toracat Note Added: 0031766
2018-05-11 17:04 toracat Status assigned => feedback
2018-05-11 18:39 kstange Note Added: 0031771
2018-05-11 19:43 kjohnson Note Added: 0031772
2018-05-11 21:53 awelbers Note Added: 0031774
2018-05-11 21:53 awelbers Status feedback => assigned
2018-05-11 22:23 toracat Note Added: 0031775
2018-05-11 22:23 toracat Status assigned => resolved
2018-05-11 22:23 toracat Resolution open => fixed