View Issue Details

IDProjectCategoryView StatusLast Update
0013763CentOS-7kernelpublic2018-05-10 14:25
Reporterkstange 
PriorityhighSeveritycrashReproducibilityalways
Status resolvedResolutionfixed 
Platformx86_64OScentosOS Version7.4
Product Version 
Target VersionFixed in Version7.5.1804 
Summary0013763: CentOS 7.4 kernel (3.10.0-693*) fails to boot as Xen PV guest
DescriptionAfter a mailing list post indicating Xen PV guest not booting with CR kernel for 7.4:

https://lists.centos.org/pipermail/centos-virt/2017-August/005618.html

I began to investigate and determined that it appears something is causing a page allocation failure during systemd-udev initialization, which prevents proper booting of the system.

https://lists.centos.org/pipermail/centos-virt/2017-September/005623.html

Failure is:

[ 1.970630] ------------[ cut here ]------------
[ 1.970651] WARNING: CPU: 2 PID: 225 at mm/vmalloc.c:131
vmap_page_range_noflush+0x2c1/0x350
[ 1.970660] Modules linked in:
[ 1.970668] CPU: 2 PID: 225 Comm: systemd-udevd Not tainted
3.10.0-693.1.1.el7.x86_64 #1
[ 1.970677] 0000000000000000 000000008cddc75d ffff8803e8587bd8
ffffffff816a3d91
[ 1.970688] ffff8803e8587c18 ffffffff810879c8 00000083811c14e8
ffff8800066eb000
[ 1.970698] 0000000000000001 ffff8803e86d6940 ffffffffc0000000
0000000000000000
[ 1.970708] Call Trace:
[ 1.970725] [<ffffffff816a3d91>] dump_stack+0x19/0x1b
[ 1.970736] [<ffffffff810879c8>] __warn+0xd8/0x100
[ 1.970742] [<ffffffff81087b0d>] warn_slowpath_null+0x1d/0x20
[ 1.970748] [<ffffffff811c0781>] vmap_page_range_noflush+0x2c1/0x350
[ 1.970758] [<ffffffff811c083e>] map_vm_area+0x2e/0x40
[ 1.970765] [<ffffffff811c1590>] __vmalloc_node_range+0x170/0x270
[ 1.970774] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.970781] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.970792] [<ffffffff8105f143>] module_alloc+0x73/0xd0
[ 1.970798] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.970804] [<ffffffff810fe754>] module_alloc_update_bounds+0x14/0x70
[ 1.970811] [<ffffffff810ff2d2>] load_module+0xb02/0x29e0
[ 1.970817] [<ffffffff811c0717>] ? vmap_page_range_noflush+0x257/0x350
[ 1.970823] [<ffffffff811c083e>] ? map_vm_area+0x2e/0x40
[ 1.970829] [<ffffffff811c1590>] ? __vmalloc_node_range+0x170/0x270
[ 1.970838] [<ffffffff81101249>] ? SyS_init_module+0x99/0x110
[ 1.970846] [<ffffffff81101275>] SyS_init_module+0xc5/0x110
[ 1.970856] [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
[ 1.970862] ---[ end trace 2117480876ed90d2 ]---
[ 1.970869] vmalloc: allocation failure, allocated 24576 of 28672 bytes
[ 1.970874] systemd-udevd: page allocation failure: order:0, mode:0xd2
[ 1.970883] CPU: 2 PID: 225 Comm: systemd-udevd Tainted: G W
   ------------ 3.10.0-693.1.1.el7.x86_64 #1
[ 1.970894] 00000000000000d2 000000008cddc75d ffff8803e8587c48
ffffffff816a3d91
[ 1.970910] ffff8803e8587cd8 ffffffff81188810 ffffffff8190ea38
ffff8803e8587c68
[ 1.970923] ffffffff00000018 ffff8803e8587ce8 ffff8803e8587c88
000000008cddc75d
[ 1.970939] Call Trace:
[ 1.970946] [<ffffffff816a3d91>] dump_stack+0x19/0x1b
[ 1.970961] [<ffffffff81188810>] warn_alloc_failed+0x110/0x180
[ 1.970971] [<ffffffff811c1654>] __vmalloc_node_range+0x234/0x270
[ 1.970981] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.970989] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.970999] [<ffffffff8105f143>] module_alloc+0x73/0xd0
[ 1.971031] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.971038] [<ffffffff810fe754>] module_alloc_update_bounds+0x14/0x70
[ 1.971046] [<ffffffff810ff2d2>] load_module+0xb02/0x29e0
[ 1.971052] [<ffffffff811c0717>] ? vmap_page_range_noflush+0x257/0x350
[ 1.971061] [<ffffffff811c083e>] ? map_vm_area+0x2e/0x40
[ 1.971067] [<ffffffff811c1590>] ? __vmalloc_node_range+0x170/0x270
[ 1.971075] [<ffffffff81101249>] ? SyS_init_module+0x99/0x110
[ 1.971081] [<ffffffff81101275>] SyS_init_module+0xc5/0x110
[ 1.971088] [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
[ 1.971094] Mem-Info:
[ 1.971103] active_anon:875 inactive_anon:2049 isolated_anon:0
[ 1.971103] active_file:791 inactive_file:8841 isolated_file:0
[ 1.971103] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1.971103] slab_reclaimable:1732 slab_unreclaimable:1629
[ 1.971103] mapped:1464 shmem:2053 pagetables:480 bounce:0
[ 1.971103] free:4065966 free_pcp:763 free_cma:0
[ 1.971131] Node 0 DMA free:15912kB min:12kB low:12kB high:16kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB
managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 1.971217] lowmem_reserve[]: 0 4063 16028 16028
[ 1.971226] Node 0 DMA32 free:4156584kB min:4104kB low:5128kB
high:6156kB active_anon:952kB inactive_anon:1924kB active_file:0kB
inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB
present:4177920kB managed:4162956kB mlocked:0kB dirty:0kB writeback:0kB
mapped:4kB shmem:1928kB slab_reclaimable:240kB slab_unreclaimable:504kB
kernel_stack:32kB pagetables:592kB unstable:0kB bounce:0kB
free_pcp:1760kB local_pcp:288kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
[ 1.971264] lowmem_reserve[]: 0 0 11964 11964
[ 1.971273] Node 0 Normal free:12091564kB min:12088kB low:15108kB
high:18132kB active_anon:2352kB inactive_anon:6272kB active_file:3164kB
inactive_file:35364kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB present:12591104kB managed:12251788kB mlocked:0kB
dirty:0kB writeback:0kB mapped:5852kB shmem:6284kB
slab_reclaimable:6688kB slab_unreclaimable:6012kB kernel_stack:880kB
pagetables:1328kB unstable:0kB bounce:0kB free_pcp:1196kB
local_pcp:152kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
[ 1.971309] lowmem_reserve[]: 0 0 0 0
[ 1.971316] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 2*64kB (U)
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) =
15912kB
[ 1.971343] Node 0 DMA32: 7*4kB (M) 18*8kB (UM) 7*16kB (EM) 3*32kB
(EM) 1*64kB (E) 2*128kB (UM) 1*256kB (E) 4*512kB (UM) 4*1024kB (UEM)
4*2048kB (EM) 1011*4096kB (M) = 4156348kB
[ 1.971377] Node 0 Normal: 64*4kB (UEM) 10*8kB (UEM) 6*16kB (EM)
3*32kB (EM) 3*64kB (UE) 3*128kB (UEM) 1*256kB (E) 2*512kB (UE) 0*1024kB
1*2048kB (M) 2951*4096kB (M) = 12091728kB
[ 1.971413] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ 1.971425] 11685 total pagecache pages
[ 1.971430] 0 pages in swap cache
[ 1.971437] Swap cache stats: add 0, delete 0, find 0/0
[ 1.971444] Free swap = 0kB
[ 1.971451] Total swap = 0kB
[ 1.971456] 4196255 pages RAM
[ 1.971462] 0 pages HighMem/MovableOnly
[ 1.971467] 88591 pages reserved
Steps To ReproduceUpgrade to 3.10.0-693 or later and reboot.
Additional InformationKernel boots fine if you switch back to 514.26.2. I'm putting this here and then filing a bug upstream to RHEL as well.
TagsNo tags attached.
abrt_hash
URL

Relationships

related to 0014347 resolvedtoracat kernel-plus-3.10.0-693.11.6.el7.centos.plus under Xen4 runs in endless loop 

Activities

kstange

kstange

2017-09-01 20:12

reporter   ~0029977

Running xen version on host:

xen-4.4.4-26
kernel-4.9.44-29
kstange

kstange

2017-09-01 22:15

reporter   ~0029978

Created a bug at RH bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=1487754

Anyone who wants a CC to the bug to comment or view, please let me know the email to add... you can ping me on IRC (kstange) if you would rather.
kstange

kstange

2017-09-01 23:09

reporter  

kernel-3.10.0-693.1.1-boot.log (107,196 bytes)
arrfab

arrfab

2017-09-02 09:49

administrator   ~0029979

Confirmed through CentOS QA jobs and kernel panic so unable to setup a new PV domU guest with the 7.4.1708 tree
kstange

kstange

2017-09-06 19:06

reporter   ~0030014

Things I have done since filing this bug:

- I tested booting my CentOS virt kernel 4.9.44 and it boots correctly.

- I tested an official RHEL 7.4 PV guest and it does not boot.

- From the Red Hat bug, it was suggested that I rebuild with CONFIG_RANDOMIZE_BASE=n and this allowed the kernel to boot.

- I did some digging based on this change and found this thread:

https://lists.xen.org/archives/html/xen-devel/2014-08/msg01024.html

The patch proposed is here in its final form:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v4.13&id=0b5a50635fc916cf46e3de0b819a61fc3f17e7ee

It doesn't apply cleanly to 3.10.0, but I backported it to the 693.1.1 kernel and applied it, and I can now boot my PV guest properly with CONFIG_RANDOMIZE_BASE=y.

- I have asked, but I am doubtful that Red Hat will accept this patch. We could potentially apply it to CentOS Plus kernels, but that doesn't solve the installer problem unless we have an alternate installer for PV guests.

xen-dont-copy-bogus-duplicate-entries-into-kernel-page-tables.patch (5,502 bytes)
From 0b5a50635fc916cf46e3de0b819a61fc3f17e7ee Mon Sep 17 00:00:00 2001
From: Stefan Bader <stefan.bader@canonical.com>
Date: Tue, 2 Sep 2014 11:16:01 +0100
Subject: x86/xen: don't copy bogus duplicate entries into kernel page tables

When RANDOMIZE_BASE (KASLR) is enabled; or the sum of all loaded
modules exceeds 512 MiB, then loading modules fails with a warning
(and hence a vmalloc allocation failure) because the PTEs for the
newly-allocated vmalloc address space are not zero.

  WARNING: CPU: 0 PID: 494 at linux/mm/vmalloc.c:128
           vmap_page_range_noflush+0x2a1/0x360()

This is caused by xen_setup_kernel_pagetables() copying
level2_kernel_pgt into level2_fixmap_pgt, overwriting many non-present
entries.

Without KASLR, the normal kernel image size only covers the first half
of level2_kernel_pgt and module space starts after that.

L4[511]->level3_kernel_pgt[510]->level2_kernel_pgt[  0..255]->kernel
                                                  [256..511]->module
                          [511]->level2_fixmap_pgt[  0..505]->module

This allows 512 MiB of of module vmalloc space to be used before
having to use the corrupted level2_fixmap_pgt entries.

With KASLR enabled, the kernel image uses the full PUD range of 1G and
module space starts in the level2_fixmap_pgt. So basically:

L4[511]->level3_kernel_pgt[510]->level2_kernel_pgt[0..511]->kernel
                          [511]->level2_fixmap_pgt[0..505]->module

And now no module vmalloc space can be used without using the corrupt
level2_fixmap_pgt entries.

Fix this by properly converting the level2_fixmap_pgt entries to MFNs,
and setting level1_fixmap_pgt as read-only.

A number of comments were also using the the wrong L3 offset for
level2_kernel_pgt.  These have been corrected.

Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: stable@vger.kernel.org
---
 arch/x86/include/asm/pgtable_64.h |  1 +
 arch/x86/xen/mmu.c                | 27 ++++++++++++---------------
 2 files changed, 13 insertions(+), 15 deletions(-)

diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
index 5be9063..3874693 100644
--- a/arch/x86/include/asm/pgtable_64.h
+++ b/arch/x86/include/asm/pgtable_64.h
@@ -19,6 +19,7 @@ extern pud_t level3_ident_pgt[512];
 extern pmd_t level2_kernel_pgt[512];
 extern pmd_t level2_fixmap_pgt[512];
 extern pmd_t level2_ident_pgt[512];
+extern pte_t level1_fixmap_pgt[512];
 extern pgd_t init_level4_pgt[];
 
 #define swapper_pg_dir init_level4_pgt
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index e8a1201..16fb009 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1858,11 +1858,10 @@
  *
  * We can construct this by grafting the Xen provided pagetable into
  * head_64.S's preconstructed pagetables.  We copy the Xen L2's into
- * level2_ident_pgt, level2_kernel_pgt and level2_fixmap_pgt.  This
- * means that only the kernel has a physical mapping to start with -
- * but that's enough to get __va working.  We need to fill in the rest
- * of the physical mapping once some sort of allocator has been set
- * up.
+ * level2_ident_pgt, and level2_kernel_pgt.  This means that only the
+ * kernel has a physical mapping to start with - but that's enough to
+ * get __va working.  We need to fill in the rest of the physical
+ * mapping once some sort of allocator has been set up.
  */
 void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
 {
@@ -1892,9 +1891,12 @@
 	/* L3_i[0] -> level2_ident_pgt */
 	convert_pfn_mfn(level3_ident_pgt);
 	/* L3_k[510] -> level2_kernel_pgt
-	 * L3_i[511] -> level2_fixmap_pgt */
+	 * L3_k[511] -> level2_fixmap_pgt */
 	convert_pfn_mfn(level3_kernel_pgt);
 
+	/* L3_k[511][506] -> level1_fixmap_pgt */
+	convert_pfn_mfn(level2_fixmap_pgt);
+
 	/* We get [511][511] and have Xen's version of level2_kernel_pgt */
 	l3 = m2v(pgd[pgd_index(__START_KERNEL_map)].pgd);
 	l2 = m2v(l3[pud_index(__START_KERNEL_map)].pud);
@@ -1903,22 +1905,15 @@
 	addr[1] = (unsigned long)l3;
 	addr[2] = (unsigned long)l2;
 	/* Graft it onto L4[272][0]. Note that we creating an aliasing problem:
-	 * Both L4[272][0] and L4[511][511] have entries that point to the same
+	 * Both L4[272][0] and L4[511][510] have entries that point to the same
 	 * L2 (PMD) tables. Meaning that if you modify it in __va space
 	 * it will be also modified in the __ka space! (But if you just
 	 * modify the PMD table to point to other PTE's or none, then you
 	 * are OK - which is what cleanup_highmap does) */
 	copy_page(level2_ident_pgt, l2);
-	/* Graft it onto L4[511][511] */
+	/* Graft it onto L4[511][510] */
 	copy_page(level2_kernel_pgt, l2);
 
-	/* Get [511][510] and graft that in level2_fixmap_pgt */
-	l3 = m2v(pgd[pgd_index(__START_KERNEL_map + PMD_SIZE)].pgd);
-	l2 = m2v(l3[pud_index(__START_KERNEL_map + PMD_SIZE)].pud);
-	copy_page(level2_fixmap_pgt, l2);
-	/* Note that we don't do anything with level1_fixmap_pgt which
-	 * we don't need. */
-
 	/* Make pagetable pieces RO */
 	set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
 	set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
@@ -1927,6 +1922,7 @@
 	set_page_prot(level2_ident_pgt, PAGE_KERNEL_RO);
 	set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
 	set_page_prot(level2_fixmap_pgt, PAGE_KERNEL_RO);
+	set_page_prot(level1_fixmap_pgt, PAGE_KERNEL_RO);
 
 	/* Pin down new L4 */
 	pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE,
kstange

kstange

2017-09-07 20:21

reporter   ~0030021

With many thanks to hughesjr and toracat, the patch indicated in my previous comment is now in CentOS Plus kernel 3.10.0-693.2.1. I've asked Red Hat to apply it to some future kernel update, but that is only a dream for now.

In the meantime, if anyone who has been experiencing the issue with PV domains can try out the CentOS Plus kernel here and provide feedback, I'd appreciate it!

https://buildlogs.centos.org/c7-plus/kernel-plus/20170907163005/3.10.0-693.2.1.el7.centos.plus.x86_64/
avij

avij

2017-09-14 14:39

manager   ~0030077

For the record, those 3.10.0-693.* CentOS Plus kernels are now available via the regular centosplus repository.

https://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus
http://mirror.centos.org/centos/7.4.1708/centosplus/x86_64/Packages/
bill_mcgonigle

bill_mcgonigle

2017-09-16 15:38

reporter   ~0030095

Kevin, thanks for managing this bug for us - saved my bacon. I ran into it when rebooting a database server and have proactively switched to kernel-plus on a dozen other VM's, both in-house and cloud-hosted (on XenPV). They have been fully updated and rebooted successfully. I've started a blog entry/howto on this issue, which is really major IMO <https://www.bfccomputing.com/2017/09/15/centos-plus-kernel-for-xen.html> - if you can add me to the RHBZ, I'll verify that my claims are accurate. RHBZ id is bill-bugzilla.redhat.com@bfccomputing.com .
PryMar56

PryMar56

2017-09-22 03:21

reporter   ~0030169

In order to launch any Xen pv install we need another pxeboot pair, vmlinuz/initrd.img, that have the desired patch.
The tool of choice to make these is `lorax`, or pylorax.

It turns out that kernel-plus can be used in lorax by changing the first kernel -> kernel-plus in:
/usr/share/lorax/runtime-install.tmpl

The resulting boot.iso, pxeboot/{vmlinuz,initrd.img} can be used launch a Xen pv netinstall. I've done it.

run_lorax (280 bytes)
#sudo -i
#setenforce 0
DEFAULTKERNEL=kernel-plus lorax -p Centos-Minimal -v 7 -r 7.4 \
-s http://mirrors.us.kernel.org/centos/7/os/x86_64/ \
-s http://mirrors.us.kernel.org/centos/7/updates/x86_64/ \
-s http://mirrors.us.kernel.org/centos/7/centosplus/x86_64/ \
./results/
exit 0
run_lorax (280 bytes)
afletdinov

afletdinov

2017-09-25 02:15

reporter   ~0030194

confirmed, for 693.2.2 kernel:
[ 1.750052] ------------[ cut here ]------------
[ 1.750067] WARNING: CPU: 0 PID: 207 at mm/vmalloc.c:131 vmap_page_range_noflush+0x2c1/0x350
[ 1.750072] Modules linked in:
[ 1.750077] CPU: 0 PID: 207 Comm: systemd-udevd Not tainted 3.10.0-693.2.2.el7.x86_64 #1
[ 1.750082] 0000000000000000 0000000045c2d9dc ffff880003153bd8 ffffffff816a3db1
[ 1.750088] ffff880003153c18 ffffffff810879c8 00000083811c14e8 ffff8800036b5000
[ 1.750094] 0000000000000001 ffff8800030f1e80 ffffffffc0000000 0000000000000000
[ 1.750099] Call Trace:
[ 1.750108] [<ffffffff816a3db1>] dump_stack+0x19/0x1b
[ 1.750115] [<ffffffff810879c8>] __warn+0xd8/0x100
[ 1.750119] [<ffffffff81087b0d>] warn_slowpath_null+0x1d/0x20
[ 1.750123] [<ffffffff811c0781>] vmap_page_range_noflush+0x2c1/0x350
[ 1.750127] [<ffffffff811c083e>] map_vm_area+0x2e/0x40
[ 1.750131] [<ffffffff811c1590>] __vmalloc_node_range+0x170/0x270
[ 1.750137] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.750141] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.750147] [<ffffffff8105f143>] module_alloc+0x73/0xd0
[ 1.750150] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.750154] [<ffffffff810fe754>] module_alloc_update_bounds+0x14/0x70
[ 1.750158] [<ffffffff810ff2d2>] load_module+0xb02/0x29e0
[ 1.750161] [<ffffffff811c0717>] ? vmap_page_range_noflush+0x257/0x350
[ 1.750165] [<ffffffff811c083e>] ? map_vm_area+0x2e/0x40
[ 1.750168] [<ffffffff811c1590>] ? __vmalloc_node_range+0x170/0x270
[ 1.750172] [<ffffffff81101249>] ? SyS_init_module+0x99/0x110
[ 1.750175] [<ffffffff81101275>] SyS_init_module+0xc5/0x110
[ 1.750181] [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
[ 1.750185] ---[ end trace 5fc1bf96e70088cb ]---
[ 1.750188] vmalloc: allocation failure, allocated 24576 of 28672 bytes
[ 1.750192] systemd-udevd: page allocation failure: order:0, mode:0xd2
[ 1.750195] CPU: 0 PID: 207 Comm: systemd-udevd Tainted: G W ------------ 3.10.0-693.2.2.el7.x86_64 #1
[ 1.750200] 00000000000000d2 0000000045c2d9dc ffff880003153c48 ffffffff816a3db1
[ 1.750205] ffff880003153cd8 ffffffff81188810 ffffffff8190ea38 ffff880003153c68
[ 1.750211] ffffffff00000018 ffff880003153ce8 ffff880003153c88 0000000045c2d9dc
[ 1.750216] Call Trace:
[ 1.750219] [<ffffffff816a3db1>] dump_stack+0x19/0x1b
[ 1.750227] [<ffffffff81188810>] warn_alloc_failed+0x110/0x180
[ 1.750231] [<ffffffff811c1654>] __vmalloc_node_range+0x234/0x270
[ 1.750236] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.750239] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.750243] [<ffffffff8105f143>] module_alloc+0x73/0xd0
[ 1.750246] [<ffffffff810fe754>] ? module_alloc_update_bounds+0x14/0x70
[ 1.750250] [<ffffffff810fe754>] module_alloc_update_bounds+0x14/0x70
[ 1.750253] [<ffffffff810ff2d2>] load_module+0xb02/0x29e0
[ 1.750256] [<ffffffff811c0717>] ? vmap_page_range_noflush+0x257/0x350
[ 1.750260] [<ffffffff811c083e>] ? map_vm_area+0x2e/0x40
[ 1.750263] [<ffffffff811c1590>] ? __vmalloc_node_range+0x170/0x270
[ 1.750267] [<ffffffff81101249>] ? SyS_init_module+0x99/0x110
[ 1.750270] [<ffffffff81101275>] SyS_init_module+0xc5/0x110
[ 1.750274] [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
[ 1.750278] Mem-Info:
[ 1.750282] active_anon:908 inactive_anon:1583 isolated_anon:0
[ 1.750282] active_file:1848 inactive_file:7699 isolated_file:0
[ 1.750282] unevictable:0 dirty:0 writeback:0 unstable:0
[ 1.750282] slab_reclaimable:1677 slab_unreclaimable:1124
[ 1.750282] mapped:874 shmem:1586 pagetables:313 bounce:0
[ 1.750282] free:235835 free_pcp:52 free_cma:0
[ 1.750296] Node 0 DMA free:15536kB min:60kB low:72kB high:88kB active_anon:128kB inactive_anon:168kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15912kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:168kB slab_reclaimable:24kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:32kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 1.750315] lowmem_reserve[]: 0 971 971 971
[ 1.750320] Node 0 DMA32 free:927804kB min:3956kB low:4944kB high:5932kB active_anon:3504kB inactive_anon:6164kB active_file:7392kB inactive_file:30796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032192kB managed:996556kB mlocked:0kB dirty:0kB writeback:0kB mapped:3496kB shmem:6176kB slab_reclaimable:6684kB slab_unreclaimable:4480kB kernel_stack:720kB pagetables:1220kB unstable:0kB bounce:0kB free_pcp:208kB local_pcp:208kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 1.750337] lowmem_reserve[]: 0 0 0 0
[ 1.750341] Node 0 DMA: 3*4kB (UM) 4*8kB (UEM) 0*16kB 2*32kB (UE) 3*64kB (UEM) 3*128kB (UEM) 2*256kB (UE) 2*512kB (EM) 3*1024kB (UEM) 1*2048kB (E) 2*4096kB (M) = 15532kB
[ 1.750358] Node 0 DMA32: 1*4kB (E) 1*8kB (U) 1*16kB (E) 3*32kB (UEM) 3*64kB (UEM) 2*128kB (UM) 2*256kB (UE) 2*512kB (UM) 4*1024kB (EM) 2*2048kB (M) 224*4096kB (M) = 927804kB
[ 1.750375] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1.750379] 11133 total pagecache pages
[ 1.750382] 0 pages in swap cache
[ 1.750385] Swap cache stats: add 0, delete 0, find 0/0
[ 1.750387] Free swap = 0kB
[ 1.750390] Total swap = 0kB
[ 1.750392] 262047 pages RAM
[ 1.750394] 0 pages HighMem/MovableOnly
[ 1.750396] 8930 pages reserved
kstange

kstange

2017-10-02 19:04

reporter   ~0030281

I have some updated information from the Red Hat bug!

Red Hat has committed a version of the patch and produced it in test build 3.10.0-720.el7, which puts the commit ON_QA now. This signals that the fix most likely will not appear until RHEL 7.5 unless there is an additional push to put the patch into the 693/EL7.4 kernel series.

I'm trying to get a copy of the 720 kernel to verify that the fix resolves the issue and will report back.
Gene

Gene

2017-10-05 18:35

reporter   ~0030314

I got hit with this on the 3.10.0-693 kernel. Switched back to the previous kernel (3.10.0-514) to get the system to boot.

There was a ten month wait between the 7.3 and 7.4 releases, and 7.4 just came out last month. Waiting ~9 months for a fix is unreasonable. Can't the CentOS project put in a fix for this in the meantime?
kstange

kstange

2017-10-05 18:37

reporter   ~0030315

There is a fix. Switch to using the kernel-plus package from the centos repo. CentOS policy doesn't allow modifying the main kernel package to deviate from the Red Hat sources.
kjohnson

kjohnson

2017-10-09 02:01

reporter   ~0030333

I tried the kernel-plus and had this kernel panic on first boot. I rebooted and the second boot was fine. Any idea?

3.10.0-693.2.2.el7.centos.plus.x86_64 in PV mode:

[ 13.372417] ------------[ cut here ]------------
[ 13.372434] kernel BUG at arch/x86/kernel/paravirt.c:252!
[ 13.372441] invalid opcode: 0000 [#1] SMP
[ 13.372450] Modules linked in: xt_owner nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack iptable_mangle nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter vfat fat isofs xfs libcrc32c loop sb_edac edac_core coretemp intel_rapl iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ip_tables ext4 mbcache jbd2 xen_blkfront xen_netfront crct10dif_pclmul crct10dif_common crc32c_intel
[ 13.372545] CPU: 0 PID: 1138 Comm: mysqld Not tainted 3.10.0-693.2.2.el7.centos.plus.x86_64 #1
[ 13.372555] task: ffff8800fb1a8fd0 ti: ffff8801e9e20000 task.ti: ffff8801e9e20000
[ 13.372561] RIP: e030:[<ffffffff816ad7fe>] [<ffffffff816ad7fe>] enter_lazy.part.0+0x4/0x6
[ 13.372579] RSP: e02b:ffff8801fea03a80 EFLAGS: 00010002
[ 13.372584] RAX: 0000000000000001 RBX: ffff88017d05b280 RCX: ffffffff810215a0
[ 13.372593] RDX: ffff880000000280 RSI: 00003ffffffff000 RDI: 0000000000000001
[ 13.372599] RBP: ffff8801fea03a80 R08: ffff8801fea03b50 R09: ffffffff813f6559
[ 13.372605] R10: ffff88017fc01d00 R11: ffffea00000c0380 R12: ffffc90000c52000
[ 13.372616] R13: ffffc90000c50000 R14: 0000000000000000 R15: ffff8801e3770e00
[ 13.372632] FS: 00007febcc67a900(0000) GS:ffff8801fea00000(0000) knlGS:ffff8801fea00000
[ 13.372644] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 13.372654] CR2: 000056038881bd60 CR3: 00000000f996d000 CR4: 0000000000042660
[ 13.372663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 13.372672] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 13.372680] Stack:
[ 13.372685] ffff8801fea03a90 ffffffff81063b05 ffff8801fea03b38 ffffffff811b2a80
[ 13.372701] ffffc90000c51fff ffffc90000c52000 ffffffff81a0ac90 ffffc90000c51fff
[ 13.372718] ffffc90000c52000 ffff88017fc50000 ffffffff810215a0 00000000fead6d00
[ 13.372733] Call Trace:
[ 13.372740] <IRQ>
[ 13.372744] [<ffffffff81063b05>] paravirt_enter_lazy_mmu+0x25/0x30
[ 13.372771] [<ffffffff811b2a80>] apply_to_page_range+0x260/0x430
[ 13.372784] [<ffffffff810215a0>] ? map_pte_fn+0x60/0x60
[ 13.372794] [<ffffffff810216eb>] arch_gnttab_map_status+0x3b/0x70
[ 13.372809] [<ffffffff813f65d6>] gnttab_map_frames_v2+0xd6/0x150
[ 13.372820] [<ffffffff813f66f1>] gnttab_map+0xa1/0x140
[ 13.372831] [<ffffffff813f6890>] get_free_entries+0x100/0x2e0
kjohnson

kjohnson

2017-10-09 02:04

reporter   ~0030334

For my note above, it seems to be this bug:

https://lists.centos.org/pipermail/centos-virt/2017-May/005540.html
adrian27

adrian27

2018-01-13 14:29

reporter   ~0030925

I had the same issue booting a CentOS 7 PV instance-store AMI in AWS EC2. I started with the latest CentOS 7.4 kernel 3.10.0-693.11.6.el7.x86_64, and after a few days of troubleshooting I managed to boot the instance using older 3.10.0-514* kernels from CentOS 7.3.

Then I found this issue reported here, so I start using CentOS Plus kernels. The CentOS 7 PV instance-store AMI boots up properly with CentOS Plus kernel versions kernel-plus-3.10.0-693.2.1 to 693.11.1, however the latest CentOS Plus kernel kernel-plus-3.10.0-693.11.6 stops working, and the boot is interrupted after the following message, then the VM reboots:

(...)
[ 3.570512] usbcore: registered new interface driver usbhid
[ 3.581218] usbhid: USB HID core driver
[ 3.588925] drop_monitor: Initializing network drop monitor service
[ 3.601194] TCP: cubic registered
[ 3.608001] Initializing XFRM netlink socket
[ 3.616800] NET: Registered protocol family 10
[ 3.625903] NET: Registered protocol family 17
[ 3.634501] mce: Unable to init device /dev/mcelog (rc: -5)
[ 3.645168] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712
[ 3.656568] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[ 3.674563] Loading compiled-in X.509 certificates
[ 3.683010] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'
[ 3.700538] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'
[ 3.720636] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'
[ 3.738829] registered taskstats version 1

The full kernel boot log as retrieved from AWS EC2 is attached.

aws-kernel-plus-3.10.0-693.11.6.el7.centos.plus.x86_64.log.txt (65,581 bytes)
i-0071d198fc56b75db	e: registered new interface driver hub

[    0.128069] usbcore: registered new device driver usb

[    0.129126] dmi: Firmware registration failed.

[    0.130037] PCI: System does not support PCI

[    0.131009] PCI: System does not support PCI

[    0.132169] NetLabel: Initializing

[    0.133024] NetLabel:  domain hash size = 128

[    0.134017] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.135055] NetLabel:  unlabeled traffic allowed by default

[    0.136050] amd_nb: Cannot enumerate AMD northbridges

[    0.137037] Switched to clocksource xen

[    0.150363] pnp: PnP ACPI: disabled

[    0.158911] NET: Registered protocol family 2

[    0.166778] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.178769] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.189316] TCP: Hash tables configured (established 16384 bind 16384)

[    0.199539] TCP: reno registered

[    0.204854] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.214132] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.224257] NET: Registered protocol family 1

[    0.231702] Unpacking initramfs...

[    0.322421] random: fast init done

[    0.795814] Freeing initrd memory: 19404k freed

[    0.836023] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.849038] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.858265] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.868441] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.877679] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.889792] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.899877] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.910971] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.921383] Initialise system trusted keyring

[    0.929467] audit: initializing netlink socket (disabled)

[    0.938389] type=2000 audit(1515851068.643:1): initialized

[    0.981758] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    0.994775] zpool: loaded

[    0.999321] zbud: loaded

[    1.003769] VFS: Disk quotas dquot_6.5.2

[    1.010182] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.020523] msgmni has been set to 3031

[    1.027239] Key type big_key registered

[    1.034631] NET: Registered protocol family 38

[    1.041808] Key type asymmetric registered

[    1.048082] Asymmetric key parser 'x509' registered

[    1.055638] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.067494] io scheduler noop registered

[    1.073972] io scheduler deadline registered (default)

[    1.082193] io scheduler cfq registered

[    1.088598] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.097006] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.108019] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.118109] Non-volatile memory driver v1.3

[    1.124671] Linux agpgart interface v0.103

[    1.131759] crash memory driver: version 1.1

[    1.138607] rdac: device handler registered

[    1.145365] hp_sw: device handler registered

[    1.152263] emc: device handler registered

[    1.158974] alua: device handler registered

[    1.166059] libphy: Fixed MDIO Bus: probed

[    1.173435] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.184515] ehci-pci: EHCI PCI platform driver

[    1.191747] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.201627] ohci-pci: OHCI PCI platform driver

[    1.209097] uhci_hcd: USB Universal Host Controller Interface driver

[    1.219336] usbcore: registered new interface driver usbserial

[    1.228610] usbcore: registered new interface driver usbserial_generic

[    1.239959] usbserial: USB Serial support registered for generic

[    1.249822] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.268318] i8042: No controller found

[    3.274232] mousedev: PS/2 mouse device common for all mice

[    3.391486] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.426704] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.436801] dmi-sysfs: dmi entry is absent.

[    3.444127] hidraw: raw HID events driver (C) Jiri Kosina

[    3.452806] usbcore: registered new interface driver usbhid

[    3.461781] usbhid: USB HID core driver

[    3.469664] drop_monitor: Initializing network drop monitor service

[    3.481349] TCP: cubic registered

[    3.488241] Initializing XFRM netlink socket

[    3.497567] NET: Registered protocol family 10

[    3.507254] NET: Registered protocol family 17

[    3.515980] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.528628] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.539843] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.556197] Loading compiled-in X.509 certificates

[    3.565330] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.583448] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.603163] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.622367] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21479/2049

Failed to read /local/domain/0/backend/vbd/21479/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21479/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21479/2050

Failed to read /local/domain/0/backend/vbd/21479/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21479/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21479/2051

Failed to read /local/domain/0/backend/vbd/21479/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21479/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21479/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21479/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21479/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.712 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795672)

[    0.004019] pid_max: default: 32768 minimum: 301

[    0.005080] Security Framework initialized

[    0.006027] SELinux:  Initializing.

[    0.007047] Yama: becoming mindful.

[    0.008631] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.009826] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.010285] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011046] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.012401] Initializing cgroup subsys memory

[    0.013032] Initializing cgroup subsys devices

[    0.014015] Initializing cgroup subsys freezer

[    0.015022] Initializing cgroup subsys net_cls

[    0.016015] Initializing cgroup subsys blkio

[    0.017021] Initializing cgroup subsys perf_event

[    0.018022] Initializing cgroup subsys hugetlb

[    0.019015] Initializing cgroup subsys pids

[    0.020022] Initializing cgroup subsys net_prio

[    0.021131] FEATURE SPEC_CTRL Not Present

[    0.022017] FEATURE IBPB_SUPPORT Not Present

[    0.023011] CPU: Physical Processor ID: 0

[    0.024017] CPU: Processor Core ID: 0

[    0.025025] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.026010] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.027018] tlb_flushall_shift: 6

[    0.073170] Freeing SMP alternatives: 24k freed

[    0.080600] ftrace: allocating 26850 entries in 105 pages

[    0.087145] cpu 0 spinlock event irq 17

[    0.088017] smpboot: Max logical packages: 1

[    0.089072] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.093354] Brought up 1 CPUs

[    0.094159] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.095009] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.096161] devtmpfs: initialized

[    0.099151] EVM: security.selinux

[    0.100017] EVM: security.ima

[    0.101017] EVM: security.capability

[    0.103000] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.103032] pinctrl core: initialized pinctrl subsystem

[    0.104072] xen:grant_table: Grant tables using version 1 layout

[    0.105064] Grant table initialized

[    0.119000] RTC time: 165:165:165, date: 165/165/65

[    0.119167] NET: Registered protocol family 16

[    0.121019] PCI: setting up Xen PCI frontend stub

[    0.123321] ACPI: Interpreter disabled.

[    0.124054] xen:balloon: Initialising balloon driver

[    0.127085] xen_balloon: Initialising balloon driver

[    0.128239] vgaarb: loaded

[    0.129123] SCSI subsystem initialized

[    0.130082] usbcore: registered new interface driver usbfs

[    0.131052] usbcore: registered new interface driver hub

[    0.132077] usbcore: registered new device driver usb

[    0.133130] dmi: Firmware registration failed.

[    0.134043] PCI: System does not support PCI

[    0.135009] PCI: System does not support PCI

[    0.136183] NetLabel: Initializing

[    0.137018] NetLabel:  domain hash size = 128

[    0.138008] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.139043] NetLabel:  unlabeled traffic allowed by default

[    0.140058] amd_nb: Cannot enumerate AMD northbridges

[    0.141033] Switched to clocksource xen

[    0.155171] pnp: PnP ACPI: disabled

[    0.164175] NET: Registered protocol family 2

[    0.173230] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.187132] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.199634] TCP: Hash tables configured (established 16384 bind 16384)

[    0.211944] TCP: reno registered

[    0.218365] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.230016] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.242567] NET: Registered protocol family 1

[    0.251336] Unpacking initramfs...

[    0.331396] random: fast init done

[    0.811863] Freeing initrd memory: 19404k freed

[    0.828857] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.844194] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.855245] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.866217] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.876589] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.891732] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.903982] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.917256] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.929113] Initialise system trusted keyring

[    0.937869] audit: initializing netlink socket (disabled)

[    0.948101] type=2000 audit(1515851106.545:1): initialized

[    0.992975] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    1.006345] zpool: loaded

[    1.011753] zbud: loaded

[    1.016974] VFS: Disk quotas dquot_6.5.2

[    1.024513] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.037019] msgmni has been set to 3031

[    1.044514] Key type big_key registered

[    1.053089] NET: Registered protocol family 38

[    1.061772] Key type asymmetric registered

[    1.069658] Asymmetric key parser 'x509' registered

[    1.079075] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.093266] io scheduler noop registered

[    1.100910] io scheduler deadline registered (default)

[    1.110763] io scheduler cfq registered

[    1.118346] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.128918] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.141975] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.154649] Non-volatile memory driver v1.3

[    1.162813] Linux agpgart interface v0.103

[    1.170746] crash memory driver: version 1.1

[    1.179257] rdac: device handler registered

[    1.187305] hp_sw: device handler registered

[    1.195603] emc: device handler registered

[    1.203646] alua: device handler registered

[    1.211784] libphy: Fixed MDIO Bus: probed

[    1.219453] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.231762] ehci-pci: EHCI PCI platform driver

[    1.240466] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.252368] ohci-pci: OHCI PCI platform driver

[    1.261066] uhci_hcd: USB Universal Host Controller Interface driver

[    1.273539] usbcore: registered new interface driver usbserial

[    1.285321] usbcore: registered new interface driver usbserial_generic

[    1.297925] usbserial: USB Serial support registered for generic

[    1.309542] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.339419] i8042: No controller found

[    3.346952] mousedev: PS/2 mouse device common for all mice

[    3.465359] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.478094] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.489522] dmi-sysfs: dmi entry is absent.

[    3.497741] hidraw: raw HID events driver (C) Jiri Kosina

[    3.508257] usbcore: registered new interface driver usbhid

[    3.519108] usbhid: USB HID core driver

[    3.526549] drop_monitor: Initializing network drop monitor service

[    3.538685] TCP: cubic registered

[    3.545311] Initializing XFRM netlink socket

[    3.553795] NET: Registered protocol family 10

[    3.562884] NET: Registered protocol family 17

[    3.571651] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.582560] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.593957] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.611079] Loading compiled-in X.509 certificates

[    3.620654] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.638693] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.659264] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.677322] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21480/2049

Failed to read /local/domain/0/backend/vbd/21480/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21480/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21480/2050

Failed to read /local/domain/0/backend/vbd/21480/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21480/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21480/2051

Failed to read /local/domain/0/backend/vbd/21480/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21480/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21480/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21480/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21480/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.819 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795673)

[    0.004009] pid_max: default: 32768 minimum: 301

[    0.005088] Security Framework initialized

[    0.006027] SELinux:  Initializing.

[    0.007046] Yama: becoming mindful.

[    0.008643] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.009752] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.010278] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011040] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.012392] Initializing cgroup subsys memory

[    0.013038] Initializing cgroup subsys devices

[    0.014016] Initializing cgroup subsys freezer

[    0.015021] Initializing cgroup subsys net_cls

[    0.016013] Initializing cgroup subsys blkio

[    0.017017] Initializing cgroup subsys perf_event

[    0.018017] Initializing cgroup subsys hugetlb

[    0.019022] Initializing cgroup subsys pids

[    0.020014] Initializing cgroup subsys net_prio

[    0.021133] FEATURE SPEC_CTRL Not Present

[    0.022009] FEATURE IBPB_SUPPORT Not Present

[    0.023019] CPU: Physical Processor ID: 0

[    0.024008] CPU: Processor Core ID: 0

[    0.025028] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.026017] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.027010] tlb_flushall_shift: 6

[    0.067936] Freeing SMP alternatives: 24k freed

[    0.074423] ftrace: allocating 26850 entries in 105 pages

[    0.080150] cpu 0 spinlock event irq 17

[    0.081019] smpboot: Max logical packages: 1

[    0.082072] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.086512] Brought up 1 CPUs

[    0.087178] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.088019] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.089209] devtmpfs: initialized

[    0.092000] EVM: security.selinux

[    0.092015] EVM: security.ima

[    0.093017] EVM: security.capability

[    0.095000] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.095034] pinctrl core: initialized pinctrl subsystem

[    0.096067] xen:grant_table: Grant tables using version 1 layout

[    0.097070] Grant table initialized

[    0.109710] RTC time: 165:165:165, date: 165/165/65

[    0.110157] NET: Registered protocol family 16

[    0.111899] PCI: setting up Xen PCI frontend stub

[    0.113000] ACPI: Interpreter disabled.

[    0.113064] xen:balloon: Initialising balloon driver

[    0.116035] xen_balloon: Initialising balloon driver

[    0.117229] vgaarb: loaded

[    0.118112] SCSI subsystem initialized

[    0.119099] usbcore: registered new interface driver usbfs

[    0.120030] usbcore: registered new interface driver hub

[    0.121068] usbcore: registered new device driver usb

[    0.122104] dmi: Firmware registration failed.

[    0.123037] PCI: System does not support PCI

[    0.124012] PCI: System does not support PCI

[    0.125154] NetLabel: Initializing

[    0.126007] NetLabel:  domain hash size = 128

[    0.127007] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.128045] NetLabel:  unlabeled traffic allowed by default

[    0.129044] amd_nb: Cannot enumerate AMD northbridges

[    0.130032] Switched to clocksource xen

[    0.142672] pnp: PnP ACPI: disabled

[    0.149900] NET: Registered protocol family 2

[    0.159748] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.171361] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.181446] TCP: Hash tables configured (established 16384 bind 16384)

[    0.191429] TCP: reno registered

[    0.196457] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.205809] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.215831] NET: Registered protocol family 1

[    0.222682] Unpacking initramfs...

[    0.313537] random: fast init done

[    0.751449] Freeing initrd memory: 19404k freed

[    0.790937] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.803321] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.812277] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.821075] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.829506] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.841398] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.850936] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.861264] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.870747] Initialise system trusted keyring

[    0.877954] audit: initializing netlink socket (disabled)

[    0.886444] type=2000 audit(1515851144.172:1): initialized

[    0.928735] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    0.940142] zpool: loaded

[    0.944447] zbud: loaded

[    0.948811] VFS: Disk quotas dquot_6.5.2

[    0.955179] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    0.965481] msgmni has been set to 3031

[    0.971734] Key type big_key registered

[    0.978980] NET: Registered protocol family 38

[    0.985925] Key type asymmetric registered

[    0.992181] Asymmetric key parser 'x509' registered

[    0.999734] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.011351] io scheduler noop registered

[    1.017455] io scheduler deadline registered (default)

[    1.025385] io scheduler cfq registered

[    1.031529] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.040211] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.050422] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.060390] Non-volatile memory driver v1.3

[    1.066928] Linux agpgart interface v0.103

[    1.073336] crash memory driver: version 1.1

[    1.080100] rdac: device handler registered

[    1.086457] hp_sw: device handler registered

[    1.093264] emc: device handler registered

[    1.099722] alua: device handler registered

[    1.106137] libphy: Fixed MDIO Bus: probed

[    1.112447] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.122401] ehci-pci: EHCI PCI platform driver

[    1.129752] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.139547] ohci-pci: OHCI PCI platform driver

[    1.146220] uhci_hcd: USB Universal Host Controller Interface driver

[    1.156146] usbcore: registered new interface driver usbserial

[    1.165566] usbcore: registered new interface driver usbserial_generic

[    1.175533] usbserial: USB Serial support registered for generic

[    1.185025] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.175642] i8042: No controller found

[    3.181966] mousedev: PS/2 mouse device common for all mice

[    3.298903] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.332947] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.341847] dmi-sysfs: dmi entry is absent.

[    3.348309] hidraw: raw HID events driver (C) Jiri Kosina

[    3.356475] usbcore: registered new interface driver usbhid

[    3.365203] usbhid: USB HID core driver

[    3.371135] drop_monitor: Initializing network drop monitor service

[    3.380740] TCP: cubic registered

[    3.385987] Initializing XFRM netlink socket

[    3.392872] NET: Registered protocol family 10

[    3.400357] NET: Registered protocol family 17

[    3.407609] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.416494] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.426053] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.439861] Loading compiled-in X.509 certificates

[    3.447617] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.462289] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.478885] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.493902] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21481/2049

Failed to read /local/domain/0/backend/vbd/21481/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21481/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21481/2050

Failed to read /local/domain/0/backend/vbd/21481/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21481/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21481/2051

Failed to read /local/domain/0/backend/vbd/21481/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21481/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21481/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21481/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21481/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.719 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795673)

[    0.004017] pid_max: default: 32768 minimum: 301

[    0.005098] Security Framework initialized

[    0.006026] SELinux:  Initializing.

[    0.007050] Yama: becoming mindful.

[    0.008686] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.009828] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.010305] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011046] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.012439] Initializing cgroup subsys memory

[    0.013040] Initializing cgroup subsys devices

[    0.014023] Initializing cgroup subsys freezer

[    0.015023] Initializing cgroup subsys net_cls

[    0.016027] Initializing cgroup subsys blkio

[    0.017028] Initializing cgroup subsys perf_event

[    0.018030] Initializing cgroup subsys hugetlb

[    0.019020] Initializing cgroup subsys pids

[    0.020020] Initializing cgroup subsys net_prio

[    0.021153] FEATURE SPEC_CTRL Not Present

[    0.022021] FEATURE IBPB_SUPPORT Not Present

[    0.023014] CPU: Physical Processor ID: 0

[    0.024016] CPU: Processor Core ID: 0

[    0.025017] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.026024] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.027014] tlb_flushall_shift: 6

[    0.097031] Freeing SMP alternatives: 24k freed

[    0.108049] ftrace: allocating 26850 entries in 105 pages

[    0.118148] cpu 0 spinlock event irq 17

[    0.119024] smpboot: Max logical packages: 1

[    0.120081] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.124976] Brought up 1 CPUs

[    0.125181] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.126021] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.127247] devtmpfs: initialized

[    0.131000] EVM: security.selinux

[    0.132016] EVM: security.ima

[    0.133019] EVM: security.capability

[    0.135425] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.136038] pinctrl core: initialized pinctrl subsystem

[    0.137076] xen:grant_table: Grant tables using version 1 layout

[    0.138097] Grant table initialized

[    0.158860] RTC time: 165:165:165, date: 165/165/65

[    0.159193] NET: Registered protocol family 16

[    0.161024] PCI: setting up Xen PCI frontend stub

[    0.163416] ACPI: Interpreter disabled.

[    0.164094] xen:balloon: Initialising balloon driver

[    0.168063] xen_balloon: Initialising balloon driver

[    0.169239] vgaarb: loaded

[    0.170157] SCSI subsystem initialized

[    0.171131] usbcore: registered new interface driver usbfs

[    0.172071] usbcore: registered new interface driver hub

[    0.173126] usbcore: registered new device driver usb

[    0.174216] dmi: Firmware registration failed.

[    0.175063] PCI: System does not support PCI

[    0.176016] PCI: System does not support PCI

[    0.177325] NetLabel: Initializing

[    0.178020] NetLabel:  domain hash size = 128

[    0.179025] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.180073] NetLabel:  unlabeled traffic allowed by default

[    0.181114] amd_nb: Cannot enumerate AMD northbridges

[    0.182087] Switched to clocksource xen

[    0.199245] pnp: PnP ACPI: disabled

[    0.208597] NET: Registered protocol family 2

[    0.219064] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.234639] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.248032] TCP: Hash tables configured (established 16384 bind 16384)

[    0.261014] TCP: reno registered

[    0.267848] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.279841] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.292765] NET: Registered protocol family 1

[    0.301892] Unpacking initramfs...

[    0.315045] random: fast init done

[    0.864749] Freeing initrd memory: 19404k freed

[    0.882503] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.898833] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.910245] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.921758] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.932642] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.948836] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.961361] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.975966] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.988407] Initialise system trusted keyring

[    0.997143] audit: initializing netlink socket (disabled)

[    1.007983] type=2000 audit(1515851183.567:1): initialized

[    1.054484] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    1.068680] zpool: loaded

[    1.074123] zbud: loaded

[    1.079618] VFS: Disk quotas dquot_6.5.2

[    1.087497] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.100464] msgmni has been set to 3031

[    1.108301] Key type big_key registered

[    1.117363] NET: Registered protocol family 38

[    1.126362] Key type asymmetric registered

[    1.134694] Asymmetric key parser 'x509' registered

[    1.144411] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.159339] io scheduler noop registered

[    1.167597] io scheduler deadline registered (default)

[    1.177823] io scheduler cfq registered

[    1.185634] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.196439] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.210179] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.223251] Non-volatile memory driver v1.3

[    1.231632] Linux agpgart interface v0.103

[    1.240300] crash memory driver: version 1.1

[    1.249463] rdac: device handler registered

[    1.257886] hp_sw: device handler registered

[    1.266558] emc: device handler registered

[    1.274755] alua: device handler registered

[    1.283045] libphy: Fixed MDIO Bus: probed

[    1.291250] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.304124] ehci-pci: EHCI PCI platform driver

[    1.312871] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.325432] ohci-pci: OHCI PCI platform driver

[    1.334448] uhci_hcd: USB Universal Host Controller Interface driver

[    1.347205] usbcore: registered new interface driver usbserial

[    1.358851] usbcore: registered new interface driver usbserial_generic

[    1.371622] usbserial: USB Serial support registered for generic

[    1.383781] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.416560] i8042: No controller found

[    3.424758] mousedev: PS/2 mouse device common for all mice

[    3.543888] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.556864] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.568766] dmi-sysfs: dmi entry is absent.

[    3.577438] hidraw: raw HID events driver (C) Jiri Kosina

[    3.588561] usbcore: registered new interface driver usbhid

[    3.599564] usbhid: USB HID core driver

[    3.607238] drop_monitor: Initializing network drop monitor service

[    3.620381] TCP: cubic registered

[    3.627568] Initializing XFRM netlink socket

[    3.636822] NET: Registered protocol family 10

[    3.646443] NET: Registered protocol family 17

[    3.655642] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.666897] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.679130] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.697134] Loading compiled-in X.509 certificates

[    3.707843] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.726809] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.748045] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.767979] registered taskstats version 1

Xen Minimal OS!

  start_info: 0xcba000(VA)

    nr_pages: 0x61d05

  shared_inf: 0x68e37000(MA)

     pt_base: 0xcbd000(VA)

nr_pt_frames: 0xb

    mfn_list: 0x9ab000(VA)

   mod_start: 0x0(VA)

     mod_len: 0

       flags: 0x300

    cmd_line: root=/dev/sda1 ro console=hvc0 4

  stack:      0x96a100-0x98a100

MM: Init

      _text: 0x0(VA)

     _etext: 0x7b824(VA)

   _erodata: 0x97000(VA)

     _edata: 0x9cce0(VA)

stack start: 0x96a100(VA)

       _end: 0x9aa700(VA)

  start_pfn: ccb

    max_pfn: 61d05

Mapping memory range 0x1000000 - 0x61d05000

setting 0x0-0x97000 readonly

skipped 0x1000

MM: Initialise page allocator for fd3000(fd3000)-61d05000(61d05000)

MM: done

Demand map pfns at 61d06000-2061d06000.

Heap resides at 2061d07000-4061d07000.

Initialising timer interface

Initialising console ... done.

gnttab_table mapped at 0x61d06000.

Initialising scheduler

Thread "Idle": pointer: 0x2061d07050, stack: 0x619f0000

Thread "xenstore": pointer: 0x2061d07800, stack: 0x61a00000

xenbus initialised on irq 2 mfn 0xfeffc

Thread "shutdown": pointer: 0x2061d07fb0, stack: 0x61a20000

Dummy main: start_info=0x98a200

Thread "main": pointer: 0x2061d08760, stack: 0x61a30000

"main" "root=/dev/sda1" "ro" "console=hvc0" "4" 

vbd 2049 is hd0

******************* BLKFRONT for device/vbd/2049 **********





backend at /local/domain/0/backend/vbd/21482/2049

Failed to read /local/domain/0/backend/vbd/21482/2049/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21482/2049/feature-flush-cache.

4194304 sectors of 512 bytes

**************************

vbd 2050 is hd1

******************* BLKFRONT for device/vbd/2050 **********





backend at /local/domain/0/backend/vbd/21482/2050

Failed to read /local/domain/0/backend/vbd/21482/2050/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21482/2050/feature-flush-cache.

312705024 sectors of 512 bytes

**************************

vbd 2051 is hd2

******************* BLKFRONT for device/vbd/2051 **********





backend at /local/domain/0/backend/vbd/21482/2051

Failed to read /local/domain/0/backend/vbd/21482/2051/feature-barrier.

Failed to read /local/domain/0/backend/vbd/21482/2051/feature-flush-cache.

1835008 sectors of 512 bytes

**************************

  Booting 'CentOS Linux (3.10.0-693.11.6.el7.centos.plus.x86_64) 7 (Core)'





root (hd0)


 Filesystem type is ext2fs, using whole disk


kernel /boot/vmlinuz-3.10.0-693.11.6.el7.centos.plus.x86_64 ro root=UUID=ade108


6f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0


initrd /boot/initramfs-3.10.0-693.11.6.el7.centos.plus.x86_64.img





============= Init TPM Front ================

Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront initialization! error = ENOENT

Tpmfront:Info Shutting down tpmfront

close blk: backend=/local/domain/0/backend/vbd/21482/2049 node=device/vbd/2049

close blk: backend=/local/domain/0/backend/vbd/21482/2050 node=device/vbd/2050

close blk: backend=/local/domain/0/backend/vbd/21482/2051 node=device/vbd/2051

[    0.000000] Initializing cgroup subsys cpuset

[    0.000000] Initializing cgroup subsys cpu

[    0.000000] Initializing cgroup subsys cpuacct

[    0.000000] Linux version 3.10.0-693.11.6.el7.centos.plus.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 4 14:27:13 UTC 2018

[    0.000000] Command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] ACPI in unprivileged domain disabled

[    0.000000] e820: BIOS-provided physical RAM map:

[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable

[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved

[    0.000000] Xen: [mem 0x0000000000100000-0x0000000062504fff] usable

[    0.000000] NX (Execute Disable) protection: active

[    0.000000] DMI not present or invalid.

[    0.000000] e820: last_pfn = 0x62505 max_arch_pfn = 0x400000000

[    0.000000] RAMDISK: [mem 0x0243e000-0x03730fff]

[    0.000000] NUMA turned off

[    0.000000] Faking a node at [mem 0x0000000000000000-0x0000000062504fff]

[    0.000000] NODE_DATA(0) allocated [mem 0x61cde000-0x61d04fff]

[    0.000000] Zone ranges:

[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]

[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]

[    0.000000]   Normal   empty

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]

[    0.000000]   node   0: [mem 0x00100000-0x62504fff]

[    0.000000] Initmem setup node 0 [mem 0x00001000-0x62504fff]

[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org

[    0.000000] No local APIC present

[    0.000000] APIC: disable apic facility

[    0.000000] APIC: switched to apic NOOP

[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs

[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]

[    0.000000] e820: [mem 0x62505000-0xffffffff] available for PCI devices

[    0.000000] Booting paravirtualized kernel on Xen

[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)

[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1

[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff880061400000 s104600 r8192 d30568 u2097152

[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)

[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 396282

[    0.000000] Policy zone: DMA32

[    0.000000] Kernel command line: ro root=UUID=ade1086f-b766-4089-9f8b-e3b0c28979f3 consoleblank=0 console=hvc0

[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)

[    0.000000] Memory: 1532640k/1610772k available (6984k kernel code, 388k absent, 77744k reserved, 4581k data, 1800k init)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1

[    0.000000] x86/pti: Xen PV detected, disabling PTI protection

[    0.000000] Hierarchical RCU implementation.

[    0.000000] 	RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=1.

[    0.000000] NR_IRQS:327936 nr_irqs:24 0

[    0.000000] Console: colour dummy device 80x25

[    0.000000] console [tty0] enabled

[    0.000000] console [hvc0] enabled

[    0.000000] allocated 6815744 bytes of page_cgroup

[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups

[    0.000000] installing Xen timer for CPU 0

[    0.000000] tsc: Fast TSC calibration using PIT

[    0.000000] tsc: Detected 1795.806 MHz processor

[    0.003000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3591.34 BogoMIPS (lpj=1795672)

[    0.004000] pid_max: default: 32768 minimum: 301

[    0.004097] Security Framework initialized

[    0.005028] SELinux:  Initializing.

[    0.006053] Yama: becoming mindful.

[    0.007650] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)

[    0.008825] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)

[    0.009272] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.010050] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)

[    0.011423] Initializing cgroup subsys memory

[    0.012037] Initializing cgroup subsys devices

[    0.013023] Initializing cgroup subsys freezer

[    0.014031] Initializing cgroup subsys net_cls

[    0.015034] Initializing cgroup subsys blkio

[    0.016020] Initializing cgroup subsys perf_event

[    0.017020] Initializing cgroup subsys hugetlb

[    0.018020] Initializing cgroup subsys pids

[    0.019028] Initializing cgroup subsys net_prio

[    0.020173] FEATURE SPEC_CTRL Not Present

[    0.021022] FEATURE IBPB_SUPPORT Not Present

[    0.022023] CPU: Physical Processor ID: 0

[    0.023019] CPU: Processor Core ID: 0

[    0.024031] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0

[    0.025033] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32

[    0.026016] tlb_flushall_shift: 6

[    0.064237] Freeing SMP alternatives: 24k freed

[    0.070363] ftrace: allocating 26850 entries in 105 pages

[    0.076160] cpu 0 spinlock event irq 17

[    0.077018] smpboot: Max logical packages: 1

[    0.078083] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.

[    0.082401] Brought up 1 CPUs

[    0.083223] NMI watchdog: disabled (cpu0): hardware events not enabled

[    0.084013] NMI watchdog: Shutting down hard lockup detector on all cpus

[    0.085233] devtmpfs: initialized

[    0.088000] EVM: security.selinux

[    0.088026] EVM: security.ima

[    0.089018] EVM: security.capability

[    0.091000] atomic64 test passed for x86-64 platform with CX8 and with SSE

[    0.091040] pinctrl core: initialized pinctrl subsystem

[    0.092123] xen:grant_table: Grant tables using version 1 layout

[    0.093114] Grant table initialized

[    0.105228] RTC time: 165:165:165, date: 165/165/65

[    0.106214] NET: Registered protocol family 16

[    0.107909] PCI: setting up Xen PCI frontend stub

[    0.109000] ACPI: Interpreter disabled.

[    0.109082] xen:balloon: Initialising balloon driver

[    0.112029] xen_balloon: Initialising balloon driver

[    0.113210] vgaarb: loaded

[    0.114140] SCSI subsystem initialized

[    0.115106] usbcore: registered new interface driver usbfs

[    0.116041] usbcore: registered new interface driver hub

[    0.117099] usbcore: registered new device driver usb

[    0.118153] dmi: Firmware registration failed.

[    0.119052] PCI: System does not support PCI

[    0.120016] PCI: System does not support PCI

[    0.121242] NetLabel: Initializing

[    0.122018] NetLabel:  domain hash size = 128

[    0.123009] NetLabel:  protocols = UNLABELED CIPSOv4

[    0.124060] NetLabel:  unlabeled traffic allowed by default

[    0.125092] amd_nb: Cannot enumerate AMD northbridges

[    0.126043] Switched to clocksource xen

[    0.140743] pnp: PnP ACPI: disabled

[    0.149606] NET: Registered protocol family 2

[    0.158530] TCP established hash table entries: 16384 (order: 5, 131072 bytes)

[    0.172639] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)

[    0.185290] TCP: Hash tables configured (established 16384 bind 16384)

[    0.197702] TCP: reno registered

[    0.204071] UDP hash table entries: 1024 (order: 3, 32768 bytes)

[    0.216075] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)

[    0.228633] NET: Registered protocol family 1

[    0.237750] Unpacking initramfs...

[    0.369479] random: fast init done

[    0.812227] Freeing initrd memory: 19404k freed

[    0.829207] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer

[    0.847981] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules

[    0.858970] RAPL PMU: hw unit of domain package 2^-16 Joules

[    0.870339] RAPL PMU: hw unit of domain dram 2^-16 Joules

[    0.881408] platform rtc_cmos: registered platform RTC device (no PNP device found)

[    0.896268] sha1_ssse3: Using SSSE3 optimized SHA-1 implementation

[    0.908284] sha256_ssse3: Using SSSE3 optimized SHA-256 implementation

[    0.921239] futex hash table entries: 256 (order: 2, 16384 bytes)

[    0.932918] Initialise system trusted keyring

[    0.941410] audit: initializing netlink socket (disabled)

[    0.951788] type=2000 audit(1515851217.858:1): initialized

[    1.021248] HugeTLB registered 2 MB page size, pre-allocated 0 pages

[    1.035196] zpool: loaded

[    1.035196] zbud: loaded

[    1.045980] VFS: Disk quotas dquot_6.5.2

[    1.053654] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

[    1.066576] msgmni has been set to 3031

[    1.067244] Key type big_key registered

[    1.082965] NET: Registered protocol family 38

[    1.091844] Key type asymmetric registered

[    1.099826] Asymmetric key parser 'x509' registered

[    1.109567] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)

[    1.123911] io scheduler noop registered

[    1.131564] io scheduler deadline registered (default)

[    1.141809] io scheduler cfq registered

[    1.149393] pci_hotplug: PCI Hot Plug PCI Core version: 0.5

[    1.160247] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

[    1.173417] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled

[    1.186569] Non-volatile memory driver v1.3

[    1.194905] Linux agpgart interface v0.103

[    1.203037] crash memory driver: version 1.1

[    1.211751] rdac: device handler registered

[    1.220166] hp_sw: device handler registered

[    1.229060] emc: device handler registered

[    1.237808] alua: device handler registered

[    1.246499] libphy: Fixed MDIO Bus: probed

[    1.254925] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver

[    1.268099] ehci-pci: EHCI PCI platform driver

[    1.269023] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver

[    1.289595] ohci-pci: OHCI PCI platform driver

[    1.290550] uhci_hcd: USB Universal Host Controller Interface driver

[    1.311474] usbcore: registered new interface driver usbserial

[    1.322858] usbcore: registered new interface driver usbserial_generic

[    1.335999] usbserial: USB Serial support registered for generic

[    1.349056] i8042: PNP: No PS/2 controller found. Probing ports directly.

[    3.400514] i8042: No controller found

[    3.408389] mousedev: PS/2 mouse device common for all mice

[    3.527531] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0

[    3.540338] rtc_cmos: probe of rtc_cmos failed with error -38

[    3.551716] dmi-sysfs: dmi entry is absent.

[    3.559914] hidraw: raw HID events driver (C) Jiri Kosina

[    3.570512] usbcore: registered new interface driver usbhid

[    3.581218] usbhid: USB HID core driver

[    3.588925] drop_monitor: Initializing network drop monitor service

[    3.601194] TCP: cubic registered

[    3.608001] Initializing XFRM netlink socket

[    3.616800] NET: Registered protocol family 10

[    3.625903] NET: Registered protocol family 17

[    3.634501] mce: Unable to init device /dev/mcelog (rc: -5)

[    3.645168] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x712

[    3.656568] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba

[    3.674563] Loading compiled-in X.509 certificates

[    3.683010] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717'

[    3.700538] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b'

[    3.720636] Loaded X.509 cert 'CentOS Linux kernel signing key: 2bade9ee54e382c6692955a95011589b663d6aef'

[    3.738829] registered taskstats version 1

	2018-01-13T13:47:04.000Z
adrian27

adrian27

2018-01-13 14:36

reporter   ~0030926

As a side note, the same issue reported above with the latest CentOS Plus kernel 3.10.0-693.11.6 happens with the latest CentOS 7.4 standard kernel 3.10.0-693.11.6 as well. So this might be another issue introduced by RHEL linux kernels.
toracat

toracat

2018-01-13 14:47

manager   ~0030927

@adrian27

See https://lists.centos.org/pipermail/centos-virt/2018-January/005716.html for a report on the centos-virt mailing list. In that thread, a possible solution is provided by @JohnnyHughes:

https://lists.centos.org/pipermail/centos-virt/2018-January/005721.html
adrian27

adrian27

2018-01-13 14:48

reporter   ~0030928

As another side node, I made a diff between the config files of the CentOS Plus kernels 693.11.1 and 693.11.6, and the only difference is the CONFIG_KAISER=y that appears in the 693.11.6 kernel only ( and which I think causes the boot to hang at message [ 3.738829] registered taskstats version 1)

# diff 11.1/boot/config-3.10.0-693.11.1.el7.centos.plus.x86_64 11.6/boot/config-3.10.0-693.11.6.el7.centos.plus.x86_64
3c3
< # Linux/x86_64 3.10.0-693.11.1.el7.centos.plus.x86_64 Kernel Configuration
---
> # Linux/x86_64 3.10.0-693.11.6.el7.centos.plus.x86_64 Kernel Configuration
5735a5736
> CONFIG_KAISER=y
bill_mcgonigle

bill_mcgonigle

2018-01-13 17:01

reporter   ~0030930

The KAISER bug (seems to be a new problem, also causing a Xen boot failure) is being tracked here:

https://bugs.centos.org/view.php?id=14347
arrfab

arrfab

2018-05-10 14:25

administrator   ~0031745

CentOS 7.5.1804 was released with kernel kernel-3.10.0-862.el7.x86_64 and during QA we tested that node could be started and provisioned as before (so in PV mode)
Marking bug as resolved

Issue History

Date Modified Username Field Change
2017-09-01 20:11 kstange New Issue
2017-09-01 20:12 kstange Note Added: 0029977
2017-09-01 22:15 kstange Note Added: 0029978
2017-09-01 23:09 kstange File Added: kernel-3.10.0-693.1.1-boot.log
2017-09-02 09:49 arrfab Status new => acknowledged
2017-09-02 09:49 arrfab Note Added: 0029979
2017-09-06 19:06 kstange File Added: xen-dont-copy-bogus-duplicate-entries-into-kernel-page-tables.patch
2017-09-06 19:06 kstange Note Added: 0030014
2017-09-07 20:21 kstange Note Added: 0030021
2017-09-14 14:39 avij Note Added: 0030077
2017-09-16 15:38 bill_mcgonigle Note Added: 0030095
2017-09-22 03:21 PryMar56 File Added: run_lorax
2017-09-22 03:21 PryMar56 Note Added: 0030169
2017-09-25 02:15 afletdinov Note Added: 0030194
2017-10-02 19:04 kstange Note Added: 0030281
2017-10-05 18:35 Gene Note Added: 0030314
2017-10-05 18:37 kstange Note Added: 0030315
2017-10-09 02:01 kjohnson Note Added: 0030333
2017-10-09 02:04 kjohnson Note Added: 0030334
2018-01-13 14:29 adrian27 File Added: aws-kernel-plus-3.10.0-693.11.6.el7.centos.plus.x86_64.log.txt
2018-01-13 14:29 adrian27 Note Added: 0030925
2018-01-13 14:36 adrian27 Note Added: 0030926
2018-01-13 14:47 toracat Note Added: 0030927
2018-01-13 14:48 adrian27 Note Added: 0030928
2018-01-13 17:01 bill_mcgonigle Note Added: 0030930
2018-02-06 23:47 toracat Relationship added related to 0014347
2018-05-10 14:25 arrfab Status acknowledged => resolved
2018-05-10 14:25 arrfab Resolution open => fixed
2018-05-10 14:25 arrfab Fixed in Version => 7.5.1804
2018-05-10 14:25 arrfab Note Added: 0031745