View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0014336CentOS-6kernelpublic2018-01-05 16:152018-02-07 15:35
ReporterJeff_S 
PriorityurgentSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
PlatformXen VMOSEL6OS Version6.9
Product Version6.9 
Target VersionFixed in Version 
Summary0014336: EL6 VM fails to boot after updating to kernel-2.6.32-696.18.7.el6.x86_64
DescriptionAfter updating a fairly stock 6.9 VM (on a Xen host) to the latest kernel package, the VM will not boot. It must hang early in the boot process because there are no logs written about the boot attempts, and I'm unable to get on the Rackspace console to see any details about where it's hanging.
Steps To ReproduceUpdate kernel, reboot.
Additional InformationServer boots fine when reverting to the previously installed kernel, kernel-2.6.32-696.13.2.el6.x86_64.
TagsNo tags attached.

Relationships

related to 0014347 resolvedtoracat CentOS-7 kernel-plus-3.10.0-693.11.6.el7.centos.plus under Xen4 runs in endless loop 
related to 0014415 resolvedtoracat CentOS-6 Xen DomU with kernel-2.6.32-696.18.7.el6.i686 booting fail, Domain shut down, reason code 3 0x3 

Activities

themiz

themiz

2018-01-05 16:54

reporter   ~0030857

I can confirm and reproduce the error.
Cannot even get to a console/grub to choose older kernel (so non logs, non error messages, nothing): I had to restore the server from snapshot.

Affected: Centos 6.9 PV hosts under xenserver 7.x with kernel 2.6.32-696.18.7.el6.x86_64
Not affected: Centos 7.4 HVM hosts (kernel 3.10.0-693.11.6.el7.x86_64)
(I haven't tried other combinations yet)
themiz

themiz

2018-01-05 17:11

reporter   ~0030858

The topic is discussed in the forum:

https://www.centos.org/forums/viewtopic.php?f=13&t=65602
dadapea

dadapea

2018-01-06 03:54

reporter   ~0030861

xl dmesg shows:

(XEN) domain_crash_sync called from entry.S: fault at ffff82d080230983 create_bounce_frame+0x12b/0x13a
(XEN) Domain 32 (vcpu#0) crashed on cpu#20:
(XEN) ----[ Xen-4.6.6-8.el6 x86_64 debug=n Not tainted ]----
(XEN) CPU: 20
(XEN) RIP: e033:[<ffffffff812b7872>]
(XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest (d32v0)
(XEN) rax: 0000000000000000 rbx: ffffffffff400000 rcx: 0000000000000004
(XEN) rdx: 0000000000000010 rsi: ffffffffff400000 rdi: ffffffff81a03e68
(XEN) rbp: ffffffff81a03e48 rsp: ffffffff81a03e00 r8: 0000000000000000
(XEN) r9: 0000000000007ff0 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: ffffffff81a03e58 r13: ffffffffff400000 r14: ffffffffff410000
(XEN) r15: ffffffff81a03e68 cr0: 000000008005003b cr4: 00000000000026e0
(XEN) cr3: 000000124628e000 cr2: ffffffffff400000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffffffff81a03e00:
(XEN) 0000000000000004 0000000000000000 0000000000000000 ffffffff812b7872
(XEN) 000000010000e030 0000000000010002 ffffffff81a03e48 000000000000e02b
(XEN) ffffffff812b7869 ffffffff81a03eb8 ffffffff81c80153 0000000000000000
(XEN) 0000000000000000 ffffffff81a03e98 ffffffff81c872a0 0000000000000000
(XEN) 6ece555bc906fdd1 ffffffffffffffff ffffffff81c872a0 0000000000000000
(XEN) ffffffff81a03f80 ffffffffffffffff 0000000000000000 ffffffff81a03f68
(XEN) ffffffff81c45605 0000000000000010 ffffffff81c872a0 0000000000000000
(XEN) 0000000000000000 ffffffffffffffff 0000000000000000 ffffffff81a03f08
(XEN) ffffffff8107e0ee ffffffff81a03f68 ffffffff8154b0cd 0000000000000010
(XEN) ffffffff81a03f78 ffffffff81a03f38 6ece555bc906fdd1 ffffffff81a03f58
(XEN) ffffffff81c872a0 0000000000000000 0000000000000000 ffffffffffffffff
(XEN) 0000000000000000 ffffffff81a03fa8 ffffffff81c3fdda 6ece555bc906fdd1
(XEN) ffffffff81c8a820 000000000204aa24 0000000000000000 0000000000000000
(XEN) 0000000000000000 ffffffff81a03fc8 ffffffff81c3f33a ffffffff81c34640
(XEN) ffffffff884f4000 ffffffff81a03ff8 ffffffff81c4309c 829822031f898975
(XEN) 000206c235200800 0000000000000000 0000000000000000 0000000000000000
(XEN) ffffffff864f1000 ffffffff864f2000 ffffffff864f3000 ffffffff864f4000
(XEN) ffffffff864f5000 ffffffff864f6000 ffffffff864f7000 ffffffff864f8000
(XEN) ffffffff864f9000 ffffffff864fa000 ffffffff864fb000 ffffffff864fc000
(XEN) ffffffff864fd000 ffffffff864fe000 ffffffff864ff000 ffffffff86500000
(XEN) d33v0: unhandled page fault (ec=0000)
(XEN) Pagetable walk from ffffffffff400000:
(XEN) L4[0x1ff] = 0000001246291067 0000000000001a91
(XEN) L3[0x1ff] = 0000001246292067 0000000000001a92
(XEN) L2[0x1fa] = 0000000000000000 ffffffffffffffff
chrisdb

chrisdb

2018-01-06 15:10

reporter   ~0030863

I get a diffent type of crash:

(XEN) mm.c:2554:d238 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn 100abc7 (pfn 1e04)
(XEN) mm.c:986:d238 Attempt to create linear p.t. with write perms
(XEN) d238:v0: unhandled page fault (ec=0000)
(XEN) Pagetable walk from ffffffffff400000:
(XEN) L4[0x1ff] = 000000100af3a067 0000000000001a91
(XEN) L3[0x1ff] = 000000100af39067 0000000000001a92
(XEN) L2[0x1fa] = 0000000000000000 ffffffffffffffff
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 238 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.2.2 x86_64 debug=n Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e033:[<ffffffff812b7872>]
(XEN) RFLAGS: 0000000000000202 EM: 1 CONTEXT: pv guest
(XEN) rax: 0000000000000000 rbx: ffffffffff400000 rcx: 0000000000000004
(XEN) rdx: 0000000000000010 rsi: ffffffffff400000 rdi: ffffffff81a03e68
(XEN) rbp: ffffffff81a03e48 rsp: ffffffff81a03e00 r8: 0000000000000000
(XEN) r9: 0000000000007ff0 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: ffffffff81a03e58 r13: ffffffffff400000 r14: ffffffffff410000
(XEN) r15: ffffffff81a03e68 cr0: 000000008005003b cr4: 00000000000426f0
(XEN) cr3: 000000100af3d000 cr2: ffffffffff400000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=ffffffff81a03e00:
(XEN) 0000000000000004 0000000000000000 0000000000000000 ffffffff812b7872
(XEN) 000000010000e030 0000000000010002 ffffffff81a03e48 000000000000e02b
(XEN) ffffffff812b7869 ffffffff81a03eb8 ffffffff81c80153 0000000000000000
(XEN) 0000000000000000 ffffffff81a03e98 ffffffff81c872a0 0000000000000000
(XEN) ba1afdab0d64b832 ffffffffffffffff ffffffff81c872a0 0000000000000000
(XEN) ffffffff81a03f80 ffffffffffffffff 0000000000000000 ffffffff81a03f68
(XEN) ffffffff81c45605 0000000000000010 ffffffff81c872a0 0000000000000000
(XEN) 0000000000000000 ffffffffffffffff 0000000000000000 ffffffff81a03f08
(XEN) ffffffff8107e0ee ffffffff81a03f68 ffffffff8154b0cd 0000000000000010
(XEN) ffffffff81a03f78 ffffffff81a03f38 ba1afdab0d64b832 ffffffff81a03f58
(XEN) ffffffff81c872a0 0000000000000000 0000000000000000 ffffffffffffffff
(XEN) 0000000000000000 ffffffff81a03fa8 ffffffff81c3fdda ba1afdab0d64b832
(XEN) ffffffff81c8a820 000000000204aa24 0000000000000000 0000000000000000
(XEN) 0000000000000000 ffffffff81a03fc8 ffffffff81c3f33a ffffffff81c34640
(XEN) ffffffff86127000 ffffffff81a03ff8 ffffffff81c4309c 9f9822031f898975
(XEN) 000206d700200800 0000000000000000 0000000000000000 0000000000000000
(XEN) ffffffff85d24000 ffffffff85d25000 ffffffff85d26000 ffffffff85d27000
(XEN) ffffffff85d28000 ffffffff85d29000 ffffffff85d2a000 ffffffff85d2b000
(XEN) ffffffff85d2c000 ffffffff85d2d000 ffffffff85d2e000 ffffffff85d2f000
(XEN) ffffffff85d30000 ffffffff85d31000 ffffffff85d32000 ffffffff85d33000
toracat

toracat

2018-01-09 07:08

manager   ~0030887

See also:

https://lists.centos.org/pipermail/centos-virt/2018-January/005712.html
toracat

toracat

2018-01-17 12:41

manager   ~0030955

According to https://access.redhat.com/solutions/3312501 , Red Hat Engineering is currently working on this issue.
brasilwork

brasilwork

2018-01-27 16:49

reporter   ~0031065

Hy,

 

this kernel version worked for our SL6/ Centos 6 VMs : kernel-2.6.32-696.20.1.el6.x86_64

 

Greets
toracat

toracat

2018-01-27 18:23

manager   ~0031066

Changelog entry [2.6.32-696.20.1.el6]:

[x86] pti/mm: Fix XEN PV boot failure (Waiman Long) [1519799 1519802] {CVE-2017-5754}
themiz

themiz

2018-01-29 11:18

reporter   ~0031080

Hi, still no luck here; boot sequence goes further but I get segmentation fault.
I cannot reproduce the full logs, I attach some xen console screenshots:

Schermata a 2018-01-29 11-27-44.png (15,496 bytes)
toracat

toracat

2018-01-29 12:41

manager   ~0031081

@themiz

Can you try reinstalling the latest kernel? It looks as if the installation did not complete successfully.
themiz

themiz

2018-01-29 15:02

reporter   ~0031085

I was finally able to track down the problem, which is related
to xen-tools:

With kernel 2.6.32-696.20.1.el6.x86_64 AND the latest version of xe-tools-distribution enabled (xe-linux-distribution startup service, provided by these to official rpm):

xe-guest-utilities-xenstore-7.1.0-41.x86_64
xe-guest-utilities-7.1.0-41.x86_64

VM hangs on boot (see screnshot).

If I disabled the xe-linux-distribution service (which I obviously need) VM starts correctly.

Non problem at all with kernel 2.6.32-696.16.1.el6.x86_64

Any advice would be appreciated...

Schermata a 2018-01-29 15-56-38.png (14,372 bytes)
themiz

themiz

2018-01-31 14:21

reporter   ~0031135

Some further testing and update on this, I have been trying the latest kernel available on elrepo:

kernel-ml -> 4.15.0-1.el6.elrepo.x86_64 -> no boot at all
kernel-el -> 4.4.113-1.el6.elrepo.x86_64 -> VM boots fine, xe-linux-distribution service ok, everything working !
Jeff_S

Jeff_S

2018-02-01 18:08

developer   ~0031144

2.6.32-696.20.1.el6.x86_64 boots cleanly for me on the VM that was breaking in the original report.
toracat

toracat

2018-02-01 21:03

manager   ~0031146

Closing as 'resolved' per the OP's reply.

@themiz

Feel free to open a new bug report since yours may be different.

Issue History

Date Modified Username Field Change
2018-01-05 16:15 Jeff_S New Issue
2018-01-05 16:54 themiz Note Added: 0030857
2018-01-05 17:11 themiz Note Added: 0030858
2018-01-06 03:54 dadapea Note Added: 0030861
2018-01-06 15:10 chrisdb Note Added: 0030863
2018-01-09 07:01 toracat Relationship added related to 0014347
2018-01-09 07:01 toracat Status new => acknowledged
2018-01-09 07:08 toracat Note Added: 0030887
2018-01-17 12:41 toracat Note Added: 0030955
2018-01-27 16:49 brasilwork Note Added: 0031065
2018-01-27 18:23 toracat Note Added: 0031066
2018-01-27 19:35 toracat Relationship added related to 0014415
2018-01-29 11:18 themiz File Added: Schermata a 2018-01-29 11-27-44.png
2018-01-29 11:18 themiz Note Added: 0031080
2018-01-29 12:41 toracat Note Added: 0031081
2018-01-29 15:02 themiz File Added: Schermata a 2018-01-29 15-56-38.png
2018-01-29 15:02 themiz Note Added: 0031085
2018-01-31 14:21 themiz Note Added: 0031135
2018-02-01 18:08 Jeff_S Note Added: 0031144
2018-02-01 21:03 toracat Note Added: 0031146
2018-02-01 21:03 toracat Status acknowledged => resolved
2018-02-01 21:03 toracat Resolution open => fixed