View Issue Details

IDProjectCategoryView StatusLast Update
0018117CentOS-7kernelpublic2021-06-09 07:33
Reporterraydecampo Assigned To 
PrioritynormalSeverityblockReproducibilityalways
Status resolvedResolutionfixed 
Product Version7.9.2009 
Summary0018117: Cannot boot kernel 3.10.0-1160.21.1.el7.x86_64 on Hyper-V
DescriptionAfter updating to kernel version 3.10.0-1160.21.1.el7.x86_64 on a Hyper-V (Windows 10) VM, the OS will not boot. There appear to be error messages output but the display is affected (squashed) and they are illegible. Selecting another kernel from the grub menu allows the OS to boot.

I tried "yum reinstall kernel" and also removing the kernel version and reinstalling it, all with no change in the outcome. This has occurred on two separate Windows 10 machines.
Steps To ReproduceUpdate the kernel to version 3.10.0-1160.21.1.el7.x86_64 on a CentOS 7 guest on Windows 10 Hyper-V. Then reboot the guest.
Additional InformationI am assuming that the fact that this is happening for guests in a Hyper-V host is relevant but I haven't tested this in any other environment.
TagsNo tags attached.
abrt_hash
URL

Activities

toracat

toracat

2021-03-22 18:07

manager   ~0038312

A RHEL user is reporting the same issue:

https://access.redhat.com/discussions/5895461
toracat

toracat

2021-03-22 18:19

manager   ~0038313

Because this is a bug in the RHEL kernel code, this needs to be reported upstream at bugzilla.redhat.com . The following patch in the changelog is a suspect:

[Hyper-V][RHEL-7.9]video: hyperv_fb: Fix the cache type when mapping the VRAM Edit (BZ#1908896)
toracat

toracat

2021-03-22 21:46

manager   ~0038314

Bug report filed:

https://bugzilla.redhat.com/show_bug.cgi?id=1941841

* Description of problem (copied from https://bugs.centos.org/view.php?id=18117):

After updating to kernel version 3.10.0-1160.21.1.el7.x86_64 on a Hyper-V (Windows 10) VM, the OS will not boot. There appear to be error messages output but the display is affected (squashed) and they are illegible. Selecting another kernel from the grub menu allows the OS to boot.

* The following patch seems to have caused the issue:

[Hyper-V][RHEL-7.9]video: hyperv_fb: Fix the cache type when mapping the VRAM Edit (BZ#1908896)

kernel commit:
commit 5f1251a48c17b54939d7477305e39679a565382c
Author: Dexuan Cui <decui@microsoft.com>
Date: Tue Nov 17 16:03:05 2020 -0800

    video: hyperv_fb: Fix the cache type when mapping the VRAM

* The issue has been fixed by:

Stable kernel commit:
commit 452f087d2ff6decf298149e0bfd9fa5c212a636d
Author: Dexuan Cui <decui@microsoft.com>
Date: Sat Jan 9 14:53:58 2021 -0800

    video: hyperv_fb: Fix the mmap() regression for v5.4.y and older
toracat

toracat

2021-03-23 00:37

manager   ~0038315

I've built a plus kernel with the patch that is supposed to fix the problem. It is available from:

https://people.centos.org/toracat/kernel/7/bugs/18117/

Please test, if you can.
toracat

toracat

2021-03-24 09:22

manager   ~0038321

@raydecampo

The following question was asked in the referenced RH bug report. Could you find out the host build version of your system?

===================
I am trying to reproduce the issue with 3.10.0-1160.21.1.el7.x86_64 in VM on Hyper-V host, but failed to reproduce on Windows server 2019 (17763-10.0-1-0.1757) or Windows server 2016 (14393-10.0-4-0.4169), perhaps it is related to specify host build, could you help to check the host build version on which you meet this problem?

To get host build, you can run 'dmesg | grep "Hyper-V Host Build"' on the vm
===================
raydecampo

raydecampo

2021-03-24 12:25

reporter   ~0038322

@toracat

Edition: Windows 10 Business
Version: 20H2
OS build: 19042.867
Experience: Windows Feature Experience Pack 120.2212.551.0

There is something more going on, yesterday I spun up a fresh install of CentOS 7 on a new VM in the same host (with the intent of testing the plus kernel) and I was unable to reproduce the issue. It still reliably reproduces on the existing CentOS 7 installs however. Perhaps it is because the existing instances have a number of kernels installed?

I will try to test the plus kernel on the system exhibiting the issue, although I am a little uncertain concerning how to do that. Should I install the problematic kernel version and then install the plus kernel RPMs from the link? Or just install the plus kernel RPMs? Or something else?
toracat

toracat

2021-03-24 15:31

manager   ~0038324

Thank you. I copied your response in the RHBZ.

Regarding the plus kernel, the best test should be something that determines whether or not the proposed patch fixes the problem. So, take a system that is exhibiting the issue and then install the plus kernel and boot to it. If the problem is corrected, I would say the patch worked.
raydecampo

raydecampo

2021-03-24 17:29

reporter   ~0038325

Unfortunately the plus kernel did not address the issue, no change in behavior.

First I tried installing just the plus kernel. When that failed I installed all the RPMs from the linked location.
toracat

toracat

2021-03-24 17:59

manager   ~0038326

Thanks for testing and reporting back. This could mean that the culprit is something else. Needs more digging.
toracat

toracat

2021-03-27 08:22

manager   ~0038338

@raydecampo

As seen in the upstream BZ, a microsoft developer identified the cause of the bug and proposed a fix:

https://bugzilla.redhat.com/show_bug.cgi?id=1941841#c53

I have used the suggested patch:

--- drivers/video/hyperv_fb.c.old 2021-03-26 19:09:14.694996600 -0700
+++ drivers/video/hyperv_fb.c 2021-03-26 19:09:37.602996600 -0700
@@ -956,13 +956,14 @@
                goto error1;
        }

+ hvfb_get_option(info);
+
        ret = hvfb_getmem(hdev, info);
        if (ret) {
                pr_err("No memory for framebuffer\n");
                goto error2;
        }

- hvfb_get_option(info);
        pr_info("Screen resolution: %dx%d, Color depth: %d\n",
                screen_width, screen_height, screen_depth);

and rebuilt the plus kernel (kernel-plus-3.10.0-1160.21.1.el7.centos.plus.bug18117.1.x86_64.rpm). You can find it here:

https://people.centos.org/toracat/kernel/7/bugs/18117/

Could you try testing this kernel and see if the patch has fixed the issue?
toracat

toracat

2021-03-28 23:32

manager   ~0038339

I have updated the plus kernel set to kernel-plus-3.10.0-1160.21.1.el7.centos.plus.bug18117.2.x86_64.rpm.

This version contains the following patch:

commit 67e7cdb4829d ("video: hyperv: hyperv_fb: Obtain screen resolution from Hyper-V host")

It is referenced as "Patch A" in https://bugzilla.redhat.com/show_bug.cgi?id=1941841#c58

Please test, if you can.
raydecampo

raydecampo

2021-03-29 10:16

reporter   ~0038342

@toracat

I tested the new plus kernel you provided and that does resolve the issue. Thank you for your help.
toracat

toracat

2021-03-29 16:18

manager   ~0038345

@raydecampo

Thanks for the feedback. Great to hear the fix was confirmed. According to the latest comment on the RHBZ:

"We plan to include the missing patch and push a fix out in an upcoming RHEL 7.9.z batch update. You can follow the current status in this BZ."

If the patch does not make it in the next update, the plus kernel will carry it until the distro kernel gets fixed.
toracat

toracat

2021-04-06 22:23

manager   ~0038364

kernel-4.18.0-240.22.1.el8_3 was released upstream. The hyper-v patch is not in there, so the plus kernel will be built with it.
toracat

toracat

2021-06-09 07:33

manager   ~0038483

Fixed in kernel-3.10.0-1160.31.1.el7.

Issue History

Date Modified Username Field Change
2021-03-22 14:36 raydecampo New Issue
2021-03-22 18:07 toracat Note Added: 0038312
2021-03-22 18:19 toracat Status new => acknowledged
2021-03-22 18:19 toracat Note Added: 0038313
2021-03-22 21:46 toracat Note Added: 0038314
2021-03-23 00:37 toracat Note Added: 0038315
2021-03-23 00:49 toracat Status acknowledged => assigned
2021-03-24 09:22 toracat Note Added: 0038321
2021-03-24 12:25 raydecampo Note Added: 0038322
2021-03-24 15:31 toracat Note Added: 0038324
2021-03-24 17:29 raydecampo Note Added: 0038325
2021-03-24 17:59 toracat Note Added: 0038326
2021-03-27 08:22 toracat Note Added: 0038338
2021-03-28 23:32 toracat Note Added: 0038339
2021-03-29 10:16 raydecampo Note Added: 0038342
2021-03-29 16:18 toracat Note Added: 0038345
2021-04-06 22:23 toracat Note Added: 0038364
2021-06-09 07:33 toracat Status assigned => resolved
2021-06-09 07:33 toracat Resolution open => fixed
2021-06-09 07:33 toracat Note Added: 0038483