View Issue Details

IDProjectCategoryView StatusLast Update
0017631CentOS-7shimpublic2020-08-03 19:41
Reporter0clue 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version7.8-2003 
Target VersionFixed in Version7.8-2003 
Summary0017631: Old hardware UEFI, on 7.8.2003 upgrade to latest release today, unbootable
DescriptionThis appears to be similar to a previous bug/issue with mokutil + shim packages.

See: https://bugs.centos.org/view.php?id=15522

The work-around back then was straight-forward:
download last-known-working-with-reboot shim and mokutil packages, then force-downgrade install and reboots worked again.

This same work-around appears to work the same way.

System was upgraded to all updates for CentOS Linux release 7.8.2003

One or both of these packages provided in 7.8.2003 appear to cause problems with older hardware set to boot with UEFI:
 * mokutil-15-7.el7_9.x86_64
 * shim-x64-15-7.el7_9.x86_64

Breaks reboot on old hardware set to use UEFI boot.
Works fine on new hardware set to use UEFI boot.

For broken hardware, a work-around to make bootable again:
 * Use DRAC with virtual media, remote boot ISO, rescue, shell, "bind" mount {/dev, /proc, /sys, /run to original volume root}, chroot to original volume root, assign IP to interface, configure /etc/resolv.conf, then
 * wget http://mirror.centos.org/centos/7/os/x86_64/Packages/mokutil-15-2.el7.centos.x86_64.rpm
 * wget http://mirror.centos.org/centos/7/os/x86_64/Packages/shim-x64-15-2.el7.centos.x86_64.rpm
 * rpm --force -U mokutil-15-2.el7.centos.x86_64.rpm shim-x64-15-2.el7.centos.x86_64.rpm
 * exit chroot, umount all "bind" mounts, reboot server, and once again this older hardware using UEFI can reboot. to CentOS.

After reboot, login and you can still see:
$ cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

But it at least boots.

Further workaround, blacklist those 2 packages from getting upgraded with future yum calls (and periodically check to see if issue is fixed so you can disable the block for upgrade), or keep around those same 2 RPM and be ready to replace them any time you upgrade in the future until bug is fixed in new package version.
Steps To ReproduceUpgrade older hardware running CentOS with UEFI boot to latest mokutil and shim (I'm on x86_64) then reboot.

Work-around: downgrade to last-known working packages shim and mokutil.
Additional InformationIf it helps, this is the older hardware:
Dell PowerEdge R710 (I said it was old)
BIOS: 6.4.0
RAID Controller:
 * OS View: [megaraid_sas] RAID bus controller: Broadcom / LSI MegaRAID SAS 1078 (rev 04)
 * Product Name : PERC 6/i Integrated
 * FW Package Build: 6.3.3.0002
 * FW Version : 1.22.52-1909
 * BIOS Version : 2.04.00
 * WebBIOS Version : 1.1-46-e_15-Rel
 * Ctrl-R Version : 1.02-015B
 * Preboot CLI Version: 01.00-022:#%00005
 * Boot Block Version : 1.00.00.01-0011

Please let me know if you need more information.

TagsNowFixedForMe shim-x64-15-8, reboot fail uefi efi shim mokutil
abrt_hash
URL

Relationships

related to 0017638 new CentOS-8 Upgrade to 8.2.2004 results in unbootable machine 

Activities

ManuelWolfshant

ManuelWolfshant

2020-07-30 11:23

manager   ~0037448

can you try the method reported at https://bugzilla.redhat.com/show_bug.cgi?id=1861977#c7 ?
dasergatskov

dasergatskov

2020-07-30 15:19

reporter   ~0037450

I have a fairly new hardware with Centos 8.2 and I have the same problem.

Dmitri.
--
b@enlnt.com

b@enlnt.com

2020-07-30 16:02

reporter   ~0037452

I've got hit by this as well.

I have Supermicro based system (motherboard X11SSH-F), with Secure boot disabled. UEFI only boot.

Resolved by booting latest minimal ISO, chrooting and running

yum downgrade grub2* shim mokutil
jshrader

jshrader

2020-07-30 20:01

reporter   ~0037453

Same issue here with two Centos 8.2 VMs in Microsoft Hyper-V.

Both Gen2.

One version 5 secure boot disabled on Server 2012r2; the other version 9 secure boot enabled on Server 2019.
gbailey

gbailey

2020-07-30 22:15

reporter   ~0037455

Same here w/ Intel NUC running CentOS 8.2. The "yum downgrade" workaround doesn't work for me, it says:
Package shim-x64 of lowest version already installed, cannot downgrade it.
Package grub... (same status)
etc.

Any other workaround ideas? There's a shimx64.efi file attached to the RH bugzilla, but I don't know if that's specific to RHEL, and my CentOS 8 box (using rescue mode) shows both "shimx86.efi" and "shimx64-centos.efi", and I'm not sure if I can replace either of these with the RHEL bugzilla attachment.
dasergatskov

dasergatskov

2020-07-30 22:53

reporter   ~0037456

If you have your installation USB/CDROM you can do
cd /var/run/path_to_USB/CentOS-8-1-1911-x86_64-dvd/BaseOS/Packages/
(adjust the path as appropriate for your case)
dnf downgrade ./grub2* ./shim-64*
Also, i think, if you do not use secure boot, you can just uninstall shim (at least it worked for me).

Dmitri.
--
dbennett

dbennett

2020-07-31 00:16

reporter   ~0037458

Hello,

Can confirm this breaks systems using Supermicro based Motherboards, one client is running x11scz-f and started failing after updates. As well, x11scm-f seems to be affected as well, must be the X11 Series. I am trying to verify on a X11SSL-f and get some debug output to help assist.

Checking if the above workaround works as well...
toracat

toracat

2020-07-31 00:23

manager   ~0037460

Today's announcement from Red Hat says, "Red Hat has identified a cause and is working on a resolution to provide to our customers."
0clue

0clue

2020-07-31 10:31

reporter   ~0037461

On 2020-07-30 11:23 ManuelWolfshant wrote:
> can you try the method reported at https://bugzilla.redhat.com/show_bug.cgi?id=1861977#c7 ?

From https://bugzilla.redhat.com/show_bug.cgi?id=1861977#c7:
> solved this by actually copying the grubenv file to /boot/grub2 instead of relying on the symlink to efi.

On the old hardware, /boot/grub2/grubenv was a symlink to "../efi/EFI/centos/grubenv"
# cd /boot/grub2
# mv grubenv grubenv.old-link
# cp -a ../efi/EFI/centos/grubenv ./
(Now, grubenv is a regular file)
# chmod 644 grubenv
# chown root:root grubenv
(duplicated owner and mode from DIFFERENT server with grubenv as a regular file which works with latest mokutil and shim.)
# grub2-mkconfig
(in case it is needed)
# yum update
(upgrades to latest mokutil and shim)
# reboot

Machine becomes unbootable.

For broken hardware, a work-around to make bootable again:
 * Use DRAC with virtual media, remote boot ISO, rescue, shell, "bind" mount {/dev, /proc, /sys, /run to original volume root}, chroot to original volume root, assign IP to interface, configure /etc/resolv.conf, then
 * (Previously downloaded 2 older RPM with wget found)
 * rpm --force -U mokutil-15-2.el7.centos.x86_64.rpm shim-x64-15-2.el7.centos.x86_64.rpm
 * exit chroot, umount all "bind" mounts, reboot server, and once again this older hardware using UEFI can reboot. to CentOS.
 * reboot
 * rhel-autorelabel runs on boot, takes a while, then reboots (just like last time after downgrade I reported about in top post)
 * boots normally, and services running

Sorry. That suggestion did not address this specific issue on this older server.

Thanks for the idea. :-)

The old server boots with the grubenv being a symlink or being a regular file once I downgrade to the older mokutil and shim.
With the newest mokutil and shim, neither the grubenv symlink nor regular file allow the older machine to boot.

Hope this feedback helps you.
vpssupport

vpssupport

2020-07-31 17:24

reporter   ~0037463

1. boot rescue centos CD
2. allow the system to find and mount /mnt/sysimage
3. Mount the needfuls:
mount --bind /dev /mnt/sysimage/dev
mount -t proc none /mnt/sysimage/proc
mount -t sysfs none /mnt/sysimage/sys
4. chroot /mnt/sysimage
5. ifup eno1
6. yum history list (get the first id in the list which would be the latest, the further you go down the further back you go)
7. yum history info <ID>
8. Match and see if grub2, mokutil, and similar were updated -- if it was do:
9. yum history undo <ID>
10. exit
 11.reboot now
ManuelWolfshant

ManuelWolfshant

2020-07-31 17:57

manager   ~0037464

Red Hat published this article: https://access.redhat.com/solutions/5272311
ManuelWolfshant

ManuelWolfshant

2020-08-02 13:39

manager   ~0037476

I am closing this bug as the shim packages with the correct fix were pushed to mirrors. Make sure you use shim-x64-15-15.el8_2.x86_64.rpm ( EL8 ) or respectively shim-x64-15-8.el7_8.x86_64.rpm ( EL7 ) ( or newer )

Issue History

Date Modified Username Field Change
2020-07-30 09:26 0clue New Issue
2020-07-30 09:26 0clue Tag Attached: reboot fail uefi efi shim mokutil
2020-07-30 11:23 ManuelWolfshant Note Added: 0037448
2020-07-30 15:19 dasergatskov Note Added: 0037450
2020-07-30 16:02 b@enlnt.com Note Added: 0037452
2020-07-30 20:01 jshrader Note Added: 0037453
2020-07-30 22:15 gbailey Note Added: 0037455
2020-07-30 22:53 dasergatskov Note Added: 0037456
2020-07-31 00:10 toracat Status new => assigned
2020-07-31 00:16 dbennett Note Added: 0037458
2020-07-31 00:23 toracat Note Added: 0037460
2020-07-31 10:31 0clue Note Added: 0037461
2020-07-31 17:24 vpssupport Note Added: 0037463
2020-07-31 17:57 ManuelWolfshant Note Added: 0037464
2020-08-02 07:13 toracat Relationship added related to 0017638
2020-08-02 13:39 ManuelWolfshant Status assigned => closed
2020-08-02 13:39 ManuelWolfshant Resolution open => fixed
2020-08-02 13:39 ManuelWolfshant Fixed in Version => 7.8-2003
2020-08-02 13:39 ManuelWolfshant Note Added: 0037476
2020-08-03 05:41 0clue Tag Attached: NowFixedForMe shim-x64-15-8