View Issue Details

IDProjectCategoryView StatusLast Update
0017452CentOS-7microcode_ctlpublic2020-06-28 16:50
Reporterdbennett 
PriorityurgentSeveritycrashReproducibilityalways
Status resolvedResolutionfixed 
Product Version7.8-2003 
Target VersionFixed in Version7.8-2003 
Summary0017452: microcode_ctl for CVE-2020-0543 is causing X11SSL-F with E3-12xx v5 Formerly Known as Skylake
DescriptionYesterday, after a recent change in response to RHSA-2020:2432 a microcode_ctl update was pushed from upstream Redhat of which now causes servers running the following hardware configuration to lock up in the middle of a yum update -- it seems to halt at microcode_ctl and then causes the server to hang.

Supermicro X11SSL-f Motherboard
Processor: E3-12xx v5 Formerly known as Skylake
RAM: Any type of configuration sizes (16GB, 32GB ,64GB)

I have tried both CentOS 7.8-2003 and CentOS 7.6-1810 we are deploying CentOS 7.6-1810 at the moment from our images.
Steps To ReproduceOnce installing fresh CentOS 7 run yum update and allow it to install microcode_ctl it crashes the server and then it reboots, once you get it to reboot it stops at switching root because its failing to execute /sbin/init



Additional Information[root@X11TESTSERVER ~]# yum list installed | grep microcode_ctl
Failed to set locale, defaulting to C
microcode_ctl.x86_64 2:2.1-61.el7 @anaconda
[root@X11TESTSERVER ~]# cat /etc/*release
CentOS Linux release 7.8.2003 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.8.2003 (Core)
CentOS Linux release 7.8.2003 (Core)
[root@X11TESTSERVER ~]# yum update -y
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.ash.fastserv.com
 * extras: mirror.umd.edu
 * updates: ftpmirror.your.org
Resolving Dependencies
--> Running transaction check
---> Package bind-export-libs.x86_64 32:9.11.4-16.P2.el7 will be updated
---> Package bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6 will be an update
---> Package binutils.x86_64 0:2.27-43.base.el7 will be updated
---> Package binutils.x86_64 0:2.27-43.base.el7_8.1 will be an update
---> Package device-mapper.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package firewalld.noarch 0:0.6.3-8.el7 will be updated
---> Package firewalld.noarch 0:0.6.3-8.el7_8.1 will be an update---> Package firewalld-filesystem.noarch 0:0.6.3-8.el7 will be updated
---> Package firewalld-filesystem.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package iprutils.x86_64 0:2.4.17.1-3.el7 will be updated
---> Package iprutils.x86_64 0:2.4.17.1-3.el7_7 will be an update
---> Package kernel.x86_64 0:3.10.0-1127.10.1.el7 will be installed
---> Package kernel-tools.x86_64 0:3.10.0-1127.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-1127.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package lvm2.x86_64 7:2.02.186-7.el7 will be updated
---> Package lvm2.x86_64 7:2.02.186-7.el7_8.2 will be an update
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7 will be updated
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
---> Package microcode_ctl.x86_64 2:2.1-61.el7 will be updated
---> Package microcode_ctl.x86_64 2:2.1-61.6.el7_8 will be an update
---> Package python-firewall.noarch 0:0.6.3-8.el7 will be updated
---> Package python-firewall.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package python-perf.x86_64 0:3.10.0-1127.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package systemd.x86_64 0:219-73.el7.1 will be updated
---> Package systemd.x86_64 0:219-73.el7_8.6 will be an update
---> Package systemd-libs.x86_64 0:219-73.el7.1 will be updated
---> Package systemd-libs.x86_64 0:219-73.el7_8.6 will be an update
---> Package systemd-sysv.x86_64 0:219-73.el7.1 will be updated
---> Package systemd-sysv.x86_64 0:219-73.el7_8.6 will be an update
---> Package tzdata.noarch 0:2019c-1.el7 will be updated
---> Package tzdata.noarch 0:2020a-1.el7 will be an update
---> Package yum-plugin-fastestmirror.noarch 0:1.1.31-53.el7 will be updated
---> Package yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================== Package Arch Version Repository Size
==========================================================================================================================Installing:
 kernel x86_64 3.10.0-1127.10.1.el7 updates 50 M
Updating:
 bind-export-libs x86_64 32:9.11.4-16.P2.el7_8.6 updates 1.1 M
 binutils x86_64 2.27-43.base.el7_8.1 updates 5.9 M
 device-mapper x86_64 7:1.02.164-7.el7_8.2 updates 295 k
 device-mapper-event x86_64 7:1.02.164-7.el7_8.2 updates 191 k
 device-mapper-event-libs x86_64 7:1.02.164-7.el7_8.2 updates 190 k
 device-mapper-libs x86_64 7:1.02.164-7.el7_8.2 updates 324 k
 firewalld noarch 0.6.3-8.el7_8.1 updates 443 k
 firewalld-filesystem noarch 0.6.3-8.el7_8.1 updates 51 k
 iprutils x86_64 2.4.17.1-3.el7_7 updates 243 k
 kernel-tools x86_64 3.10.0-1127.10.1.el7 updates 8.0 M
 kernel-tools-libs x86_64 3.10.0-1127.10.1.el7 updates 8.0 M
 lvm2 x86_64 7:2.02.186-7.el7_8.2 updates 1.3 M
 lvm2-libs x86_64 7:2.02.186-7.el7_8.2 updates 1.1 M
 microcode_ctl x86_64 2:2.1-61.6.el7_8 updates 2.6 M
 python-firewall noarch 0.6.3-8.el7_8.1 updates 354 k
 python-perf x86_64 3.10.0-1127.10.1.el7 updates 8.0 M
 systemd x86_64 219-73.el7_8.6 updates 5.1 M
 systemd-libs x86_64 219-73.el7_8.6 updates 416 k
 systemd-sysv x86_64 219-73.el7_8.6 updates 94 k
 tzdata noarch 2020a-1.el7 updates 495 k
 yum-plugin-fastestmirror noarch 1.1.31-54.el7_8 updates 34 k

Transaction Summary
==========================================================================================================================Install 1 Package
Upgrade 21 Packages

Total size: 94 M
Total download size: 92 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/14): kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64.rpm | 8.0 MB 00:00:00
(2/14): lvm2-2.02.186-7.el7_8.2.x86_64.rpm | 1.3 MB 00:00:00
(3/14): lvm2-libs-2.02.186-7.el7_8.2.x86_64.rpm | 1.1 MB 00:00:00
(4/14): python-firewall-0.6.3-8.el7_8.1.noarch.rpm | 354 kB 00:00:00
(5/14): kernel-tools-3.10.0-1127.10.1.el7.x86_64.rpm | 8.0 MB 00:00:00
(6/14): binutils-2.27-43.base.el7_8.1.x86_64.rpm | 5.9 MB 00:00:00
(7/14): python-perf-3.10.0-1127.10.1.el7.x86_64.rpm | 8.0 MB 00:00:00
(8/14): systemd-sysv-219-73.el7_8.6.x86_64.rpm | 94 kB 00:00:00
(9/14): microcode_ctl-2.1-61.6.el7_8.x86_64.rpm | 2.6 MB 00:00:00
(10/14): systemd-219-73.el7_8.6.x86_64.rpm | 5.1 MB 00:00:00
(11/14): tzdata-2020a-1.el7.noarch.rpm | 495 kB 00:00:00
(12/14): yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch.rpm | 34 kB 00:00:00
(13/14): systemd-libs-219-73.el7_8.6.x86_64.rpm | 416 kB 00:00:00
(14/14): kernel-3.10.0-1127.10.1.el7.x86_64.rpm | 50 MB 00:00:05
--------------------------------------------------------------------------------------------------------------------------Total 17 MB/s | 92 MB 00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating : systemd-libs-219-73.el7_8.6.x86_64 1/43
  Updating : systemd-219-73.el7_8.6.x86_64 2/43
  Updating : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64 3/43
  Updating : 7:device-mapper-1.02.164-7.el7_8.2.x86_64 4/43
  Updating : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64 5/43
  Updating : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64 6/43
  Updating : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64 7/43
  Updating : firewalld-filesystem-0.6.3-8.el7_8.1.noarch 8/43
  Updating : python-firewall-0.6.3-8.el7_8.1.noarch 9/43
  Updating : kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64 10/43
  Updating : kernel-tools-3.10.0-1127.10.1.el7.x86_64 11/43
  Updating : firewalld-0.6.3-8.el7_8.1.noarch 12/43
  Updating : 7:lvm2-2.02.186-7.el7_8.2.x86_64 13/43
  Updating : 2:microcode_ctl-2.1-61.6.el7_8.x86_64 14/43

Then I drop the SSH Session

Ubuntu reports the same thing after rolling their patches out: https://usn.ubuntu.com/4385-2/


tl;dr video of it via IPMI: https://screencaps.donaldbennett.net/76KrAuABVj.mp4

Here is the boot process after the crash: https://screencaps.donaldbennett.net/UMJ7HJUMbW.mp4
TagsNo tags attached.
abrt_hash
URL

Relationships

has duplicate 0017484 closedIssue Tracker Problems after update with microcode_ctl-2.1-61.6.el7_8.x86_64 

Activities

tru

tru

2020-06-12 08:38

administrator   ~0037083

see also: https://www.hivelocity.net/blog/centos-update-advisory/
dbennett

dbennett

2020-06-12 08:39

reporter   ~0037084

Correct, I am apart of the Company Hivelocity.
TrevorH

TrevorH

2020-06-12 10:25

manager   ~0037085

You power cycled the machine while it was in mid-update, specifically while the microcode_ctl package was being updated. If you do this then the new kernel that was just installed is not complete as the initramfs for the newly installed kernel is not generated until the cleanup cycle of the yum update process. So by power cycling the machine you have half installed the new kernel and there is no valid initramfs file for it so it will not boot. Now, *why* the microcode_ctl update appeared to be hung is a different question but power cycling the machine in mid-update is pretty much guaranteed to mess things up badly. You probably have a whole bunch of duplicate packages installed as well.

Does this work if you exclude microcode_ctl from the update process?
tinpotnick

tinpotnick

2020-06-12 10:42

reporter   ~0037086

I am adding a note to second this issue. I have tried on 2 separate servers with the same result. The final time I left the server more than an hour trying to install microcode_ctl. I am also using Supermicro motherboards but with a Xeon E3-1585V5 processor.
tinpotnick

tinpotnick

2020-06-12 11:22

reporter   ~0037087

I have just done a clean install. Excluded microcode_ctl and it all worked fine.
tru

tru

2020-06-12 12:03

administrator   ~0037088

tinpotnick: thanks for the update.
Does the machine freeze also when you only install the microcode_ctl package after all the other upgrade are applied?
TrevorH

TrevorH

2020-06-12 12:20

manager   ~0037089

I believe microcode_ctl has scripts which attempt to activate the new microcode as part of the install. That's why it's hung. It would do the same thing on reboot since the package has been installed, even though it's not yet complete. The weird error about /bin/init not being the correct format executable is almost certainly due to the power cycle and an incompletely installed kernel/initramfs but even if it wasn't, it would probably hang on reboot when it got to trying to apply the broken microcode.

For now, the best thing that anyone with the affected processors can do is to exclude this microcode_ctl package from the update.
ManuelWolfshant

ManuelWolfshant

2020-06-12 12:42

manager   ~0037090

... and file bugs at RedHat, Intel and Supermicro
dbennett

dbennett

2020-06-12 16:34

reporter   ~0037091

Thanks for all the info all, I can leave the server alone for hours and it will not go anywhere, I will setup a fresh install and exclude microcode_ctl in exclude= in /etc/yum.conf, run the update, see if it hangs and let the updates complete for microcode_ctl after removing, per tru. i will start working on filing those cases elsewhere as well and will let you all know the headway I make.

Thanks!
ManuelWolfshant

ManuelWolfshant

2020-06-13 11:47

manager   ~0037092

@dbennett: it was reported that the update worked if used as rpm -U --noscripts. maybe you can try that as well ?
nccsoft

nccsoft

2020-06-14 07:08

reporter   ~0037096

I also encountered the same problem. My motherboard was X11SSL-F. Even worse, after the update, my RAID data was damaged and many files became 0byte. I wanted to cry when I saw it.
robetek

robetek

2020-06-14 16:32

reporter   ~0037098

I encountered this issue yesterday on HPE ProLiant ML10 Gen9 Server running CentOS release 7.8.2003 with Linux 3.10.0-1127.10.1.el7.x86_64 kernel. Fortunately the microcode_ctl package was the only update involved so my path to recovery after hard shut down was:
1. Used grub editor during boot to include dis_ucode_ldr in boot arguments
2. Entered ctrl-x from grub editor to continue boot
3. yum remove microcode_ctl (note that this removed microcode_ctl 2.1-61.6.el7-8)
4. yum install microcode_ctl-2.1-61.el7
5. Subsequent shut down and boot up was then normal
nccsoft

nccsoft

2020-06-15 02:43

reporter   ~0037102

robetek, I am rather unlucky. grub.cfg becomes 0byte, and cannot be booted. The rules of selinux are all changed to 0byte, and many configuration files have become 0byte. Use the previous day's backup to restore. I thought the HDD in the RAID was broken. After replacing the HDD with another one, yum update was done again, and the RAID was damaged again. I realized that it was not the HDD.
dbennett

dbennett

2020-06-15 03:27

reporter   ~0037103

After getting a clean install of CentOS 7 going, I excluded microcode_ctl from /etc/yum.conf and ran updates successfully, removed it after the reboot, updated, the server hung but after 2 minutes, it rebooted and is online.

It seems that something while its in the middle of updating all the other packages causes this one to hang, I will work on giving rpm -U --noscripts a try in a bit per @ManuelWolfshant

history.txt (25,839 bytes)
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ssh root@23.111.168.58
root@23.111.168.58's password:
[root@X11TESTSERVER ~]# vi /etc/yum.conf
[root@X11TESTSERVER ~]# yum update -y
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.netdepot.com
 * extras: mirror.teklinks.com
 * updates: mirror.mojohost.com
base                                                                                              | 3.6 kB  00:00:00     
extras                                                                                            | 2.9 kB  00:00:00     
updates                                                                                           | 2.9 kB  00:00:00     
(1/4): base/7/x86_64/group_gz                                                                     | 153 kB  00:00:00     
(2/4): updates/7/x86_64/primary_db                                                                | 2.1 MB  00:00:00     
(3/4): extras/7/x86_64/primary_db                                                                 | 194 kB  00:00:00     
(4/4): base/7/x86_64/primary_db                                                                   | 6.1 MB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package bind-export-libs.x86_64 32:9.11.4-16.P2.el7 will be updated
---> Package bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6 will be an update
---> Package binutils.x86_64 0:2.27-43.base.el7 will be updated
---> Package binutils.x86_64 0:2.27-43.base.el7_8.1 will be an update
---> Package device-mapper.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package firewalld.noarch 0:0.6.3-8.el7 will be updated
---> Package firewalld.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package firewalld-filesystem.noarch 0:0.6.3-8.el7 will be updated
---> Package firewalld-filesystem.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package iprutils.x86_64 0:2.4.17.1-3.el7 will be updated
---> Package iprutils.x86_64 0:2.4.17.1-3.el7_7 will be an update
---> Package kernel.x86_64 0:3.10.0-1127.10.1.el7 will be installed
---> Package kernel-tools.x86_64 0:3.10.0-1127.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-1127.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package lvm2.x86_64 7:2.02.186-7.el7 will be updated
---> Package lvm2.x86_64 7:2.02.186-7.el7_8.2 will be an update
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7 will be updated
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
---> Package python-firewall.noarch 0:0.6.3-8.el7 will be updated
---> Package python-firewall.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package python-perf.x86_64 0:3.10.0-1127.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package systemd.x86_64 0:219-73.el7.1 will be updated
---> Package systemd.x86_64 0:219-73.el7_8.6 will be an update
---> Package systemd-libs.x86_64 0:219-73.el7.1 will be updated
---> Package systemd-libs.x86_64 0:219-73.el7_8.6 will be an update
---> Package systemd-sysv.x86_64 0:219-73.el7.1 will be updated
---> Package systemd-sysv.x86_64 0:219-73.el7_8.6 will be an update
---> Package tzdata.noarch 0:2019c-1.el7 will be updated
---> Package tzdata.noarch 0:2020a-1.el7 will be an update
---> Package yum-plugin-fastestmirror.noarch 0:1.1.31-53.el7 will be updated
---> Package yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================= Package                              Arch               Version                               Repository           Size 
=========================================================================================================================Installing:
 kernel                               x86_64             3.10.0-1127.10.1.el7                  updates              50 M 
Updating:
 bind-export-libs                     x86_64             32:9.11.4-16.P2.el7_8.6               updates             1.1 M 
 binutils                             x86_64             2.27-43.base.el7_8.1                  updates             5.9 M 
 device-mapper                        x86_64             7:1.02.164-7.el7_8.2                  updates             295 k 
 device-mapper-event                  x86_64             7:1.02.164-7.el7_8.2                  updates             191 k 
 device-mapper-event-libs             x86_64             7:1.02.164-7.el7_8.2                  updates             190 k 
 device-mapper-libs                   x86_64             7:1.02.164-7.el7_8.2                  updates             324 k 
 firewalld                            noarch             0.6.3-8.el7_8.1                       updates             443 k 
 firewalld-filesystem                 noarch             0.6.3-8.el7_8.1                       updates              51 k 
 iprutils                             x86_64             2.4.17.1-3.el7_7                      updates             243 k 
 kernel-tools                         x86_64             3.10.0-1127.10.1.el7                  updates             8.0 M 
 kernel-tools-libs                    x86_64             3.10.0-1127.10.1.el7                  updates             8.0 M 
 lvm2                                 x86_64             7:2.02.186-7.el7_8.2                  updates             1.3 M 
 lvm2-libs                            x86_64             7:2.02.186-7.el7_8.2                  updates             1.1 M 
 python-firewall                      noarch             0.6.3-8.el7_8.1                       updates             354 k 
 python-perf                          x86_64             3.10.0-1127.10.1.el7                  updates             8.0 M 
 systemd                              x86_64             219-73.el7_8.6                        updates             5.1 M 
 systemd-libs                         x86_64             219-73.el7_8.6                        updates             416 k 
 systemd-sysv                         x86_64             219-73.el7_8.6                        updates              94 k 
 tzdata                               noarch             2020a-1.el7                           updates             495 k 
 yum-plugin-fastestmirror             noarch             1.1.31-54.el7_8                       updates              34 k 

Transaction Summary
=========================================================================================================================Install   1 Package
Upgrade  20 Packages

Total download size: 92 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/updates/packages/device-mapper-event-1.02.164-7.el7_8.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for device-mapper-event-1.02.164-7.el7_8.2.x86_64.rpm is not installed
(1/21): device-mapper-event-1.02.164-7.el7_8.2.x86_64.rpm                                         | 191 kB  00:00:00     
(2/21): binutils-2.27-43.base.el7_8.1.x86_64.rpm                                                  | 5.9 MB  00:00:00     
(3/21): device-mapper-1.02.164-7.el7_8.2.x86_64.rpm                                               | 295 kB  00:00:00     
(4/21): bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64.rpm                                          | 1.1 MB  00:00:00     
(5/21): device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64.rpm                                    | 190 kB  00:00:00     
(6/21): firewalld-filesystem-0.6.3-8.el7_8.1.noarch.rpm                                           |  51 kB  00:00:00     
(7/21): device-mapper-libs-1.02.164-7.el7_8.2.x86_64.rpm                                          | 324 kB  00:00:00     
(8/21): iprutils-2.4.17.1-3.el7_7.x86_64.rpm                                                      | 243 kB  00:00:00     
(9/21): firewalld-0.6.3-8.el7_8.1.noarch.rpm                                                      | 443 kB  00:00:00     
(10/21): kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64.rpm                                        | 8.0 MB  00:00:00     
(11/21): kernel-tools-3.10.0-1127.10.1.el7.x86_64.rpm                                             | 8.0 MB  00:00:00     
(12/21): lvm2-2.02.186-7.el7_8.2.x86_64.rpm                                                       | 1.3 MB  00:00:00     
(13/21): python-firewall-0.6.3-8.el7_8.1.noarch.rpm                                               | 354 kB  00:00:00     
(14/21): lvm2-libs-2.02.186-7.el7_8.2.x86_64.rpm                                                  | 1.1 MB  00:00:00     
(15/21): python-perf-3.10.0-1127.10.1.el7.x86_64.rpm                                              | 8.0 MB  00:00:00     
(16/21): systemd-libs-219-73.el7_8.6.x86_64.rpm                                                   | 416 kB  00:00:00     
(17/21): systemd-sysv-219-73.el7_8.6.x86_64.rpm                                                   |  94 kB  00:00:00     
(18/21): systemd-219-73.el7_8.6.x86_64.rpm                                                        | 5.1 MB  00:00:00     
(19/21): yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch.rpm                                      |  34 kB  00:00:00     
(20/21): tzdata-2020a-1.el7.noarch.rpm                                                            | 495 kB  00:00:00     
(21/21): kernel-3.10.0-1127.10.1.el7.x86_64.rpm                                                   |  50 MB  00:00:02     
-------------------------------------------------------------------------------------------------------------------------Total                                                                                     33 MB/s |  92 MB  00:00:02     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-8.2003.0.el7.centos.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : systemd-libs-219-73.el7_8.6.x86_64                                                                   1/41 
  Updating   : systemd-219-73.el7_8.6.x86_64                                                                        2/41 
  Updating   : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                            3/41 
  Updating   : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                       4/41 
  Updating   : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                 5/41 
  Updating   : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                      6/41 
  Updating   : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                7/41 
  Updating   : firewalld-filesystem-0.6.3-8.el7_8.1.noarch                                                          8/41 
  Updating   : python-firewall-0.6.3-8.el7_8.1.noarch                                                               9/41 
  Updating   : kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64                                                       10/41 
  Updating   : kernel-tools-3.10.0-1127.10.1.el7.x86_64                                                            11/41 
  Updating   : firewalld-0.6.3-8.el7_8.1.noarch                                                                    12/41 
  Updating   : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                    13/41 
  Updating   : systemd-sysv-219-73.el7_8.6.x86_64                                                                  14/41 
  Updating   : tzdata-2020a-1.el7.noarch                                                                           15/41 
  Updating   : 32:bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64                                                     16/41 
  Updating   : iprutils-2.4.17.1-3.el7_7.x86_64                                                                    17/41 
  Updating   : binutils-2.27-43.base.el7_8.1.x86_64                                                                18/41 
  Updating   : yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch                                                     19/41 
  Installing : kernel-3.10.0-1127.10.1.el7.x86_64                                                                  20/41 
  Updating   : python-perf-3.10.0-1127.10.1.el7.x86_64                                                             21/41 
  Cleanup    : 7:lvm2-2.02.186-7.el7.x86_64                                                                        22/41 
  Cleanup    : firewalld-0.6.3-8.el7.noarch                                                                        23/41 
  Cleanup    : systemd-sysv-219-73.el7.1.x86_64                                                                    24/41 
  Cleanup    : firewalld-filesystem-0.6.3-8.el7.noarch                                                             25/41 
  Cleanup    : python-firewall-0.6.3-8.el7.noarch                                                                  26/41 
  Cleanup    : tzdata-2019c-1.el7.noarch                                                                           27/41 
  Cleanup    : yum-plugin-fastestmirror-1.1.31-53.el7.noarch                                                       28/41 
  Cleanup    : 7:lvm2-libs-2.02.186-7.el7.x86_64                                                                   29/41 
  Cleanup    : 7:device-mapper-event-1.02.164-7.el7.x86_64                                                         30/41 
  Cleanup    : 7:device-mapper-event-libs-1.02.164-7.el7.x86_64                                                    31/41 
  Cleanup    : 7:device-mapper-1.02.164-7.el7.x86_64                                                               32/41 
  Cleanup    : 7:device-mapper-libs-1.02.164-7.el7.x86_64                                                          33/41 
  Cleanup    : systemd-219-73.el7.1.x86_64                                                                         34/41 
  Cleanup    : kernel-tools-3.10.0-1127.el7.x86_64                                                                 35/41 
  Cleanup    : kernel-tools-libs-3.10.0-1127.el7.x86_64                                                            36/41 
  Cleanup    : systemd-libs-219-73.el7.1.x86_64                                                                    37/41 
  Cleanup    : 32:bind-export-libs-9.11.4-16.P2.el7.x86_64                                                         38/41 
  Cleanup    : iprutils-2.4.17.1-3.el7.x86_64                                                                      39/41 
  Cleanup    : binutils-2.27-43.base.el7.x86_64                                                                    40/41 
  Cleanup    : python-perf-3.10.0-1127.el7.x86_64                                                                  41/41 
  Verifying  : python-perf-3.10.0-1127.10.1.el7.x86_64                                                              1/41 
  Verifying  : kernel-3.10.0-1127.10.1.el7.x86_64                                                                   2/41 
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                       3/41 
  Verifying  : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                     4/41 
  Verifying  : yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch                                                      5/41 
  Verifying  : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                            6/41 
  Verifying  : systemd-sysv-219-73.el7_8.6.x86_64                                                                   7/41 
  Verifying  : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                      8/41 
  Verifying  : kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64                                                        9/41 
  Verifying  : binutils-2.27-43.base.el7_8.1.x86_64                                                                10/41 
  Verifying  : systemd-219-73.el7_8.6.x86_64                                                                       11/41 
  Verifying  : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                               12/41 
  Verifying  : firewalld-0.6.3-8.el7_8.1.noarch                                                                    13/41 
  Verifying  : iprutils-2.4.17.1-3.el7_7.x86_64                                                                    14/41 
  Verifying  : systemd-libs-219-73.el7_8.6.x86_64                                                                  15/41 
  Verifying  : kernel-tools-3.10.0-1127.10.1.el7.x86_64                                                            16/41 
  Verifying  : 32:bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64                                                     17/41 
  Verifying  : tzdata-2020a-1.el7.noarch                                                                           18/41 
  Verifying  : python-firewall-0.6.3-8.el7_8.1.noarch                                                              19/41 
  Verifying  : firewalld-filesystem-0.6.3-8.el7_8.1.noarch                                                         20/41 
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                21/41 
  Verifying  : tzdata-2019c-1.el7.noarch                                                                           22/41 
  Verifying  : firewalld-0.6.3-8.el7.noarch                                                                        23/41 
  Verifying  : 7:device-mapper-event-1.02.164-7.el7.x86_64                                                         24/41 
  Verifying  : 32:bind-export-libs-9.11.4-16.P2.el7.x86_64                                                         25/41 
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7.x86_64                                                    26/41 
  Verifying  : 7:lvm2-2.02.186-7.el7.x86_64                                                                        27/41 
  Verifying  : systemd-219-73.el7.1.x86_64                                                                         28/41 
  Verifying  : systemd-sysv-219-73.el7.1.x86_64                                                                    29/41 
  Verifying  : firewalld-filesystem-0.6.3-8.el7.noarch                                                             30/41 
  Verifying  : systemd-libs-219-73.el7.1.x86_64                                                                    31/41 
[main]
  Verifying  : python-firewall-0.6.3-8.el7.noarch                                                                  32/41 
  Verifying  : 7:device-mapper-1.02.164-7.el7.x86_64                                                               33/41 
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7.x86_64                                                          34/41 
  Verifying  : binutils-2.27-43.base.el7.x86_64                                                                    35/41 
  Verifying  : kernel-tools-3.10.0-1127.el7.x86_64                                                                 36/41 
  Verifying  : 7:lvm2-libs-2.02.186-7.el7.x86_64                                                                   37/41 
  Verifying  : kernel-tools-libs-3.10.0-1127.el7.x86_64                                                            38/41 
  Verifying  : iprutils-2.4.17.1-3.el7.x86_64                                                                      39/41 
  Verifying  : python-perf-3.10.0-1127.el7.x86_64                                                                  40/41 
  Verifying  : yum-plugin-fastestmirror-1.1.31-53.el7.noarch                                                       41/41 Installed:
  kernel.x86_64 0:3.10.0-1127.10.1.el7

Updated:
  bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6              binutils.x86_64 0:2.27-43.base.el7_8.1
  device-mapper.x86_64 7:1.02.164-7.el7_8.2                    device-mapper-event.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2         device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2
  firewalld.noarch 0:0.6.3-8.el7_8.1                           firewalld-filesystem.noarch 0:0.6.3-8.el7_8.1
  iprutils.x86_64 0:2.4.17.1-3.el7_7                           kernel-tools.x86_64 0:3.10.0-1127.10.1.el7
  kernel-tools-libs.x86_64 0:3.10.0-1127.10.1.el7              lvm2.x86_64 7:2.02.186-7.el7_8.2
  lvm2-libs.x86_64 7:2.02.186-7.el7_8.2                        python-firewall.noarch 0:0.6.3-8.el7_8.1
  python-perf.x86_64 0:3.10.0-1127.10.1.el7                    systemd.x86_64 0:219-73.el7_8.6
  systemd-libs.x86_64 0:219-73.el7_8.6                         systemd-sysv.x86_64 0:219-73.el7_8.6
  tzdata.noarch 0:2020a-1.el7                                  yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8

Complete!
[root@X11TESTSERVER ~]# reboot now
Connection to 23.111.168.58 closed by remote host.
Connection to 23.111.168.58 closed.
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ping 23.111.168.58
PING 23.111.168.58 (23.111.168.58) 56(84) bytes of data.
64 bytes from 23.111.168.58: icmp_seq=65 ttl=56 time=1.64 ms
64 bytes from 23.111.168.58: icmp_seq=66 ttl=56 time=1.72 ms
64 bytes from 23.111.168.58: icmp_seq=67 ttl=56 time=1.34 ms
64 bytes from 23.111.168.58: icmp_seq=68 ttl=56 time=1.31 ms
64 bytes from 23.111.168.58: icmp_seq=69 ttl=56 time=1.40 ms
64 bytes from 23.111.168.58: icmp_seq=70 ttl=56 time=1.36 ms
64 bytes from 23.111.168.58: icmp_seq=71 ttl=56 time=1.35 ms
64 bytes from 23.111.168.58: icmp_seq=72 ttl=56 time=1.36 ms
^C
--- 23.111.168.58 ping statistics ---
72 packets transmitted, 8 received, 88% packet loss, time 71072ms
rtt min/avg/max/mdev = 1.318/1.441/1.727/0.147 ms
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ssh root@23.111.168.58
root@23.111.168.58's password:
Last login: Sun Jun 14 23:07:53 2020 from 68-233-224-175.static.hvvc.us
[root@X11TESTSERVER ~]# nano /etc/yum.conf
-bash: nano: command not found
[root@X11TESTSERVER ~]# vi /etc/yum.conf
[root@X11TESTSERVER ~]# yum update -y
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.netdepot.com
 * extras: mirror.teklinks.com
 * updates: mirror.mojohost.com
base                                                                                              | 3.6 kB  00:00:00     
extras                                                                                            | 2.9 kB  00:00:00     
updates                                                                                           | 2.9 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package microcode_ctl.x86_64 2:2.1-61.el7 will be updated
---> Package microcode_ctl.x86_64 2:2.1-61.6.el7_8 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================= Package                        Arch                    Version                           Repository                Size 
=========================================================================================================================Updating:
 microcode_ctl                  x86_64                  2:2.1-61.6.el7_8                  updates                  2.6 M 

Transaction Summary
=========================================================================================================================Upgrade  1 Package

Total download size: 2.6 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
microcode_ctl-2.1-61.6.el7_8.x86_64.rpm                                                           | 2.6 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 2:microcode_ctl-2.1-61.6.el7_8.x86_64                                                                 1/2 
packet_write_wait: Connection to 23.111.168.58 port 22: Broken pipe
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ssh root@23.111.168.58
root@23.111.168.58's password: 
Last login: Sun Jun 14 23:24:38 2020
[root@X11TESTSERVER ~]# rpm -qa | grep microcode_ctl
microcode_ctl-2.1-61.el7.x86_64
microcode_ctl-2.1-61.6.el7_8.x86_64
[root@X11TESTSERVER ~]# 
history.txt (25,839 bytes)
dbennett

dbennett

2020-06-15 04:11

reporter   ~0037104

Interesting things, I setup a new test and I yum updated without microcode_ctl, then proceeded to grab the update and installed it per @ManuelWolfshant recommendation, reboot and the server hangs at https://screencaps.donaldbennett.net/jp2launcher_KvC8sTt5Xg.png once you give it about 2-5 minutes the system will reboot and retry with no signs of pinging, happens on both kernels.

Using @robetek method of restoring access to the server works for me and subsequent shutdowns and boot-ups, are then successful.

1. Used grub editor during boot to include dis_ucode_ldr in boot arguments
2. Entered ctrl-x from grub editor to continue boot
3. yum remove microcode_ctl (note that this removed microcode_ctl 2.1-61.6.el7-8)
4. yum install microcode_ctl-2.1-61.el7

history2.txt (31,261 bytes)
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ssh root@23.111.168.58
The authenticity of host '23.111.168.58 (23.111.168.58)' can't be established.
ECDSA key fingerprint is SHA256:uFpvWWzR4wUIi9MjhWLAiidG7ILa3jxpnkPAbpa9CTY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '23.111.168.58' (ECDSA) to the list of known hosts.
root@23.111.168.58's password: 
Last failed login: Sun Jun 14 23:51:07 EDT 2020 from 159.65.13.233 on ssh:notty
There was 1 failed login attempt since the last successful login.
[root@X11TESTSERVER ~]# yum update --exclude=microcode_ctl -y
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: bay.uchicago.edu
 * extras: mirror.teklinks.com 
 * updates: mirror.mojohost.com
base                                                                                              | 3.6 kB  00:00:00     
extras                                                                                            | 2.9 kB  00:00:00     
updates                                                                                           | 2.9 kB  00:00:00     
(1/4): updates/7/x86_64/primary_db                                                                | 2.1 MB  00:00:00     
(2/4): base/7/x86_64/group_gz                                                                     | 153 kB  00:00:00     
(3/4): extras/7/x86_64/primary_db                                                                 | 194 kB  00:00:00     
(4/4): base/7/x86_64/primary_db                                                                   | 6.1 MB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package bind-export-libs.x86_64 32:9.11.4-16.P2.el7 will be updated
---> Package bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6 will be an update
---> Package binutils.x86_64 0:2.27-43.base.el7 will be updated      
---> Package binutils.x86_64 0:2.27-43.base.el7_8.1 will be an update
---> Package device-mapper.x86_64 7:1.02.164-7.el7 will be updated      
---> Package device-mapper.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-event.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2 will be an update
---> Package firewalld.noarch 0:0.6.3-8.el7 will be updated
---> Package firewalld.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package firewalld-filesystem.noarch 0:0.6.3-8.el7 will be updated
---> Package firewalld-filesystem.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package iprutils.x86_64 0:2.4.17.1-3.el7 will be updated
---> Package iprutils.x86_64 0:2.4.17.1-3.el7_7 will be an update
---> Package kernel.x86_64 0:3.10.0-1127.10.1.el7 will be installed
---> Package kernel-tools.x86_64 0:3.10.0-1127.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-1127.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package lvm2.x86_64 7:2.02.186-7.el7 will be updated
---> Package lvm2.x86_64 7:2.02.186-7.el7_8.2 will be an update
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7 will be updated
---> Package lvm2-libs.x86_64 7:2.02.186-7.el7_8.2 will be an update
---> Package python-firewall.noarch 0:0.6.3-8.el7 will be updated
---> Package python-firewall.noarch 0:0.6.3-8.el7_8.1 will be an update
---> Package python-perf.x86_64 0:3.10.0-1127.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-1127.10.1.el7 will be an update
---> Package systemd.x86_64 0:219-73.el7.1 will be updated
---> Package systemd.x86_64 0:219-73.el7_8.6 will be an update
---> Package systemd-libs.x86_64 0:219-73.el7.1 will be updated
---> Package systemd-libs.x86_64 0:219-73.el7_8.6 will be an update
---> Package systemd-sysv.x86_64 0:219-73.el7.1 will be updated
---> Package systemd-sysv.x86_64 0:219-73.el7_8.6 will be an update
---> Package tzdata.noarch 0:2019c-1.el7 will be updated
---> Package tzdata.noarch 0:2020a-1.el7 will be an update
---> Package yum-plugin-fastestmirror.noarch 0:1.1.31-53.el7 will be updated
---> Package yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================= Package                              Arch               Version                               Repository           Size 
=========================================================================================================================Installing:
 kernel                               x86_64             3.10.0-1127.10.1.el7                  updates              50 M 
Updating:
 bind-export-libs                     x86_64             32:9.11.4-16.P2.el7_8.6               updates             1.1 M 
 binutils                             x86_64             2.27-43.base.el7_8.1                  updates             5.9 M 
 device-mapper                        x86_64             7:1.02.164-7.el7_8.2                  updates             295 k 
 device-mapper-event                  x86_64             7:1.02.164-7.el7_8.2                  updates             191 k 
 device-mapper-event-libs             x86_64             7:1.02.164-7.el7_8.2                  updates             190 k 
 device-mapper-libs                   x86_64             7:1.02.164-7.el7_8.2                  updates             324 k 
 firewalld                            noarch             0.6.3-8.el7_8.1                       updates             443 k 
 firewalld-filesystem                 noarch             0.6.3-8.el7_8.1                       updates              51 k 
 iprutils                             x86_64             2.4.17.1-3.el7_7                      updates             243 k 
 kernel-tools                         x86_64             3.10.0-1127.10.1.el7                  updates             8.0 M 
 kernel-tools-libs                    x86_64             3.10.0-1127.10.1.el7                  updates             8.0 M 
 lvm2                                 x86_64             7:2.02.186-7.el7_8.2                  updates             1.3 M 
 lvm2-libs                            x86_64             7:2.02.186-7.el7_8.2                  updates             1.1 M 
 python-firewall                      noarch             0.6.3-8.el7_8.1                       updates             354 k 
 python-perf                          x86_64             3.10.0-1127.10.1.el7                  updates             8.0 M 
 systemd                              x86_64             219-73.el7_8.6                        updates             5.1 M 
 systemd-libs                         x86_64             219-73.el7_8.6                        updates             416 k 
 systemd-sysv                         x86_64             219-73.el7_8.6                        updates              94 k 
 tzdata                               noarch             2020a-1.el7                           updates             495 k 
 yum-plugin-fastestmirror             noarch             1.1.31-54.el7_8                       updates              34 k 

Transaction Summary
=========================================================================================================================Install   1 Package
Upgrade  20 Packages

Total download size: 92 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/updates/packages/bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64.rpm: Header V3 RSA/SHA256 
Signature, key ID f4a80eb5: NOKEY
Public key for bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64.rpm is not installed
(1/21): bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64.rpm                                          | 1.1 MB  00:00:00     
(2/21): device-mapper-1.02.164-7.el7_8.2.x86_64.rpm                                               | 295 kB  00:00:00     
(3/21): binutils-2.27-43.base.el7_8.1.x86_64.rpm                                                  | 5.9 MB  00:00:00     
(4/21): device-mapper-event-1.02.164-7.el7_8.2.x86_64.rpm                                         | 191 kB  00:00:00     
(5/21): device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64.rpm                                    | 190 kB  00:00:00     
(6/21): device-mapper-libs-1.02.164-7.el7_8.2.x86_64.rpm                                          | 324 kB  00:00:00     
(7/21): firewalld-filesystem-0.6.3-8.el7_8.1.noarch.rpm                                           |  51 kB  00:00:00     
(8/21): firewalld-0.6.3-8.el7_8.1.noarch.rpm                                                      | 443 kB  00:00:00     
(9/21): iprutils-2.4.17.1-3.el7_7.x86_64.rpm                                                      | 243 kB  00:00:00     
(10/21): kernel-tools-3.10.0-1127.10.1.el7.x86_64.rpm                                             | 8.0 MB  00:00:00     
(11/21): kernel-3.10.0-1127.10.1.el7.x86_64.rpm                                                   |  50 MB  00:00:00     
(12/21): kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64.rpm                                        | 8.0 MB  00:00:00     
(13/21): lvm2-2.02.186-7.el7_8.2.x86_64.rpm                                                       | 1.3 MB  00:00:00     
(14/21): lvm2-libs-2.02.186-7.el7_8.2.x86_64.rpm                                                  | 1.1 MB  00:00:00     
(15/21): python-firewall-0.6.3-8.el7_8.1.noarch.rpm                                               | 354 kB  00:00:00     
(16/21): python-perf-3.10.0-1127.10.1.el7.x86_64.rpm                                              | 8.0 MB  00:00:00     
(17/21): systemd-219-73.el7_8.6.x86_64.rpm                                                        | 5.1 MB  00:00:00     
(18/21): systemd-sysv-219-73.el7_8.6.x86_64.rpm                                                   |  94 kB  00:00:00     
(19/21): systemd-libs-219-73.el7_8.6.x86_64.rpm                                                   | 416 kB  00:00:00     
(20/21): yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch.rpm                                      |  34 kB  00:00:00     
(21/21): tzdata-2020a-1.el7.noarch.rpm                                                            | 495 kB  00:00:00     
-------------------------------------------------------------------------------------------------------------------------Total                                                                                     50 MB/s |  92 MB  00:00:01     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-8.2003.0.el7.centos.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : systemd-libs-219-73.el7_8.6.x86_64                                                                   1/41 
  Updating   : systemd-219-73.el7_8.6.x86_64                                                                        2/41 
  Updating   : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                            3/41 
  Updating   : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                       4/41 
  Updating   : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                 5/41 
  Updating   : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                      6/41 
  Updating   : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                                7/41 
  Updating   : firewalld-filesystem-0.6.3-8.el7_8.1.noarch                                                          8/41 
  Updating   : python-firewall-0.6.3-8.el7_8.1.noarch                                                               9/41 
  Updating   : kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64                                                       10/41 
  Updating   : kernel-tools-3.10.0-1127.10.1.el7.x86_64                                                            11/41 
  Updating   : firewalld-0.6.3-8.el7_8.1.noarch                                                                    12/41 
  Updating   : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                    13/41 
  Updating   : systemd-sysv-219-73.el7_8.6.x86_64                                                                  14/41 
  Updating   : tzdata-2020a-1.el7.noarch                                                                           15/41 
  Updating   : 32:bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64                                                     16/41 
  Updating   : iprutils-2.4.17.1-3.el7_7.x86_64                                                                    17/41 
  Updating   : binutils-2.27-43.base.el7_8.1.x86_64                                                                18/41 
  Updating   : yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch                                                     19/41 
  Installing : kernel-3.10.0-1127.10.1.el7.x86_64                                                                  20/41 
  Updating   : python-perf-3.10.0-1127.10.1.el7.x86_64                                                             21/41 
  Cleanup    : 7:lvm2-2.02.186-7.el7.x86_64                                                                        22/41 
  Cleanup    : firewalld-0.6.3-8.el7.noarch                                                                        23/41 
  Cleanup    : systemd-sysv-219-73.el7.1.x86_64                                                                    24/41 
  Cleanup    : firewalld-filesystem-0.6.3-8.el7.noarch                                                             25/41 
  Cleanup    : python-firewall-0.6.3-8.el7.noarch                                                                  26/41 
  Cleanup    : tzdata-2019c-1.el7.noarch                                                                           27/41 
  Cleanup    : yum-plugin-fastestmirror-1.1.31-53.el7.noarch                                                       28/41 
  Cleanup    : 7:lvm2-libs-2.02.186-7.el7.x86_64                                                                   29/41 
  Cleanup    : 7:device-mapper-event-1.02.164-7.el7.x86_64                                                         30/41 
  Cleanup    : 7:device-mapper-event-libs-1.02.164-7.el7.x86_64                                                    31/41 
  Cleanup    : 7:device-mapper-1.02.164-7.el7.x86_64                                                               32/41 
  Cleanup    : 7:device-mapper-libs-1.02.164-7.el7.x86_64                                                          33/41 
  Cleanup    : systemd-219-73.el7.1.x86_64                                                                         34/41 
  Cleanup    : kernel-tools-3.10.0-1127.el7.x86_64                                                                 35/41 
  Cleanup    : kernel-tools-libs-3.10.0-1127.el7.x86_64                                                            36/41 
  Cleanup    : systemd-libs-219-73.el7.1.x86_64                                                                    37/41 
  Cleanup    : 32:bind-export-libs-9.11.4-16.P2.el7.x86_64                                                         38/41 
  Cleanup    : iprutils-2.4.17.1-3.el7.x86_64                                                                      39/41 
  Cleanup    : binutils-2.27-43.base.el7.x86_64                                                                    40/41 
  Cleanup    : python-perf-3.10.0-1127.el7.x86_64                                                                  41/41 
  Verifying  : python-perf-3.10.0-1127.10.1.el7.x86_64                                                              1/41 
  Verifying  : kernel-3.10.0-1127.10.1.el7.x86_64                                                                   2/41 
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7_8.2.x86_64                                                       3/41 
  Verifying  : 7:lvm2-2.02.186-7.el7_8.2.x86_64                                                                     4/41 
  Verifying  : yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch                                                      5/41 
  Verifying  : 7:device-mapper-1.02.164-7.el7_8.2.x86_64                                                            6/41 
  Verifying  : systemd-sysv-219-73.el7_8.6.x86_64                                                                   7/41 
  Verifying  : 7:device-mapper-event-1.02.164-7.el7_8.2.x86_64                                                      8/41 
  Verifying  : kernel-tools-libs-3.10.0-1127.10.1.el7.x86_64                                                        9/41 
  Verifying  : binutils-2.27-43.base.el7_8.1.x86_64                                                                10/41 
  Verifying  : systemd-219-73.el7_8.6.x86_64                                                                       11/41 
  Verifying  : 7:lvm2-libs-2.02.186-7.el7_8.2.x86_64                                                               12/41 
  Verifying  : firewalld-0.6.3-8.el7_8.1.noarch                                                                    13/41 
  Verifying  : iprutils-2.4.17.1-3.el7_7.x86_64                                                                    14/41 
  Verifying  : systemd-libs-219-73.el7_8.6.x86_64                                                                  15/41 
  Verifying  : kernel-tools-3.10.0-1127.10.1.el7.x86_64                                                            16/41 
  Verifying  : 32:bind-export-libs-9.11.4-16.P2.el7_8.6.x86_64                                                     17/41 
  Verifying  : tzdata-2020a-1.el7.noarch                                                                           18/41 
  Verifying  : python-firewall-0.6.3-8.el7_8.1.noarch                                                              19/41 
  Verifying  : firewalld-filesystem-0.6.3-8.el7_8.1.noarch                                                         20/41 
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7_8.2.x86_64                                                21/41 
  Verifying  : tzdata-2019c-1.el7.noarch                                                                           22/41 
  Verifying  : firewalld-0.6.3-8.el7.noarch                                                                        23/41 
  Verifying  : 7:device-mapper-event-1.02.164-7.el7.x86_64                                                         24/41 
  Verifying  : 32:bind-export-libs-9.11.4-16.P2.el7.x86_64                                                         25/41 
  Verifying  : 7:device-mapper-event-libs-1.02.164-7.el7.x86_64                                                    26/41 
  Verifying  : 7:lvm2-2.02.186-7.el7.x86_64                                                                        27/41 
  Verifying  : systemd-219-73.el7.1.x86_64                                                                         28/41 
  Verifying  : systemd-sysv-219-73.el7.1.x86_64                                                                    29/41 
  Verifying  : firewalld-filesystem-0.6.3-8.el7.noarch                                                             30/41 
  Verifying  : systemd-libs-219-73.el7.1.x86_64                                                                    31/41 
  Verifying  : python-firewall-0.6.3-8.el7.noarch                                                                  32/41 
  Verifying  : 7:device-mapper-1.02.164-7.el7.x86_64                                                               33/41 
  Verifying  : 7:device-mapper-libs-1.02.164-7.el7.x86_64                                                          34/41 
  Verifying  : binutils-2.27-43.base.el7.x86_64                                                                    35/41 
  Verifying  : kernel-tools-3.10.0-1127.el7.x86_64                                                                 36/41 
  Verifying  : 7:lvm2-libs-2.02.186-7.el7.x86_64                                                                   37/41 
  Verifying  : kernel-tools-libs-3.10.0-1127.el7.x86_64                                                            38/41 
  Verifying  : iprutils-2.4.17.1-3.el7.x86_64                                                                      39/41 
  Verifying  : python-perf-3.10.0-1127.el7.x86_64                                                                  40/41 
  Verifying  : yum-plugin-fastestmirror-1.1.31-53.el7.noarch                                                       41/41 

Installed:
  kernel.x86_64 0:3.10.0-1127.10.1.el7

Updated:
  bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6              binutils.x86_64 0:2.27-43.base.el7_8.1
  device-mapper.x86_64 7:1.02.164-7.el7_8.2                    device-mapper-event.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2         device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2
  firewalld.noarch 0:0.6.3-8.el7_8.1                           firewalld-filesystem.noarch 0:0.6.3-8.el7_8.1
  iprutils.x86_64 0:2.4.17.1-3.el7_7                           kernel-tools.x86_64 0:3.10.0-1127.10.1.el7
  kernel-tools-libs.x86_64 0:3.10.0-1127.10.1.el7              lvm2.x86_64 7:2.02.186-7.el7_8.2
  lvm2-libs.x86_64 7:2.02.186-7.el7_8.2                        python-firewall.noarch 0:0.6.3-8.el7_8.1
  python-perf.x86_64 0:3.10.0-1127.10.1.el7                    systemd.x86_64 0:219-73.el7_8.6
  systemd-libs.x86_64 0:219-73.el7_8.6                         systemd-sysv.x86_64 0:219-73.el7_8.6
  tzdata.noarch 0:2020a-1.el7                                  yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8

Complete!
[root@X11TESTSERVER ~]# wget http://ftp.usf.edu/pub/centos/7.8.2003/updates/x86_64/Packages/microcode_ctl-2.1-61.6.el7_8.x86_64.rpm
-bash: wget: command not found
[root@X11TESTSERVER ~]# yum install wget -y
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.teklinks.com
 * updates: mirror.mojohost.com
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.14-18.el7_6.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================= Package                  Arch                       Version                              Repository                Size 
=========================================================================================================================Installing:
 wget                     x86_64                     1.14-18.el7_6.1                      base                     547 k 

Transaction Summary
=========================================================================================================================Install  1 Package

Total download size: 547 k
Installed size: 2.0 M
Downloading packages:
wget-1.14-18.el7_6.1.x86_64.rpm                                                                   | 547 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wget-1.14-18.el7_6.1.x86_64                                                                           1/1 
  Verifying  : wget-1.14-18.el7_6.1.x86_64                                                                           1/1 

Installed:
  wget.x86_64 0:1.14-18.el7_6.1

Complete!
[root@X11TESTSERVER ~]# wget http://ftp.usf.edu/pub/centos/7.8.2003/updates/x86_64/Packages/microcode_ctl-2.1-61.6.el7_8.x86_64.rpm
--2020-06-14 23:54:53--  http://ftp.usf.edu/pub/centos/7.8.2003/updates/x86_64/Packages/microcode_ctl-2.1-61.6.el7_8.x86_64.rpm
Resolving ftp.usf.edu (ftp.usf.edu)... 131.247.253.85, 2607:fe50:0:f30b::85
Connecting to ftp.usf.edu (ftp.usf.edu)|131.247.253.85|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2720752 (2.6M) [application/x-rpm]
Saving to: 'microcode_ctl-2.1-61.6.el7_8.x86_64.rpm'

100%[===============================================================================>] 2,720,752   6.19MB/s   in 0.4s   

2020-06-14 23:54:53 (6.19 MB/s) - 'microcode_ctl-2.1-61.6.el7_8.x86_64.rpm' saved [2720752/2720752]

[root@X11TESTSERVER ~]# rpm -U --noscripts microcode_ctl-2.1-61.6.el7_8.x86_64.rpm 
[root@X11TESTSERVER ~]# reboot now
Connection to 23.111.168.58 closed by remote host.
Connection to 23.111.168.58 closed.
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ping 23.111.168.58
PING 23.111.168.58 (23.111.168.58) 56(84) bytes of data.
64 bytes from 23.111.168.58: icmp_seq=484 ttl=56 time=2.11 ms
64 bytes from 23.111.168.58: icmp_seq=485 ttl=56 time=1.58 ms
64 bytes from 23.111.168.58: icmp_seq=486 ttl=56 time=1.69 ms
^C
--- 23.111.168.58 ping statistics ---
486 packets transmitted, 3 received, 99% packet loss, time 485533ms
rtt min/avg/max/mdev = 1.584/1.796/2.112/0.232 ms
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ssh root@23.111.168.58
root@23.111.168.58's password: 
Last failed login: Mon Jun 15 00:03:22 EDT 2020 from 80.ip-51-178-82.eu on ssh:notty
There were 2 failed login attempts since the last successful login.
Last login: Sun Jun 14 23:51:11 2020 from 68-233-224-175.static.hvvc.us
[root@X11TESTSERVER ~]# yum remove microcode_ctl
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package microcode_ctl.x86_64 2:2.1-61.6.el7_8 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================= Package                       Arch                   Version                            Repository                 Size 
=========================================================================================================================Removing:
 microcode_ctl                 x86_64                 2:2.1-61.6.el7_8                   installed                 3.2 M 

Transaction Summary
=========================================================================================================================Remove  1 Package

Installed size: 3.2 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Erasing    : 2:microcode_ctl-2.1-61.6.el7_8.x86_64                                                                 1/1 
  Verifying  : 2:microcode_ctl-2.1-61.6.el7_8.x86_64                                                                 1/1 

Removed:
  microcode_ctl.x86_64 2:2.1-61.6.el7_8

Complete!
[root@X11TESTSERVER ~]# yum install microcode_ctl-2.1-61.el7
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.teklinks.com
 * updates: mirror.mojohost.com
Resolving Dependencies
--> Running transaction check
---> Package microcode_ctl.x86_64 2:2.1-61.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================= Package                         Arch                     Version                           Repository              Size 
=========================================================================================================================Installing:
 microcode_ctl                   x86_64                   2:2.1-61.el7                      base                   2.5 M 

Transaction Summary
=========================================================================================================================Install  1 Package

Total download size: 2.5 M
Installed size: 3.1 M
Is this ok [y/d/N]: y
Downloading packages:
microcode_ctl-2.1-61.el7.x86_64.rpm                                                               | 2.5 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 2:microcode_ctl-2.1-61.el7.x86_64                                                                     1/1 

  Verifying  : 2:microcode_ctl-2.1-61.el7.x86_64                                                                     1/1 

Installed:
  microcode_ctl.x86_64 2:2.1-61.el7

Complete!
[root@X11TESTSERVER ~]# 
[root@X11TESTSERVER ~]# reboot now
Connection to 23.111.168.58 closed by remote host.
Connection to 23.111.168.58 closed.
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ping 23.111.168.58
PING 23.111.168.58 (23.111.168.58) 56(84) bytes of data.
64 bytes from 23.111.168.58: icmp_seq=74 ttl=56 time=1.68 ms
64 bytes from 23.111.168.58: icmp_seq=75 ttl=56 time=1.76 ms
^C
--- 23.111.168.58 ping statistics ---
75 packets transmitted, 2 received, 97% packet loss, time 74083ms
rtt min/avg/max/mdev = 1.688/1.724/1.760/0.036 ms
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ssh root@23.111.168.58
root@23.111.168.58's password: 
Last failed login: Mon Jun 15 00:04:09 EDT 2020 from 106.13.203.171 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Mon Jun 15 00:03:28 2020 from 68-233-224-175.static.hvvc.us
[root@X11TESTSERVER ~]# rpm -qa | grep microcode_ctl
microcode_ctl-2.1-61.el7.x86_64
[root@X11TESTSERVER ~]# reboot now
Connection to 23.111.168.58 closed by remote host.
Connection to 23.111.168.58 closed.
dbennett@DonaldB-Work /mnt/c/Users/dbennett $ ping 23.111.168.58
PING 23.111.168.58 (23.111.168.58) 56(84) bytes of data.
64 bytes from 23.111.168.58: icmp_seq=71 ttl=56 time=1.93 ms
64 bytes from 23.111.168.58: icmp_seq=72 ttl=56 time=1.79 ms
64 bytes from 23.111.168.58: icmp_seq=73 ttl=56 time=1.62 ms
64 bytes from 23.111.168.58: icmp_seq=74 ttl=56 time=1.27 ms
^C
--- 23.111.168.58 ping statistics ---
74 packets transmitted, 4 received, 94% packet loss, time 73075ms
rtt min/avg/max/mdev = 1.277/1.659/1.939/0.249 ms
history2.txt (31,261 bytes)
nccsoft

nccsoft

2020-06-15 06:12

reporter   ~0037105

My server also hangs at https://screencaps.donaldbennett.net/jp2launcher_KvC8sTt5Xg.png
login have abrt

[root@soft1 /]# abrt-cli list --since 1592113010
id 839c14b7a66fda558af5f35505120a572883e911
reason: Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
time: 西元2020年06月15日 (週一) 08時56分31秒
uid: 0 (root)
count: 1
Directory: /var/spool/abrt/vmcore-127.0.0.1-2020-06-14-07:45:46

I don't know how to fix
nccsoft

nccsoft

2020-06-16 00:51

reporter   ~0037116

I followed your steps and cannot install microcode_ctl-2.1-61.el7

[root@soft1 tmp]# yum install microcode_ctl-2.1-61.el7
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: free.nchc.org.tw
 * epel: mirrors.tuna.tsinghua.edu.cn
 * extras: free.nchc.org.tw
 * remi-php72: mirrors.tuna.tsinghua.edu.cn
 * remi-safe: mirrors.tuna.tsinghua.edu.cn
 * updates: free.nchc.org.tw
No package microcode_ctl-2.1-61.el7 available.
Error: Nothing to do
nccsoft

nccsoft

2020-06-16 00:58

reporter   ~0037117

Sorry, I found that I added exclude=microcode_ctl to yum.conf, so I cannot install
nccsoft

nccsoft

2020-06-16 04:33

reporter   ~0037118

It's normal now, and this problem remains

reason: Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
__david__

__david__

2020-06-17 16:29

reporter   ~0037146

I had this same issue yesterday on CentOS 7.7. Everything was going well, until the kernel panic. It said that it was going to reboot after 30 seconds. I waited at least several minutes, but my netbook doesn't display disk activity. I finally shut down the laptop since it appeared to be dead. I now have the symptoms described above, with duplicate packages. I need to figure out how to recover from this, or possibly use this as an opportunity to migrate to Stream 8.

Is there an issue with the microcode_ctl package? Will it only affect CentOS and not the production RHEL servers?
TrevorH

TrevorH

2020-06-17 16:42

manager   ~0037147

Since CentOS is a rebuild of the source used to create RHEL I see no reason why the bug should not exist there too.

Intel published a new microcode tarball yesterday and it is noticeable that the 3 changed files in that vs the previous one, all are downgrades from version 0xdc to 0xd6 (now marked as "new") so this would appear to be an Intel bug not a Red Hat one. Presumably RH will pick that up in due course and there will be yet another microcode_ctl package released.
toracat

toracat

2020-06-17 17:33

manager   ~0037148

relevant RHBZ is here:

https://bugzilla.redhat.com/show_bug.cgi?id=1846133

which refers to:

https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/issues/31
aj_potc

aj_potc

2020-06-18 13:47

reporter   ~0037159

My Skylake-based systems were also affected. One was able to reboot by itself (I'm not sure why), while the other froze and required manual intervention (editing the Grub boot parameters). I see both systems now have two microcode_ctl packages installed, as reported by others:

microcode_ctl-2.1-61.el7
microcode_ctl-2.1-61.6.el7_8

When the the new microcode_ctl package is released and pushed to repos, will yum be able to clean up this problem? Or should I manually remove one or both of the already installed packages, as dbennet did above?

Thanks for any tips.
TrevorH

TrevorH

2020-06-18 14:18

manager   ~0037160

You'll almost certainly need to clean up duplicate packages yourself and I suspect there will be more than just that one.

https://wiki.centos.org/HowTos/CleaningUpDuplicatePackages might help but don't forget the backups!
nccsoft

nccsoft

2020-06-19 01:20

reporter   ~0037164

aj_potc
If there are more than one duplicate packages, you should pay attention, because the duplicate packages configuration file will become 0byte, I am the victim, and I have updated php selinux bind... At the same time, these packages are duplicated, and the configuration file has become 0byte
ManuelWolfshant

ManuelWolfshant

2020-06-19 20:54

manager   ~0037185

@nccsoft : https://access.redhat.com/solutions/3924551
__david__

__david__

2020-06-19 21:02

reporter   ~0037186

That Red Hat solution is a great link, and it's what I used to resolve my issue. I almost posted it, but ultimately decided against it because it wasn't available to the general public.
The one thing it didn't do is get my GNOME desktop working again. I never get to the login prompt, so I have to press Ctrl-Alt F2, and log in at the command line. Oh, well, maybe this is a good excuse to ry CentOS Stream 8. Anyone have some ideas to get my desktop back?
ManuelWolfshant

ManuelWolfshant

2020-06-19 21:10

manager   ~0037188

@__david__ : you could try removing the configuration files and/or reinstalling the DE. assuming the logs do not help, that is.
aj_potc

aj_potc

2020-06-20 11:28

reporter   ~0037190

Just an update on my attempt to clean up this mess on my two servers:

In my case, the only duplicate package on my systems was microcode_ctl. I attempted to use the link posted by @TrevorH, which suggests running the following command to remove duplicates:
package-cleanup --cleandupes --removenewestdupes

This appeared to do what it should by removing the latest, incompletely installed microcode_ctl version. Next, I did the following:

1. Rebooted the systems. One system rebooted fine, while the other hung. (No idea why -- they are identical.)
2. On the hung system, edited the Grub boot command to include dis_ucode_ldr in the boot arguments. System booted normally.
3. Ran "yum reinstall microcode_ctl-2.1-61.el7". (A regular "yum install" would not work here, as yum reported that the package was still installed. I though the package-cleanup would have resolved that, but apparently not.)
4. After a reboot, both systems now appear to boot properly.
5. Running "grep microcode /proc/cpuinfo" reveals that my systems are now running microcode revision 0xd6 -- for better or for worse.

At this point, I'm not sure whether I should block all future microcode_ctl updates, or pray that the future ones will be better tested.
ManuelWolfshant

ManuelWolfshant

2020-06-20 15:59

manager   ~0037191

@aj_potc: if you are absolutely 1000% sure that there are no more bugs in your CPUs that would require new versions of microcode, blocking future updates is perfectly fine. As of praying... it depends to whom you pray. If you pray to Intel, it might be useful. If you pray to anyone else.... doubtful.
__david__

__david__

2020-06-20 16:17

reporter   ~0037192

I have not tried this, but if you are using lvm, you can make a snapshot. If the update goes bad, you can roll back the lvm snapshot.
__david__

__david__

2020-06-20 16:21

reporter   ~0037193

@ManuelWolfshant Thank you for the suggestion. I tried a yum groupremove "GNOME Desktop", and it removed only four files. I may try a yum reinstall, and see if that works.
ManuelWolfshant

ManuelWolfshant

2020-06-20 16:33

manager   ~0037194

Last edited: 2020-06-20 16:36

View 2 revisions

@__david__ : I am dead sure that the group contains way more than just 4 packages
LE: just tested "yum group install "Gnome Desktop" on a machine which had no DE installed:

Transaction Summary
===============================================================================================================================================================
Install 292 Packages (+756 Dependent packages)

aj_potc

aj_potc

2020-06-20 17:04

reporter   ~0037195

@ManuelWolfshant In this usage, the word "pray" is a synonym for "hope," as that's all we can do.
At this point the only thing I can be sure of is that Intel's poorly-tested microcode mitigations have caused more harm than the bugs they're meant to fix.
nccsoft

nccsoft

2020-06-23 08:12

reporter   ~0037212

@ManuelWolfshant Thank you, I found the way you said on the Internet, but that can only help me delete duplicate packages, and cannot rescue my configuration file, because after installing microcode_ctl 2.1-61.6.el7-8, many of my configuration files have been Cleared to 0byte, but it is not important now, because I have restored it from my backup
dbennett

dbennett

2020-06-26 16:27

reporter   ~0037247

Latest update from Redhat: https://access.redhat.com/errata/RHEA-2020:2743

This looks to be resolved from Upstream, with RPM: microcode_ctl-2.1-61.10.el7_8.x86_64.rpm which replaces older RPM

https://screencaps.donaldbennett.net/jp2launcher_mb3eOYwA8o.png << Successful install with no reboot, running a reboot now after updates and OS boots just fine now with no issues or lock ups.
ManuelWolfshant

ManuelWolfshant

2020-06-26 16:34

manager   ~0037248

Fixed by microcode_ctl-2.1-61.10.el7_8.x86_64.rpm

Issue History

Date Modified Username Field Change
2020-06-12 08:27 dbennett New Issue
2020-06-12 08:38 tru Note Added: 0037083
2020-06-12 08:39 dbennett Note Added: 0037084
2020-06-12 10:25 TrevorH Note Added: 0037085
2020-06-12 10:42 tinpotnick Note Added: 0037086
2020-06-12 11:22 tinpotnick Note Added: 0037087
2020-06-12 12:03 tru Note Added: 0037088
2020-06-12 12:20 TrevorH Note Added: 0037089
2020-06-12 12:42 ManuelWolfshant Note Added: 0037090
2020-06-12 16:34 dbennett Note Added: 0037091
2020-06-13 11:47 ManuelWolfshant Note Added: 0037092
2020-06-14 07:08 nccsoft Note Added: 0037096
2020-06-14 16:32 robetek Note Added: 0037098
2020-06-15 02:43 nccsoft Note Added: 0037102
2020-06-15 03:27 dbennett File Added: history.txt
2020-06-15 03:27 dbennett Note Added: 0037103
2020-06-15 04:11 dbennett File Added: history2.txt
2020-06-15 04:11 dbennett Note Added: 0037104
2020-06-15 06:12 nccsoft Note Added: 0037105
2020-06-16 00:51 nccsoft Note Added: 0037116
2020-06-16 00:58 nccsoft Note Added: 0037117
2020-06-16 04:33 nccsoft Note Added: 0037118
2020-06-17 16:29 __david__ Note Added: 0037146
2020-06-17 16:42 TrevorH Note Added: 0037147
2020-06-17 17:33 toracat Note Added: 0037148
2020-06-17 18:43 toracat Status new => confirmed
2020-06-18 13:47 aj_potc Note Added: 0037159
2020-06-18 14:18 TrevorH Note Added: 0037160
2020-06-18 18:17 toracat Relationship added has duplicate 0017484
2020-06-19 01:20 nccsoft Note Added: 0037164
2020-06-19 20:54 ManuelWolfshant Note Added: 0037185
2020-06-19 21:02 __david__ Note Added: 0037186
2020-06-19 21:10 ManuelWolfshant Note Added: 0037188
2020-06-20 11:28 aj_potc Note Added: 0037190
2020-06-20 15:59 ManuelWolfshant Note Added: 0037191
2020-06-20 16:17 __david__ Note Added: 0037192
2020-06-20 16:21 __david__ Note Added: 0037193
2020-06-20 16:33 ManuelWolfshant Note Added: 0037194
2020-06-20 16:36 ManuelWolfshant Note Edited: 0037194 View Revisions
2020-06-20 17:04 aj_potc Note Added: 0037195
2020-06-23 08:12 nccsoft Note Added: 0037212
2020-06-26 16:27 dbennett Note Added: 0037247
2020-06-26 16:34 ManuelWolfshant Status confirmed => closed
2020-06-26 16:34 ManuelWolfshant Resolution open => fixed
2020-06-26 16:34 ManuelWolfshant Fixed in Version => 7.8-2003
2020-06-26 16:34 ManuelWolfshant Note Added: 0037248
2020-06-28 16:50 toracat Status closed => resolved