View Issue Details

IDProjectCategoryView StatusLast Update
0014309CentOS-7yumpublic2017-12-26 16:47
Reporterbruno-garcia 
PrioritynormalSeverityblockReproducibilityalways
Status newResolutionopen 
Platformx64OSCentOSOS Version7
Product Version7.2.1511 
Target VersionFixed in Version 
Summary0014309: yum update selinux deadlocks
DescriptionEverytime I try running yum update the machine locks up while taking 100% disk I/O. I narrowed it down to updating: 'selinux*'

I found similar issue at: https://www.centos.org/forums/viewtopic.php?t=60087

The VM is running under hyper-v and I can't see how there's overcommitment of the host since there's 15GB free RAM on the host. The disk (RAID 10) can be very slow on writing small chunks so perhaps its related to that?
Steps To Reproduceyum update selinux*
Additional Informationcat /proc/cpuinfo
<code>

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
stepping : 9
microcode : 0xffffffff
cpu MHz : 3252.088
cache size : 8192 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase smep erms xsaveopt
bogomips : 6504.17
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 58
model name : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
stepping : 9
microcode : 0xffffffff
cpu MHz : 3252.088
cache size : 8192 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase smep erms xsaveopt
bogomips : 6504.17
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

<code>

<code>
# hostnamectl
   Static hostname: sydney.lan
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 80639cf4faad403fb0724b9513e5f0e9
           Boot ID: 30bc56423f7d43c18b7bce722df4b7b1
    Virtualization: microsoft
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-514.21.1.el7.x86_64
      Architecture: x86-64
<code>

Failing operation:
<code>
# yum update selinux*

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

* base: mirror.easyname.at

* epel: mirror.inode.at

* extras: mirror.easyname.at

* updates: mirror.easyname.at

Resolving Dependencies

--> Running transaction check

---> Package selinux-policy.noarch 0:3.13.1-102.el7_3.16 will be updated

---> Package selinux-policy.noarch 0:3.13.1-166.el7_4.7 will be an update

---> Package selinux-policy-targeted.noarch 0:3.13.1-102.el7_3.16 will be updated

---> Package selinux-policy-targeted.noarch 0:3.13.1-166.el7_4.7 will be an update

--> Finished Dependency Resolution

  

Dependencies Resolved

  

================================================================================

Package Arch Version Repository Size

================================================================================

Updating:

selinux-policy noarch 3.13.1-166.el7_4.7 updates 437 k

selinux-policy-targeted noarch 3.13.1-166.el7_4.7 updates 6.5 M

  

Transaction Summary

================================================================================

Upgrade 2 Packages

  

Total download size: 6.9 M

Is this ok [y/d/N]: y

Downloading packages:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.

(1/2): selinux-policy-3.13.1-166.el7_4.7.noarch.rpm | 437 kB 00:00

(2/2): selinux-policy-targeted-3.13.1-166.el7_4.7.noarch.r | 6.5 MB 00:00

--------------------------------------------------------------------------------

Total 15 MB/s | 6.9 MB 00:00

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Updating : selinux-policy-3.13.1-166.el7_4.7.noarch 1/4

Full path required for exclude: net:[4026531950].

Full path required for exclude: net:[4026532171].
 

Message from syslogd@sydney at Dec 26 14:10:37 ...

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [s6-svscan:2368]
   

Message from syslogd@sydney at Dec 26 14:11:05 ...

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [s6-svscan:2368]
  

Message from syslogd@sydney at Dec 26 14:11:37 ...

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [s6-svscan:2368]


Message from syslogd@sydney at Dec 26 14:12:05 ...

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [s6-svscan:2368]
  

Message from syslogd@sydney at Dec 26 14:12:33 ...

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [s6-svscan:2368]

  
Message from syslogd@sydney at Dec 26 14:13:01 ...

kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [s6-svscan:2368]
<code>
TagsNo tags attached.
abrt_hash
URL

Activities

TrevorH

TrevorH

2017-12-26 15:55

manager   ~0030823

Updating selinux packages requires a policy rebuild which can use a large quantity of RAM - like 400MB+ extra. If you are RAM constrained and that forces it to use swap and your underlying disk subsystem is slow then you can expect to get that sort of message. The answer is to give the guest more RAM while doing the update to avoid the swap usage.
bruno-garcia

bruno-garcia

2017-12-26 16:34

reporter   ~0030824

I've added 6 GB of RAM without the dynamic RAM feature of Hyper-V and 8 Cores but the problem persists.
The Disk I/O stays around 7MB/s on writes and the screen has lots of:

...
[ 368,379018] audit: backlog limit exceeded

And the "CPU#0 stuck for 23s!" message
bruno-garcia

bruno-garcia

2017-12-26 16:47

reporter   ~0030825

I stopped docker before running 'yum update selinux*' and the update went through really quickly.

Issue History

Date Modified Username Field Change
2017-12-26 13:30 bruno-garcia New Issue
2017-12-26 15:55 TrevorH Note Added: 0030823
2017-12-26 16:34 bruno-garcia Note Added: 0030824
2017-12-26 16:47 bruno-garcia Note Added: 0030825