View Issue Details

IDProjectCategoryView StatusLast Update
0015589CentOS-7-OTHERpublic2019-01-18 23:58
Reporterdowdle 
PrioritynormalSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Platformx86_64OS7OS Version7.6.1810
Product Version 
Target VersionFixed in Version 
Summary0015589: Please update livecd-tools to 26.1 - livecd-tools-21.4-2.el7 from 2015 is in Extras and has issues
DescriptionSince the release of 7.6.1810, I've started having trouble building personal remix live media with livecd-creator. It seems to be related to this upstream bug (upstream being Fedora in this case):
https://bugzilla.redhat.com/show_bug.cgi?id=1658359
... with this fix:
https://github.com/livecd-tools/livecd-tools/commit/c6dd265af7ccb3c51c37dcd89839f178832518ea

Since Fedora has a much newer version of livecd-tools backporting the patch would require a bit of work.

According to one of the Fedora maintainers (see rhb mentioned above):

"As of EL7.6, it is now possible to run the latest version of livecd-tools on EL7. CentOS was planning on switching to livecd-tools v26, but this issue (which is now fixed in v26.1) came a bit late for the 7.6 GA. I made the spec fixes so that they could backport straight from Fedora Dist-Git to CentOS Extras.

I'm hoping that Fabian Arrotin and Pablo Greco will ship v26.1 soon in their Extras (which I believe Scientific Linux and friends get it from...)."
Steps To ReproduceUse a kickstart and built an iso with livecd-creator. The iso will build but can't login. See this video for an example of the behavior:
https://www.montanalinux.org/files/videos/fedora29-centos76-livemedia-issue-20181205.mp4

Here's a a frame from the video that shows the error when trying to login from the console:
https://www.montanalinux.org/files/videos/fedora29-centos76-livemedia-issue-20181205-frame.png
Error text: "-- root: no shell: Permission denied"
Additional InformationIf I boot my CentOS7 personal remix builds adding the following to the kernel boot line "selinux=1 enforcing=0" then my media works just fine... so it appears to be an SELinux issue.
Tagslivecd-tools
abrt_hash
URL

Activities

pgreco

pgreco

2018-12-15 11:03

developer   ~0033350

@dowdle, we're working on this, and will be available soon.
In the meantime, you can get the latest version we have built publicly (v26.0) and all its dependencies from https://buildlogs.centos.org/c7-extras.x86_64/.
Once you've done that, you'll need to apply this patch https://github.com/livecd-tools/livecd-tools/commit/c6dd265af7ccb3c51c37dcd89839f178832518ea

Please let us know how it went, so we can push the real version faster.

Thanks.
Pablo.
dowdle

dowdle

2018-12-17 16:27

reporter   ~0033372

I gave it a try this weekend on 5 different systems (1 laptop, two VMs on one host, and two VMs on another host) and was only able to make working media from one of them. I also noticed that the working media, while it allowed the liveuser to login and to the install... the post-install system booted very slowly with a few service start errors... which cleared themselves up after an selinxu relabel. So, as far as I can tell, the 26.0 package with the patch manually applied isn't working on CentOS 7.6 as it is on Fedora 28 and Fedora 29.

I emailed the patch author directly with the information.
dowdle

dowdle

2018-12-29 03:07

reporter   ~0033470

Any news on this?
dowdle

dowdle

2019-01-16 16:24

reporter   ~0033605

I've done some more testing. The remixes I was doing included EPEL, rpmfusion and a few third-party packages in a local repository... so I thought it would be important to test building the stock media using the stock kickstart files.

To the best of my knowledge, the CentOS project used these kickstarts (although the files end in .cfg rather than .ks) to build the GNOME Live and KDE Live media:
https://github.com/CentOS/sig-core-livemedia/tree/master/kickstarts

Specifically I used this file for the GNOME Live build:
https://raw.githubusercontent.com/CentOS/sig-core-livemedia/master/kickstarts/centos-7-live-gnome.cfg

...and this file for the KDE Live build:
https://raw.githubusercontent.com/CentOS/sig-core-livemedia/master/kickstarts/centos-7-live-kde.cfg

Within both of those kickstart files the repository to use is listed as:
http://qa.centos.org/centos/7/os/x86_64/

Since I could not access that host, I had to update the repository definition(s) in both kickstarts. Since part of my desire to do rebuilds in the first place is to include all of the updates at the time of build (so less post-install updating is needed), I added both the base and updates repositories. I ended up doing four builds total... two for GNOME (one with updates included and one without) and two for KDE (one with updates included and one without).

I tested the resulting media (they all built fine) and the without-updates media worked but those built with the updates failed. How did they fail? Both the GNOME and the KDE media refused to provide a graphical login... although after waiting a while for it to finish booting and it being stuck at a text output screen without providing the graphical login screen... I was able to Alt+F2 over to another console to get a text-based login. I tried to login as both the root user and the liveuser and both failed immediately without prompting for a password... which was the same failure I was having with my own extra-repos-added builds. Just to clarify, at least my builds offered a graphical login screen whereas these did not... although I haven't done one of my builds recently and it is possible they have since become more broken with additional updates.

In any event, I'm still baffled by what the specific cause of the problem is and how to best troubleshoot it. I forgot to verify that the stock media builds could be made to work with the workaround I used before of putting SELinux into permissive mode by passing additional kernel flags from the bootloader. When I get a chance I'll give that a try. I'll also attempt to build my with-additional-repos-added remixes without updates and see if I get similar results to what I got with the stock media.

If someone with additional knowledge of what's going on, how stuff works under the hood, and how to troubleshoot what's going on... could assist me in gathering more useful information toward getting the problem overcome, I'd greatly appreciate it. I also think CentOS will need to address this or their future live media builds are at risk. I know some of the CentOS devs/packagers aren't very fond of the live media but I find it extremely useful, especially for desktop oriented systems.
dowdle

dowdle

2019-01-16 16:27

reporter   ~0033606

Oh, to clarify... I am doing all of these live media builds on a fully updated CentOS 7.6 host with the latest not-pushed-to-stable build of livecd-tools / livecd-creator... with the patch from the next upstream livecd-tools release manually applied as requested earlier by the CentOS devs who have responded to this ticket.
dowdle

dowdle

2019-01-17 00:57

reporter   ~0033612

I built two of my remixes with EPEL, rpmfusion and a few other third-party packages from a "local" repo... BUT without CentOS updates... and my media boots, auto-logins, installs and works post-install just fine.

I conclude from that that something present in the updates post GA is breaking it. Not sure what yet.

Anyone?

Here's a list of updates needed by my minimal build:

--- >8 START HERE 8< ---
=====================================================================================================
 Package Arch Version Repository Size
=====================================================================================================
Installing:
 kernel x86_64 3.10.0-957.1.3.el7 updates 48 M
Updating:
 NetworkManager x86_64 1:1.12.0-8.el7_6 updates 1.7 M
 NetworkManager-glib x86_64 1:1.12.0-8.el7_6 updates 1.2 M
 NetworkManager-libnm x86_64 1:1.12.0-8.el7_6 updates 1.4 M
 NetworkManager-team x86_64 1:1.12.0-8.el7_6 updates 159 k
 NetworkManager-tui x86_64 1:1.12.0-8.el7_6 updates 239 k
 NetworkManager-wifi x86_64 1:1.12.0-8.el7_6 updates 187 k
 cronie x86_64 1.4.11-20.el7_6 updates 91 k
 cronie-anacron x86_64 1.4.11-20.el7_6 updates 36 k
 device-mapper x86_64 7:1.02.149-10.el7_6.2 updates 292 k
 device-mapper-event x86_64 7:1.02.149-10.el7_6.2 updates 188 k
 device-mapper-event-libs x86_64 7:1.02.149-10.el7_6.2 updates 187 k
 device-mapper-libs x86_64 7:1.02.149-10.el7_6.2 updates 320 k
 elinks x86_64 0.12-0.37.pre6.el7.0.1 updates 882 k
 firefox x86_64 60.4.0-1.el7.centos updates 91 M
 gdm x86_64 1:3.28.2-10.el7 updates 521 k
 ghostscript x86_64 9.07-31.el7_6.6 updates 4.3 M
 ghostscript-cups x86_64 9.07-31.el7_6.6 updates 59 k
 gnome-session x86_64 3.28.1-6.el7 updates 401 k
 ipset x86_64 6.38-3.el7_6 updates 43 k
 ipset-libs x86_64 6.38-3.el7_6 updates 50 k
 java-1.8.0-openjdk-headless x86_64 1:1.8.0.191.b12-1.el7_6 updates 32 M
 kernel-headers x86_64 3.10.0-957.1.3.el7 updates 8.0 M
 kernel-tools x86_64 3.10.0-957.1.3.el7 updates 7.1 M
 kernel-tools-libs x86_64 3.10.0-957.1.3.el7 updates 7.0 M
 libgudev1 x86_64 219-62.el7_6.2 updates 96 k
 libvirt-bash-completion x86_64 4.5.0-10.el7_6.3 updates 182 k
 libvirt-client x86_64 4.5.0-10.el7_6.3 updates 478 k
 libvirt-libs x86_64 4.5.0-10.el7_6.3 updates 4.2 M
 lvm2 x86_64 7:2.02.180-10.el7_6.2 updates 1.3 M
 lvm2-libs x86_64 7:2.02.180-10.el7_6.2 updates 1.1 M
 mutter x86_64 3.28.3-5.el7 updates 2.3 M
 python-perf x86_64 3.10.0-957.1.3.el7 updates 7.1 M
 ruby x86_64 2.0.0.648-34.el7_6 updates 71 k
 ruby-irb noarch 2.0.0.648-34.el7_6 updates 92 k
 ruby-libs x86_64 2.0.0.648-34.el7_6 updates 2.8 M
 rubygem-bigdecimal x86_64 1.2.0-34.el7_6 updates 83 k
 rubygem-io-console x86_64 0.4.2-34.el7_6 updates 54 k
 rubygem-json x86_64 1.7.7-34.el7_6 updates 79 k
 rubygem-psych x86_64 2.0.0-34.el7_6 updates 82 k
 rubygem-rdoc noarch 4.0.0-34.el7_6 updates 322 k
 rubygems noarch 2.0.14.1-34.el7_6 updates 219 k
 selinux-policy noarch 3.13.1-229.el7_6.6 updates 483 k
 selinux-policy-targeted noarch 3.13.1-229.el7_6.6 updates 6.9 M
 systemd x86_64 219-62.el7_6.2 updates 5.1 M
 systemd-libs x86_64 219-62.el7_6.2 updates 406 k
 systemd-sysv x86_64 219-62.el7_6.2 updates 83 k
 tzdata noarch 2018i-1.el7 updates 490 k
 tzdata-java noarch 2018i-1.el7 updates 187 k
 xorg-x11-server-Xorg x86_64 1.20.1-5.1.el7 updates 1.4 M
 xorg-x11-server-common x86_64 1.20.1-5.1.el7 updates 54 k

Transaction Summary
=====================================================================================================
Install 1 Package
Upgrade 50 Packages

Total download size: 240 M
--- >8 END HERE 8<---

Hmmm, nothing from that list of needed updates looks like a potential culprit to me. :(
arrfab

arrfab

2019-01-17 06:28

administrator   ~0033613

Before we even discuss updating livecd-tools to higher version (because we tried and it was broken, so we stopped), can you just
- install a centos 7 host
- fully updated (so 7.6.1810 + updates)
- install livecd-tools from extras (so the current one)
- build image with https://github.com/CentOS/sig-core-livemedia/blob/master/kickstarts/centos-7-live-gnome.cfg

As it worked for us (that's what we used for the live images for the 7.6.1810 release), I'd like to see what's the issue on your side with the version we actually have in Extras.
dowdle

dowdle

2019-01-17 15:48

reporter   ~0033616

I'll get started on your suggestion right now... and will report the specifics once completed. A new VM for the task is installing while I type this.

However, I ran into the problem BEFORE trying to use the newer version of livecd-creator. I build my remixes with the updates repos enabled. That produced broken media with the current extras livecd-tools release. I ran into the same problem in Fedora (with Fedora builds on a Fedora build host) that was fixed by their recent update(s)... which is why I requested the newer version in CentOS.

I suspect, if I go without the updates, It'll work, but if I want updates applied it will not.

I understand CentOS doesn't care much about fixing the issue for the current release because the GA media is already built and works... but for future releases, I don't think that'll be a good strategy.
dowdle

dowdle

2019-01-18 01:18

reporter   ~0033632

I got mixed results from two different build hosts... so I'm going to start all over by doing a fresh install on all of my build hosts. I can tell you on one freshly installed build host, I was able to get working media from the stock CentOS GNOMELive kickstart, with AND without updates... as well as some of my remixes. On the others that have been around for a while, not so much.

Why didn't I try this before?

I hope to test both the current Extras livecd-tools and the newer one (with manual patch applied). I'm starting to wonder if my build hosts were polluted in some way and part of the problem. I hope I didn't come off as harsh with any of my previous commentary. :) Its just been a long decade of occasional anaconda and livecd-creator breakage and fixes, repeat... mostly on Fedora. It was weird that on the suspected polluted hosts I could get working media without updates and broken media with updates. Anyhoo... more when I know more.
dowdle

dowdle

2019-01-18 19:40

reporter   ~0033635

In the immortal words of Emily Litella, "Nevermind."

The stock package from CentOS Extras works fine.
ngompa

ngompa

2019-01-18 20:57

reporter   ~0033636

The latest version of livecd-tools should work with CentOS 7.6 as well. All the issues that were encountered earlier should be fixed now.
dowdle

dowdle

2019-01-18 23:58

reporter   ~0033637

ngompa:

I did get around to testing your newer livecd-creator builds. WIthout the patch, yeah... can't login. With the patch, works fine. So, I concur with you that livecd-tools-26.0-3.el7.x86_64+patch work fine... at least with my remixes. I didn't bother to build the stock CentOS live media.

Issue History

Date Modified Username Field Change
2018-12-15 00:03 dowdle New Issue
2018-12-15 00:03 dowdle Tag Attached: livecd-tools
2018-12-15 00:23 toracat Status new => assigned
2018-12-15 00:23 toracat Summary Please update livecd-tools to 26.1 - ivecd-tools-21.4-2.el7 from 2015 is in Extras and has issues => Please update livecd-tools to 26.1 - livecd-tools-21.4-2.el7 from 2015 is in Extras and has issues
2018-12-15 11:03 pgreco Note Added: 0033350
2018-12-17 16:27 dowdle Note Added: 0033372
2018-12-29 03:07 dowdle Note Added: 0033470
2019-01-16 16:24 dowdle Note Added: 0033605
2019-01-16 16:27 dowdle Note Added: 0033606
2019-01-17 00:57 dowdle Note Added: 0033612
2019-01-17 06:28 arrfab Status assigned => feedback
2019-01-17 06:28 arrfab Note Added: 0033613
2019-01-17 15:48 dowdle Note Added: 0033616
2019-01-17 15:48 dowdle Status feedback => assigned
2019-01-18 01:18 dowdle Note Added: 0033632
2019-01-18 19:40 dowdle Note Added: 0033635
2019-01-18 20:57 ngompa Note Added: 0033636
2019-01-18 23:58 dowdle Note Added: 0033637