 |
This, along with already closed Issue (0016407), indicate what may be a common problem with automated build tools expecting to find files in repositories and they are missing around the time the new update is released. Our build system encountered this when yum update rev'd our OS version unexpectedly. It is clear from previous bug reports and notes that the observed behavior is expected. I have been manually watching the 7.7 update progress fairly closely and missed the change which appeared to start at 2019-09-15 01:00 when the target directory was created. We had deployed test nodes from the updates in the CR repository and, finding unresolved issues with 3rd party builds of kernel modules, would NOT have elected to change our AMI builds had we know the update had landed or was landing. Now we need to roll back our builds to 7.6.1810 and the RPMs have not yet landed in vault.centos.org. It is a problem of our own creation but I'm wondering what I can be watching to prevent this in the future. What I'm wondering is where the mirror change was announced and when was it announced. Neither centos-mirror or centos-mirror-announce lists have any record that I can see. Thanks |
|
|
According to the responses for issue https://bugs.centos.org/view.php?id=16407, this is due to the newest "old release" not getting copied back to the vault immediately after a new release is pushed and should be resolved in a week or two. As a side note, I'm a little disappointed at how flippantly that issue was treated given how - as seen here - it breaks Vagrant provisioning for the CentOS-published box. A workaround I've found is to use the generic/centos7 boxes for the time being - at least those will go through the default provisioning process without blowing up. |
|
|
I am experiencing the same issue. Is there any news related to solving this? |
|
|
Same here. Using Vagrant and settings up a few "centos/7" boxes. Trying to learn how to configure everything using Ansible. When this stuff just breaks all of a sudden it's quite annoying. I'm also curious about the reply in bug 16407, as noted above. https://bugs.centos.org/view.php?id=16407#c35109 "Yes, it's normal. The old release gets copied to vault after a week or two. It's not automatically there as soon as the next one comes out. Never has been, never will be." Why?? Is it technically impossible to copy the current release to the vault before releasing the new one so that all VM boxes that rely on the now older version won't all break instantly? I guess once the new Vagrant box version is out it will use the current version (if I understand the problem/workflow correctly), but still, is there anything preventing an early copy of the current version to the vault in anticipation of the release of the new version? I assume it would prevent these hickups. @stefanpeturs As mentioned above, switching to "generic/centos7" meanwhile solves the issue, but keep in mind the firewalld is running by default there, took me a while to figure out why lots of things stopped working when I switched... Also, the virtualbox guest additions plugin have a version mismatch with the pre-installed guest additions on "generic/centos7", so there's some more headache. |
|
|
@pemosi I tried downgrading the CentOS box to v1902.01, and it still failed with the same issue. The problem isn't that the Vagrant box is "obsolete" - it's that the lag in updating the new legacy repositories breaks the default workflow for presumably any of the Vagrant boxes that CentOS publishes. Also, what issues are you having with the guest additions with the generic box? I haven't had any. |
|
|
@SevernE86 I'm not exactly sure how this works, but if you look at https://app.vagrantup.com/centos/boxes/7 and check the releases, you'll see that v1905.1, v1902.1, ..., all the way back to v1811.01 say "based on CentOS 7.6.1811". So I assume all those versions will fail because they rely on 7.6.1811. Hopefully, the next Vagrant box release will be based on CentOS 7.7.1911 and it will work again (or we have to wait until 7.6.1811 makes it to the vault). Well, either that or I have misunderstood the whole thing :) Please correct me if I'm wrong. |
|
|
Thanks for the update @pemosi - I'll try to make it work with the generic version. I do agree with you that it seems obscure that it should take weeks to copy the old release to a vault - as it is actually breaking the construction of the machine. Fingers crossed we won't have to wait for too long. |
|
|
I'm seeing the same issue with vagrant. It's a bit of surprise that this is considered "expected" since it's breaking vagrant to boot-up properly. |
|
|
I lost the entire day yesterday chasing this. In my case, the culprit was the vbguest plugin which tries to update the CentOS repos (see vagrant.d/gems/2.4.6/gems/vagrant-vbguest-0.19.0/lib/vagrant-vbguest/installers/centos.rb). I just switched to the "generic/centos7" box and the issue was fixed. I also added a comment on this issue to the maintainers of vbguest (https://github.com/dotless-de/vagrant-vbguest/issues/325) because it seems likely that this will happen again for every CentOS update going forward. |
|
|
Sorry, the correct issue is: https://github.com/dotless-de/vagrant-vbguest/issues/320 |
|
|
I saw http://vault.centos.org/7.6.1810/os/x86_64/repodata/repomd.xml was now acessible so I tried switching back to "centos/7" and everything works again. I'm just knowing that next release this thing will probably happen again. |
|
|
Kind of confirms what everybody's been saying that this was due to CentOS's workflow when upgrading versions. The question is, why does there have to be a gap between releasing a new version of CentOS and setting up the legacy repository? With all of the options that are out there for automated deployment, it's hard to think of a reason why this should still be occurring besides inertia for the status quo. |
|
|
I'm seeing this issue on 7.7.1908 today when I vagrant up with centos/7: ```Vagrant assumes that this means the command failed! yum install -y kernel-devel-`uname -r` --enablerepo=C*-base --enablerepo=C*-updates Stdout from the command: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.codec-cluster.org * extras: mirror.sfo12.us.leaseweb.net * updates: repos-lax.psychz.net Stderr from the command: http://vault.centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found Trying other mirror. To address this issue please refer to the below wiki article https://wiki.centos.org/yum-errors If above article doesn't help to resolve this issue please use https://bugs.centos.org/. One of the configured repositories failed (CentOS-7.7.1908 - Base), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=C7.7.1908-base ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable C7.7.1908-base or subscription-manager repos --disable=C7.7.1908-base 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=C7.7.1908-base.skip_if_unavailable=true failure: repodata/repomd.xml from C7.7.1908-base: [Errno 256] No more mirrors to try. http://vault.centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found``` |
|
|
I've also started seeing the same vagrant-up issue today, through different mirrors. Reading through the notes above, it seems to be related to the CentOS build process temporarily dropping the x86_64 folder during upgrades. INFO interface: error: The following SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed! yum install -y kernel-devel-`uname -r` --enablerepo=C*-base --enablerepo=C*-updates Stdout from the command: Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.vorboss.net * extras: mirror.vorboss.net * updates: mirror.as29550.net Stderr from the command: http://vault.centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found |
 |
@JohnnyHughes is the one pushing all that is needed for every release and he is aware that he still need to archive 7.7.1908 to vault. But the underlying question is : *why* are some people pointing to vault.centos.org in the first place ? it shouldn't be used at all for anything and is there just as a cold archive (and slow network too) Would be curious to know who has enabled those repositories, disabled by default, as people using only normal/current/enabled-by-default repositories aren't suffering from this behavior either. |
|
|
Reminder sent to: JohnnyHughes |
|
|
@arrfab Agreed on the *why*, I've been looking through the vagrant code and the vbguest plugin code but I cannot find any mention of the vault.centos.org host in the code. Could it be some sort of default from the centos mirror system? I've implemented a local workaround just now (uninstalled the vbguest plugin) , so I can continue with my development work. |
|
|
I see that @johnnyHugues pushed the os repo to vault : http://vault.centos.org/7.7.1908/os/x86_64/ That should fix the underlying issue but still to be investigated, for people who are using vagrant (we don't) : why are the disabled-by-default repositories enabled in that vagrant box ? |
|
|
@arrfab That makes more sense. I think the way in which the vbguest plugin tries to update the kernel (using wildcards to enable repos): yum install -y kernel-devel-`uname -r` --enablerepo=C*-base --enablerepo=C*-updates is enabling the source repos. When the stars are aligned and a CentOS upgrade is in progress, the yum repo selection process inadvertently finds the 'sources' repo. I think the 'enablerepo' parameters are intended to allow specific repo versions to be specified, but I can't think when this would be necessary, perhaps for experimental kernel builds? If no repo version is specified, the plugin uses an asterisk. Anyway, I think this is square on the vbguest plugin developer. I'm thinking this ticket can be closed. Thanks for your time and nudges in the right direction. Much appreciated. |
|
|
@awltux : thanks for your investigation on the root cause. Probably worth verifying with the vbguest plugin maintainer indeed. OTOH, I asked @JohnnyHugues to also have previous release pushed to vault as part of our release process so that it will be already there when new one is out and being the default. Using/pointing to vault repositories is still a bad idea, but at least it would help like in this vagrant corner case. Closing it now |
- Anonymous
