View Issue Details

IDProjectCategoryView StatusLast Update
0016408mirrorunsortedpublic2020-04-30 06:33
Reportergenin_d 
PrioritynormalSeveritycrashReproducibilityalways
Status resolvedResolutionfixed 
PlatformVagrantOSCentOSOS Version7.6.1810
Summary0016408: Vagrant cant install box with centos7 due to missing file on the mirrir
DescriptionAfter some moment cant start centos7 box in Vagrant. Its messaging what some file miss on the mirror:
http://vault.centos.org/7.6.1810/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

I was checked link manually from internet browser and file is really missing there.

Steps To Reproduce1. select in vagrantfile box "centos/7"
2. vagrant up

Additional Information$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'centos/7'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'centos/7' version '1905.1' is up to date...
==> default: Setting the name of the VM: vagrant-configs_default_1568752865153_72095
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
[default] No Virtualbox Guest Additions installation found.
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.corbina.net
 * extras: centos-mirror.rbc.ru
 * updates: centos-mirror.rbc.ru
Resolving Dependencies
--> Running transaction check
---> Package centos-release.x86_64 0:7-6.1810.2.el7.centos will be updated
---> Package centos-release.x86_64 0:7-7.1908.0.el7.centos will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package Arch Version Repository Size
================================================================================
Updating:
 centos-release x86_64 7-7.1908.0.el7.centos base 26 k

Transaction Summary
================================================================================
Upgrade 1 Package

Total download size: 26 k
Downloading packages:
No Presto metadata available for base
Public key for centos-release-7-7.1908.0.el7.centos.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/7/base/packages/centos-release-7-7.1908.0.el7.centos.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package : centos-release-7-6.1810.2.el7.centos.x86_64 (@anaconda)
 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating : centos-release-7-7.1908.0.el7.centos.x86_64 1/2
  Cleanup : centos-release-7-6.1810.2.el7.centos.x86_64 2/2
  Verifying : centos-release-7-7.1908.0.el7.centos.x86_64 1/2
  Verifying : centos-release-7-6.1810.2.el7.centos.x86_64 2/2

Updated:
  centos-release.x86_64 0:7-7.1908.0.el7.centos

Complete!
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.corbina.net
 * extras: centos-mirror.rbc.ru
 * updates: centos-mirror.rbc.ru
http://vault.centos.org/7.6.1810/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
Tags7.6.1810, centos7, missing dependency

Activities

tcooper

tcooper

2019-09-17 22:19

reporter   ~0035111

This, along with already closed Issue (0016407), indicate what may be a common problem with automated build tools expecting to find files in repositories and they are missing around the time the new update is released.

Our build system encountered this when yum update rev'd our OS version unexpectedly.

It is clear from previous bug reports and notes that the observed behavior is expected.

I have been manually watching the 7.7 update progress fairly closely and missed the change which appeared to start at 2019-09-15 01:00 when the target directory was created.

We had deployed test nodes from the updates in the CR repository and, finding unresolved issues with 3rd party builds of kernel modules, would NOT have elected to change our AMI builds had we know the update had landed or was landing. Now we need to roll back our builds to 7.6.1810 and the RPMs have not yet landed in vault.centos.org. It is a problem of our own creation but I'm wondering what I can be watching to prevent this in the future.

What I'm wondering is where the mirror change was announced and when was it announced.

Neither centos-mirror or centos-mirror-announce lists have any record that I can see.

Thanks
SevernE86

SevernE86

2019-09-18 07:40

reporter   ~0035112

According to the responses for issue https://bugs.centos.org/view.php?id=16407, this is due to the newest "old release" not getting copied back to the vault immediately after a new release is pushed and should be resolved in a week or two. As a side note, I'm a little disappointed at how flippantly that issue was treated given how - as seen here - it breaks Vagrant provisioning for the CentOS-published box. A workaround I've found is to use the generic/centos7 boxes for the time being - at least those will go through the default provisioning process without blowing up.
stefanpeturs

stefanpeturs

2019-09-19 09:32

reporter   ~0035135

I am experiencing the same issue.
Is there any news related to solving this?
pemosi

pemosi

2019-09-19 10:31

reporter   ~0035137

Same here. Using Vagrant and settings up a few "centos/7" boxes. Trying to learn how to configure everything using Ansible.
When this stuff just breaks all of a sudden it's quite annoying.
I'm also curious about the reply in bug 16407, as noted above.
https://bugs.centos.org/view.php?id=16407#c35109
"Yes, it's normal. The old release gets copied to vault after a week or two. It's not automatically there as soon as the next one comes out. Never has been, never will be."
Why??
Is it technically impossible to copy the current release to the vault before releasing the new one so that all VM boxes that rely on the now older version won't all break instantly?
I guess once the new Vagrant box version is out it will use the current version (if I understand the problem/workflow correctly), but still, is there anything preventing an early copy of the current version to the vault in anticipation of the release of the new version?
I assume it would prevent these hickups.

@stefanpeturs As mentioned above, switching to "generic/centos7" meanwhile solves the issue, but keep in mind the firewalld is running by default there, took me a while to figure out why lots of things stopped working when I switched... Also, the virtualbox guest additions plugin have a version mismatch with the pre-installed guest additions on "generic/centos7", so there's some more headache.
SevernE86

SevernE86

2019-09-19 10:39

reporter   ~0035138

@pemosi
I tried downgrading the CentOS box to v1902.01, and it still failed with the same issue. The problem isn't that the Vagrant box is "obsolete" - it's that the lag in updating the new legacy repositories breaks the default workflow for presumably any of the Vagrant boxes that CentOS publishes. Also, what issues are you having with the guest additions with the generic box? I haven't had any.
pemosi

pemosi

2019-09-19 10:49

reporter   ~0035139

@SevernE86 I'm not exactly sure how this works, but if you look at https://app.vagrantup.com/centos/boxes/7 and check the releases, you'll see that v1905.1, v1902.1, ..., all the way back to v1811.01 say "based on CentOS 7.6.1811".
So I assume all those versions will fail because they rely on 7.6.1811.
Hopefully, the next Vagrant box release will be based on CentOS 7.7.1911 and it will work again (or we have to wait until 7.6.1811 makes it to the vault).
Well, either that or I have misunderstood the whole thing :)
Please correct me if I'm wrong.
stefanpeturs

stefanpeturs

2019-09-19 11:40

reporter   ~0035141

Thanks for the update @pemosi - I'll try to make it work with the generic version.
I do agree with you that it seems obscure that it should take weeks to copy the old release to a vault - as it is actually breaking the construction of the machine.
Fingers crossed we won't have to wait for too long.
marcelo.altmann

marcelo.altmann

2019-09-19 16:36

reporter   ~0035143

I'm seeing the same issue with vagrant. It's a bit of surprise that this is considered "expected" since it's breaking vagrant to boot-up properly.
danls

danls

2019-09-19 19:26

reporter   ~0035146

I lost the entire day yesterday chasing this. In my case, the culprit was the vbguest plugin which tries to update the CentOS repos (see vagrant.d/gems/2.4.6/gems/vagrant-vbguest-0.19.0/lib/vagrant-vbguest/installers/centos.rb). I just switched to the "generic/centos7" box and the issue was fixed. I also added a comment on this issue to the maintainers of vbguest (https://github.com/dotless-de/vagrant-vbguest/issues/325) because it seems likely that this will happen again for every CentOS update going forward.
danls

danls

2019-09-19 19:27

reporter   ~0035147

Sorry, the correct issue is: https://github.com/dotless-de/vagrant-vbguest/issues/320
pemosi

pemosi

2019-09-20 10:43

reporter   ~0035157

I saw http://vault.centos.org/7.6.1810/os/x86_64/repodata/repomd.xml was now acessible so I tried switching back to "centos/7" and everything works again.
I'm just knowing that next release this thing will probably happen again.
SevernE86

SevernE86

2019-09-20 10:48

reporter   ~0035158

Kind of confirms what everybody's been saying that this was due to CentOS's workflow when upgrading versions. The question is, why does there have to be a gap between releasing a new version of CentOS and setting up the legacy repository? With all of the options that are out there for automated deployment, it's hard to think of a reason why this should still be occurring besides inertia for the status quo.
just_another_user

just_another_user

2020-04-27 20:36

reporter   ~0036778

I'm seeing this issue on 7.7.1908 today when I vagrant up with centos/7:


```Vagrant assumes that this means the command failed!

yum install -y kernel-devel-`uname -r` --enablerepo=C*-base --enablerepo=C*-updates

Stdout from the command:

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.codec-cluster.org
 * extras: mirror.sfo12.us.leaseweb.net
 * updates: repos-lax.psychz.net


Stderr from the command:

http://vault.centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.



 One of the configured repositories failed (CentOS-7.7.1908 - Base),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=C7.7.1908-base ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable C7.7.1908-base
        or
            subscription-manager repos --disable=C7.7.1908-base

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=C7.7.1908-base.skip_if_unavailable=true

failure: repodata/repomd.xml from C7.7.1908-base: [Errno 256] No more mirrors to try.
http://vault.centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found```
awltux

awltux

2020-04-28 14:49

reporter   ~0036792

I've also started seeing the same vagrant-up issue today, through different mirrors.
Reading through the notes above, it seems to be related to the CentOS build process temporarily dropping the x86_64 folder during upgrades.


INFO interface: error: The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

yum install -y kernel-devel-`uname -r` --enablerepo=C*-base --enablerepo=C*-updates

Stdout from the command:

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.vorboss.net
 * extras: mirror.vorboss.net
 * updates: mirror.as29550.net


Stderr from the command:

http://vault.centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
arrfab

arrfab

2020-04-28 14:54

administrator   ~0036794

@JohnnyHughes is the one pushing all that is needed for every release and he is aware that he still need to archive 7.7.1908 to vault.
But the underlying question is : *why* are some people pointing to vault.centos.org in the first place ? it shouldn't be used at all for anything and is there just as a cold archive (and slow network too)
Would be curious to know who has enabled those repositories, disabled by default, as people using only normal/current/enabled-by-default repositories aren't suffering from this behavior either.
awltux

awltux

2020-04-28 21:07

reporter   ~0036803

@arrfab Agreed on the *why*, I've been looking through the vagrant code and the vbguest plugin code but I cannot find any mention of the vault.centos.org host in the code.
Could it be some sort of default from the centos mirror system?
I've implemented a local workaround just now (uninstalled the vbguest plugin) , so I can continue with my development work.
arrfab

arrfab

2020-04-29 06:10

administrator   ~0036806

I see that @johnnyHugues pushed the os repo to vault : http://vault.centos.org/7.7.1908/os/x86_64/
That should fix the underlying issue but still to be investigated, for people who are using vagrant (we don't) : why are the disabled-by-default repositories enabled in that vagrant box ?
awltux

awltux

2020-04-30 06:06

reporter   ~0036829

@arrfab That makes more sense.
I think the way in which the vbguest plugin tries to update the kernel (using wildcards to enable repos):
    yum install -y kernel-devel-`uname -r` --enablerepo=C*-base --enablerepo=C*-updates
is enabling the source repos. When the stars are aligned and a CentOS upgrade is in progress, the yum repo selection process inadvertently finds the 'sources' repo.

I think the 'enablerepo' parameters are intended to allow specific repo versions to be specified, but I can't think when this would be necessary, perhaps for experimental kernel builds?
If no repo version is specified, the plugin uses an asterisk.

Anyway, I think this is square on the vbguest plugin developer.
I'm thinking this ticket can be closed.

Thanks for your time and nudges in the right direction. Much appreciated.
arrfab

arrfab

2020-04-30 06:28

administrator   ~0036830

@awltux : thanks for your investigation on the root cause. Probably worth verifying with the vbguest plugin maintainer indeed.
OTOH, I asked @JohnnyHugues to also have previous release pushed to vault as part of our release process so that it will be already there when new one is out and being the default.
Using/pointing to vault repositories is still a bad idea, but at least it would help like in this vagrant corner case.
Closing it now

Issue History

Date Modified Username Field Change
2019-09-17 21:12 genin_d New Issue
2019-09-17 21:12 genin_d Tag Attached: 7.6.1810
2019-09-17 21:12 genin_d Tag Attached: centos7
2019-09-17 21:12 genin_d Tag Attached: missing dependency
2019-09-17 22:19 tcooper Note Added: 0035111
2019-09-18 07:40 SevernE86 Note Added: 0035112
2019-09-19 09:32 stefanpeturs Note Added: 0035135
2019-09-19 10:31 pemosi Note Added: 0035137
2019-09-19 10:39 SevernE86 Note Added: 0035138
2019-09-19 10:49 pemosi Note Added: 0035139
2019-09-19 11:40 stefanpeturs Note Added: 0035141
2019-09-19 16:36 marcelo.altmann Note Added: 0035143
2019-09-19 19:26 danls Note Added: 0035146
2019-09-19 19:27 danls Note Added: 0035147
2019-09-20 10:43 pemosi Note Added: 0035157
2019-09-20 10:48 SevernE86 Note Added: 0035158
2020-04-27 20:36 just_another_user Note Added: 0036778
2020-04-28 14:49 awltux Note Added: 0036792
2020-04-28 14:54 arrfab Note Added: 0036794
2020-04-28 14:56 arrfab Note Added: 0036795
2020-04-28 21:07 awltux Note Added: 0036803
2020-04-29 06:10 arrfab Note Added: 0036806
2020-04-30 06:06 awltux Note Added: 0036829
2020-04-30 06:28 arrfab Note Added: 0036830
2020-04-30 06:33 arrfab Status new => resolved
2020-04-30 06:33 arrfab Resolution open => fixed