View Issue Details

IDProjectCategoryView StatusLast Update
0017187mirrorunclassifiedpublic2020-04-20 08:52
Reporterpraiskup 
PrioritynormalSeveritymajorReproducibilityrandom
Status newResolutionopen 
Summary0017187: cloudfronts i3 mirror is (often?) broken
DescriptionHey all, the 'd36uatko69830t' i3 CentOS mirror is broken, and breaks builds in
Fedora Copr buildsystem (builds are done in AWS, so the i3 mirror is the
closest one).

We try to solve similar problem in Fedora infrastructure:
https://pagure.io/fedora-infrastructure/issue/8679

The logs:

$ dnf.librepo.log:
2020-03-25T22:40:18Z INFO Downloading: http://mirrorlist.centos.org/?release=7&arch=aarch64&repo=sclo-rh
2020-03-25T22:40:18Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/repodata/repomd.xml
2020-03-25T22:40:19Z INFO Downloading: http://mirrorlist.centos.org/?release=7&arch=aarch64&repo=sclo-rh
2020-03-25T22:40:19Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm (IP: 99.84.185.116)
2020-03-25T22:40:19Z INFO Downloading: http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 99.84.185.116)
2020-03-25T22:40:19Z INFO Downloading: http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm (IP: 99.84.185.116)
2020-03-25T22:40:19Z INFO Downloading: http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm (IP: 72.29.84.207)
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 72.29.84.207)
2020-03-25T22:40:19Z INFO Downloading: http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Downloading: http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm (IP: 72.29.84.207)
2020-03-25T22:40:19Z INFO Downloading: http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 72.29.84.207)
2020-03-25T22:40:19Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm (IP: 72.29.84.207)
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm (IP: 72.29.84.207)
2020-03-25T22:40:19Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm
2020-03-25T22:40:19Z INFO Downloading: http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm
2020-03-25T22:40:19Z INFO Error during transfer: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 99.84.185.116)
2020-03-25T22:40:19Z INFO Error while downloading: Cannot download devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: All mirrors were tried

$ builder-live.log:
[SKIPPED] libuuid-devel-2.23.2-61.el7_7.1.aarch64.rpm: Already downloaded
[SKIPPED] python-firewall-0.6.3-2.el7_7.3.noarch.rpm: Already downloaded
[MIRROR] devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm (IP: 99.84.185.116)
[MIRROR] devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 99.84.185.116)
[MIRROR] devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm (IP: 99.84.185.116)
[MIRROR] devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm (IP: 72.29.84.207)
[MIRROR] devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 72.29.84.207)
[MIRROR] devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm (IP: 72.29.84.207)
[MIRROR] devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 72.29.84.207)
[MIRROR] devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-gcc-6.3.1-3.1.el7.aarch64.rpm (IP: 72.29.84.207)
[MIRROR] devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm: Status code: 404 for http://mirror.centos.org/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-binutils-2.27-12.el7.1.aarch64.rpm (IP: 72.29.84.207)
[MIRROR] devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: Status code: 404 for http://d36uatko69830t.cloudfront.net/altarch/7/sclo/aarch64/rh/devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm (IP: 99.84.185.116)
[FAILED] devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: No more mirrors to try - All mirrors were already tried without success
(85-86/86): devtools 62% [============ ] 82 kB/s | 46 MB 05:50 ETA
Error: Error downloading packages:
  Cannot download devtoolset-6/devtoolset-6-runtime-6.1-1.el7.aarch64.rpm: All mirrors were tried

Example build:
https://download.copr.fedorainfracloud.org/results/ligenix/enterprise-virtualization/epel-7-aarch64/01320592-glusterfs/
TagsNo tags attached.

Activities

praiskup

praiskup

2020-04-20 08:52

reporter   ~0036721

Fedora recently switched from "manual periodic `aws sync`" to transparent
"i3 proxying". That means that no manual syncs need to be done, see the
discussion in
https://pagure.io/fedora-infrastructure/issue/8679

This way is much more (maybe even 100%) reliable because aws mirror always
gives up2date info with the top-level mirror).

Can we consider doing the same thing for CentOS?

Issue History

Date Modified Username Field Change
2020-03-27 07:03 praiskup New Issue
2020-04-20 08:52 praiskup Note Added: 0036721