View Issue Details

IDProjectCategoryView StatusLast Update
0016337CentOS CI[All Projects] generalpublic2019-08-16 09:06
Reporterevgeni 
PriorityhighSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Summary0016337: internal EPEL mirrors broken
DescriptionHi,

we run VMs on ci.centos.org and these seem not to be able to reach EPEL anymore. Example errors can be found at https://ci.centos.org/job/foreman-katello-3.13-test/5/console and https://ci.centos.org/job/foreman-nightly-centos7-test/163/console

Yum fails with:
failure: repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.
http://mirror.ci.centos.org/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
http://mirror.es.its.nyu.edu/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 12] Timeout on http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: (28, 'Connection timed out after 30001 milliseconds')
https://d2lzkl7pfhq30w.cloudfront.net/pub/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found
https://mirror.umd.edu/fedora/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found
https://iad.mirror.rackspace.com/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found
https://mirror.vcu.edu/pub/gnu+linux/epel/7/x86_64/repodata/59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd-primary.sqlite.bz2: [Errno 14] HTTPS Error 404 - Not Found

And indeed, I can't find a 59cd2d904711571ac63cfec0fec0641233dc027173c6667914bbcc2e10ea11dd metadata on any uptodate EPEL mirror I can reach from the outside world.
TagsNo tags attached.

Activities

evgeni

evgeni

2019-08-15 09:39

reporter   ~0034976

This can be "fixed" by adding "exclude=mirror.ci.centos.org" to /etc/yum/pluginconf.d/fastestmirror.conf, as mirror.ci.centos.org seems to serve bad metadata neither itself nor any public mirror can follow.
evgeni

evgeni

2019-08-15 09:42

reporter   ~0034977

I'm seeing this at least since yesterday 9:30 UTC.
pbergene

pbergene

2019-08-15 10:21

reporter   ~0034978

Can see this affecting multiple builds on cico too.
ptisnovs

ptisnovs

2019-08-16 07:45

reporter   ~0034986

Seems like the issue is fixed, at least partially: https://ci.centos.org/job/devtools-fabric8-analytics-worker-fabric8-analytics-pydoc/796/console
evgeni

evgeni

2019-08-16 08:56

reporter   ~0034988

from the linked jenkins job:

http://mirror.ci.centos.org/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
Trying other mirror.

Seems it "fixed" itself by not offering the (bad) metadata at all for epel now :)
ptisnovs

ptisnovs

2019-08-16 09:04

reporter   ~0034989

yes @evgeni :)
ptisnovs

ptisnovs

2019-08-16 09:06

reporter   ~0034990

IIRC it did not provide any metadata before this issue, so we got used to it

Issue History

Date Modified Username Field Change
2019-08-15 07:08 evgeni New Issue
2019-08-15 07:08 evgeni Status new => assigned
2019-08-15 09:39 evgeni Note Added: 0034976
2019-08-15 09:42 evgeni Note Added: 0034977
2019-08-15 10:21 pbergene Note Added: 0034978
2019-08-16 07:45 ptisnovs Note Added: 0034986
2019-08-16 08:56 evgeni Note Added: 0034988
2019-08-16 09:04 ptisnovs Note Added: 0034989
2019-08-16 09:06 ptisnovs Note Added: 0034990