View Issue Details

IDProjectCategoryView StatusLast Update
0012996mirror[All Projects] generalpublic2017-03-25 17:13
Reporteremacchi 
PriorityhighSeveritymajorReproducibilitysometimes
Status closedResolutionunable to reproduce 
Summary0012996: centos.org repositories are unreliable
DescriptionWe hit 138 times a situation where package couldn't be downloaded from centos repositories:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22No%20more%20mirrors%20to%20try%5C%22

That's way too much and our CI becomes really unstable because of that. We need to find a solution.

Issue also reported here: https://bugs.launchpad.net/tripleo/+bug/1674681

Note: it happens on all cloud providers provided by OpenStack Infra.
TagsNo tags attached.

Activities

avij

avij

2017-03-21 14:22

manager   ~0028896

From one of the logs:

Downloading packages:
https://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-pike/common/python2-pysocks-1.5.6-3.el7.noarch.rpm: [Errno 14] HTTPS Error 302 - Found
Trying other mirror.

If it isn't obvious, I believe Errno 14 means that there was an infinite loop of 302 redirects. 302 itself is OK and expected.

No idea how to fix this, though.
amoralej

amoralej

2017-03-21 15:13

reporter   ~0028902

Would it be possible to get the full list of servers serving buildlogs.cdn.centos.org?, we could try to install packatges from each one and try to isolate the issue to a specific one.
arrfab

arrfab

2017-03-21 20:46

administrator   ~0028908

We don't have a list of those nodes, as there are not under our control.
Also, that would help to see a verbose log . If you have a 302 from buildlogs.centos.org, that's working as expected. So we'd like to see where it doesn't work *after* the centos infra (so on the CDN itself)
Only option I can even think of is an invalid certificate on their backend nodes (one node not having the correct one) but needs logs on our side before relaying that to CDN provider
avij

avij

2017-03-21 20:54

manager   ~0028909

It would help tremendously if you could trigger this issue with debugging output enabled -- like "URLGRABBER_DEBUG=1 yum install python-whatever". That would show all the HTTP requests that get sent, and the IP addresses of each connection.
arrfab

arrfab

2017-03-24 18:48

administrator   ~0028930

Last edited: 2017-03-24 18:48

View 2 revisions

Putting in Feedback mode as it works on various machines we tested, so waiting for more info.
Can you reproduce that "issue" outside of your infra/environment ?

emacchi

emacchi

2017-03-25 01:23

reporter   ~0028933

I think we can close this one for now, we haven't hit this problem since more than 24h.

Issue History

Date Modified Username Field Change
2017-03-21 13:43 emacchi New Issue
2017-03-21 14:22 avij Note Added: 0028896
2017-03-21 15:13 amoralej Note Added: 0028902
2017-03-21 20:46 arrfab Note Added: 0028908
2017-03-21 20:47 arrfab Project Buildsys => mirror
2017-03-21 20:54 avij Note Added: 0028909
2017-03-24 18:48 arrfab Status new => feedback
2017-03-24 18:48 arrfab Note Added: 0028930
2017-03-24 18:48 arrfab Note Edited: 0028930 View Revisions
2017-03-25 01:23 emacchi Note Added: 0028933
2017-03-25 01:23 emacchi Status feedback => assigned
2017-03-25 17:13 arrfab Status assigned => closed
2017-03-25 17:13 arrfab Resolution open => unable to reproduce