View Issue Details

IDProjectCategoryView StatusLast Update
0016816mirrorunclassifiedpublic2020-03-27 12:56
Reportergtirloni 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Summary0016816: isoredirect: Packer getting HTML output instead of redirect
DescriptionWhile working with Packer.io, a URL using isoredirect is constantly failing. It's been determined that isoredirect is sending different output based on the user-agent, which breaks automation.
Steps To Reproducecurl -v http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso

VS.

curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" -L -O http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso
Additional InformationOriginally reported at https://github.com/hashicorp/packer/issues/8448
Tagsansible, iso

Activities

mnowak

mnowak

2020-01-23 13:21

reporter   ~0036100

Packer's User Agent seems to be defined here: https://github.com/hashicorp/packer/blob/master/helper/useragent/useragent.go. Currently is starts with "Packer/".
arrfab

arrfab

2020-01-24 05:42

administrator   ~0036110

Last edited: 2020-01-24 05:50

View 2 revisions

Well, yes, our isoredirect nodes will output a html list of mirrors in your location providing such iso (as you can see with your browser when you go to http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso)
I wasn't aware of the fact that Packer would download .iso to build artifact, so we only added wget/curl to the list of "agents" to just redirect to the first url automatically :
https://github.com/CentOS/mirrorlists-code/blob/master/frontend/isoredirect.py#L89

So adding Packer to the list and reload would fix your issue ?

I' just pushed that small change on our dev instance :

    agent = request.environ.get('HTTP_USER_AGENT')[:5].lower()
- if (agent == "curl/" or agent == "wget/") and filename != "":
+ if (agent == "curl/" or agent == "wget/" or agent == "packer/") and filename != "":
       fast_redirect = True

Can you point to isoredirect.dev.centos.org and give feedback ? If that works fine for you, I'll push it on all nodes.

PS : slow to respond as currently in travel mode

mnowak

mnowak

2020-01-24 09:56

reporter   ~0036111

I am traveling too, so, can test early next week. In the meantime, in Packer sources the user agent appears with capital P -- not sure if that makes a difference in your code. Thanks so far!
arrfab

arrfab

2020-01-24 17:27

administrator   ~0036114

haven't a chance to test it (did it quickly in the morning before having had breakfast), but you can see that agent is lower-cased ( .lower() function) so it should be indeed packer/ in the string comparison.
I'll try myself to give it a try but for sure we can add / fix this easily
mnowak

mnowak

2020-01-24 20:57

reporter   ~0036119

I think you are right, I missed that part, it ought to work.
mnowak

mnowak

2020-01-29 10:57

reporter   ~0036152

It still fails for me. Can it be that the change is not live yet? Upon inspection the downloaded file is a HTML file with my local mirrors. Perhaps you can see my Packer's User Agent on the dev isoredirect service? Thanks!

ISO_URL="http://isoredirect.dev.centos.org/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso" ISO_CHECKSUM="7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948" QCOW2_IMAGE="centos-8-x86_64" packer build packer.json
qemu output will be in this color.

==> qemu: Retrieving ISO
==> qemu: Trying http://isoredirect.dev.centos.org/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso
==> qemu: Trying http://isoredirect.dev.centos.org/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso?checksum=7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948
CentOS-8.1.1911-x86_64-boot.iso 12.15 KiB / 12.15 KiB [============================================================================================================================================================================] 100.00% 0s
==> qemu: Checksum did not match, removing /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso
==> qemu: error downloading ISO: [Checksums did not match for /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso.
==> qemu: Expected: 7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948
==> qemu: Got: 933c1c5fa2b3716d4cd100eab228f4d3474e81925db522a18f7116ff3405d7f2
==> qemu: *sha256.digest]
Build 'qemu' errored: error downloading ISO: [Checksums did not match for /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso.
Expected: 7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948
Got: 933c1c5fa2b3716d4cd100eab228f4d3474e81925db522a18f7116ff3405d7f2
*sha256.digest]

==> Some builds didn't complete successfully and had errors:
--> qemu: error downloading ISO: [Checksums did not match for /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso.
Expected: 7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948
Got: 933c1c5fa2b3716d4cd100eab228f4d3474e81925db522a18f7116ff3405d7f2
*sha256.digest]

==> Builds finished but no artifacts were created.
mnowak

mnowak

2020-02-21 08:54

reporter   ~0036360

Any idea what might have gone wrong?
cdbunch72

cdbunch72

2020-03-12 15:41

reporter   ~0036497

This also affects the ansible get_url module, which defaults to the agent string: ansible-httpget

As a workaround you can set the http_agent attribute to Wget/1.17.1, but it would be nice if the default http_agent worked.
arrfab

arrfab

2020-03-26 12:22

administrator   ~0036586

Hi guys,

Sorry for the late report here, but was caught into various fires for distro and infra so [RFE]s were stuck.
I just modified code and pushed to https://isoredirect.stg.centos.org so can you give it a try with real Packer and ansible get_url ?

What I tried on my side (simulated) and that worked :
 curl -A "Packer/10.1" -L -O http://isoredirect.stg.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso -L
 curl -A "ansible-httpget" -L -O http://isoredirect.stg.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso -L

If you can confirm that it also works for you, I'll push to master branch and ansible will deploy it
Thanks for your patience on this one ;-)
cdbunch72

cdbunch72

2020-03-26 15:20

reporter   ~0036587

Ok. It works with ansible get_url without setting the http_agent now. Thanks.
arrfab

arrfab

2020-03-27 12:56

administrator   ~0036592

Pushed the change on isoredirect.centos.org nodes
Verified also with 'curl -A "Packer/10.1" -O http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso -L' and it worked (as tested in staging)
Closing for now

Issue History

Date Modified Username Field Change
2019-12-10 23:04 gtirloni New Issue
2019-12-10 23:04 gtirloni Tag Attached: iso
2020-01-23 13:21 mnowak Note Added: 0036100
2020-01-24 05:42 arrfab Note Added: 0036110
2020-01-24 05:43 arrfab Status new => feedback
2020-01-24 05:50 arrfab Note Edited: 0036110 View Revisions
2020-01-24 09:56 mnowak Note Added: 0036111
2020-01-24 17:27 arrfab Note Added: 0036114
2020-01-24 20:57 mnowak Note Added: 0036119
2020-01-29 10:57 mnowak Note Added: 0036152
2020-02-21 08:54 mnowak Note Added: 0036360
2020-03-12 15:41 cdbunch72 Note Added: 0036497
2020-03-12 15:43 cdbunch72 Tag Attached: ansible
2020-03-26 12:22 arrfab Note Added: 0036586
2020-03-26 15:20 cdbunch72 Note Added: 0036587
2020-03-27 12:56 arrfab Status feedback => resolved
2020-03-27 12:56 arrfab Resolution open => fixed
2020-03-27 12:56 arrfab Note Added: 0036592