View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0016816||mirror||unclassified||public||2019-12-10 23:04||2020-03-27 12:56|
|Summary||0016816: isoredirect: Packer getting HTML output instead of redirect|
|Description||While working with Packer.io, a URL using isoredirect is constantly failing. It's been determined that isoredirect is sending different output based on the user-agent, which breaks automation.|
|Steps To Reproduce||curl -v http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso|
curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" -L -O http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso
|Additional Information||Originally reported at https://github.com/hashicorp/packer/issues/8448|
|Packer's User Agent seems to be defined here: https://github.com/hashicorp/packer/blob/master/helper/useragent/useragent.go. Currently is starts with "Packer/".|
Well, yes, our isoredirect nodes will output a html list of mirrors in your location providing such iso (as you can see with your browser when you go to http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso)
I wasn't aware of the fact that Packer would download .iso to build artifact, so we only added wget/curl to the list of "agents" to just redirect to the first url automatically :
So adding Packer to the list and reload would fix your issue ?
I' just pushed that small change on our dev instance :
agent = request.environ.get('HTTP_USER_AGENT')[:5].lower()
- if (agent == "curl/" or agent == "wget/") and filename != "":
+ if (agent == "curl/" or agent == "wget/" or agent == "packer/") and filename != "":
fast_redirect = True
Can you point to isoredirect.dev.centos.org and give feedback ? If that works fine for you, I'll push it on all nodes.
PS : slow to respond as currently in travel mode
|I am traveling too, so, can test early next week. In the meantime, in Packer sources the user agent appears with capital P -- not sure if that makes a difference in your code. Thanks so far!|
haven't a chance to test it (did it quickly in the morning before having had breakfast), but you can see that agent is lower-cased ( .lower() function) so it should be indeed packer/ in the string comparison.
I'll try myself to give it a try but for sure we can add / fix this easily
|I think you are right, I missed that part, it ought to work.|
It still fails for me. Can it be that the change is not live yet? Upon inspection the downloaded file is a HTML file with my local mirrors. Perhaps you can see my Packer's User Agent on the dev isoredirect service? Thanks!
ISO_URL="http://isoredirect.dev.centos.org/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso" ISO_CHECKSUM="7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948" QCOW2_IMAGE="centos-8-x86_64" packer build packer.json
qemu output will be in this color.
==> qemu: Retrieving ISO
==> qemu: Trying http://isoredirect.dev.centos.org/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso
==> qemu: Trying http://isoredirect.dev.centos.org/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x86_64-boot.iso?checksum=7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948
CentOS-8.1.1911-x86_64-boot.iso 12.15 KiB / 12.15 KiB [============================================================================================================================================================================] 100.00% 0s
==> qemu: Checksum did not match, removing /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso
==> qemu: error downloading ISO: [Checksums did not match for /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso.
==> qemu: Expected: 7fea13202bf2f26989df4175aace8fdc16e1137f7961c33512cbfad844008948
==> qemu: Got: 933c1c5fa2b3716d4cd100eab228f4d3474e81925db522a18f7116ff3405d7f2
==> qemu: *sha256.digest]
Build 'qemu' errored: error downloading ISO: [Checksums did not match for /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso.
==> Some builds didn't complete successfully and had errors:
--> qemu: error downloading ISO: [Checksums did not match for /home/newman/isc/ws/bind9-images/packer/centos/packer_cache/90f663ae9e6efa846d32fef93d7a21fac2d5236a.iso.
==> Builds finished but no artifacts were created.
|Any idea what might have gone wrong?|
This also affects the ansible get_url module, which defaults to the agent string: ansible-httpget
As a workaround you can set the http_agent attribute to Wget/1.17.1, but it would be nice if the default http_agent worked.
Sorry for the late report here, but was caught into various fires for distro and infra so [RFE]s were stuck.
I just modified code and pushed to https://isoredirect.stg.centos.org so can you give it a try with real Packer and ansible get_url ?
What I tried on my side (simulated) and that worked :
curl -A "Packer/10.1" -L -O http://isoredirect.stg.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso -L
curl -A "ansible-httpget" -L -O http://isoredirect.stg.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso -L
If you can confirm that it also works for you, I'll push to master branch and ansible will deploy it
Thanks for your patience on this one ;-)
|Ok. It works with ansible get_url without setting the http_agent now. Thanks.|
Pushed the change on isoredirect.centos.org nodes
Verified also with 'curl -A "Packer/10.1" -O http://isoredirect.centos.org/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-DVD-1908.iso -L' and it worked (as tested in staging)
Closing for now
|2019-12-10 23:04||gtirloni||New Issue|
|2019-12-10 23:04||gtirloni||Tag Attached: iso|
|2020-01-23 13:21||mnowak||Note Added: 0036100|
|2020-01-24 05:42||arrfab||Note Added: 0036110|
|2020-01-24 05:43||arrfab||Status||new => feedback|
|2020-01-24 05:50||arrfab||Note Edited: 0036110||View Revisions|
|2020-01-24 09:56||mnowak||Note Added: 0036111|
|2020-01-24 17:27||arrfab||Note Added: 0036114|
|2020-01-24 20:57||mnowak||Note Added: 0036119|
|2020-01-29 10:57||mnowak||Note Added: 0036152|
|2020-02-21 08:54||mnowak||Note Added: 0036360|
|2020-03-12 15:41||cdbunch72||Note Added: 0036497|
|2020-03-12 15:43||cdbunch72||Tag Attached: ansible|
|2020-03-26 12:22||arrfab||Note Added: 0036586|
|2020-03-26 15:20||cdbunch72||Note Added: 0036587|
|2020-03-27 12:56||arrfab||Status||feedback => resolved|
|2020-03-27 12:56||arrfab||Resolution||open => fixed|
|2020-03-27 12:56||arrfab||Note Added: 0036592|