2017-10-18 05:42 UTC

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0013431CentOS-7curlpublic2017-06-15 18:59
Reporterrobertbuchanan 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
Product Version7.3.1611 
Target VersionFixed in Version 
Summary0013431: Curl not auto-detecting server-side compression
DescriptionRan across a problem with some SOAP requests in php. After some digging, it appears to be in one of the underlying libraries (the php soap library uses the php curl library), and when I try it with simple command-line utilities not associated with PHP (curl and wget), I get the same behavior. The server is sending the data compressed (accept is */*, by default), but curl is not detecting that and deflating it - it spits out the compressed file (which opens with gunzip, but I don't have that option inside the PHP SOAP library where I need it). If I manually add the --compressed flag to the curl command-line utility, the file shows up just fine. Checking on some older installs I have lying around, it seems to have started somewhere between CentOS 5 and CentOS 6.
Steps To Reproducecurl --version ; curl -v -o test1.file --compressed http://schemas.xmlsoap.org/soap/encoding/ ; curl -v -o test2.file http://schemas.xmlsoap.org/soap/encoding/ ; wget -O test3.file http://schemas.xmlsoap.org/soap/encoding/ ; ls -l test?.file
Additional InformationRunning:
curl --version ; curl -v -o test1.file --compressed http://schemas.xmlsoap.org/soap/encoding/ ; curl -v -o test2.file http://schemas.xmlsoap.org/soap/encoding/ ; wget -O test3.file http://schemas.xmlsoap.org/soap/encoding/ ; ls -l test?.file

CentOS 7 Spits out (broken behavior):
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.21 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0* About to connect() to schemas.xmlsoap.org port 80 (#0)
* Trying 65.54.226.188...
  0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0* Connected to schemas.xmlsoap.org (65.54.226.188) port 80 (#0)
> GET /soap/encoding/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: schemas.xmlsoap.org
> Accept: */*
> Accept-Encoding: deflate, gzip
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/xml
< Last-Modified: Mon, 27 Jun 2005 16:39:47 GMT
< Accept-Ranges: bytes
< ETag: "60d3d0d4367bc51:0"
< Server: Microsoft-IIS/7.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-Powered-By: ASP.NET
< Date: Wed, 14 Jun 2017 19:26:08 GMT
< ntCoent-Length: 19204
< Content-Encoding: gzip
< Content-Length: 4142
< Age: 70869
< X-Cache: HIT from 71.216.5.113
< X-Cache-Lookup: HIT from 71.216.5.113:3128
< Via: 1.0 71.216.5.113 (squid/3.1.23)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
<
{ [data not shown]
100 4142 100 4142 0 0 749 0 0:00:05 0:00:05 --:--:-- 1033
* Connection #0 to host schemas.xmlsoap.org left intact
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0* About to connect() to schemas.xmlsoap.org port 80 (#0)
* Trying 65.54.226.188...
  0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0* Connected to schemas.xmlsoap.org (65.54.226.188) port 80 (#0)
> GET /soap/encoding/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: schemas.xmlsoap.org
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/xml
< Last-Modified: Mon, 27 Jun 2005 16:39:47 GMT
< Accept-Ranges: bytes
< ETag: "60d3d0d4367bc51:0"
< Server: Microsoft-IIS/7.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-Powered-By: ASP.NET
< Date: Wed, 14 Jun 2017 19:26:08 GMT
< ntCoent-Length: 19204
< Content-Encoding: gzip
< Content-Length: 4142
< Age: 70874
< X-Cache: HIT from 71.216.5.113
< X-Cache-Lookup: HIT from 71.216.5.113:3128
< Via: 1.0 71.216.5.113 (squid/3.1.23)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
<
{ [data not shown]
100 4142 100 4142 0 0 750 0 0:00:05 0:00:05 --:--:-- 1033
* Connection #0 to host schemas.xmlsoap.org left intact
--2017-06-15 08:07:22-- http://schemas.xmlsoap.org/soap/encoding/
Resolving schemas.xmlsoap.org (schemas.xmlsoap.org)... 65.54.226.188
Connecting to schemas.xmlsoap.org (schemas.xmlsoap.org)|65.54.226.188|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4142 (4.0K) [text/xml]
Saving to: âtest3.fileâ

100%[=====================================================================================================================================================================================================================================>] 4,142 --.-K/s in 0s

2017-06-15 08:07:27 (272 MB/s) - âtest3.fileâ saved [4142/4142]

-rw-r--r--. 1 root root 19204 Jun 15 08:07 test1.file
-rw-r--r--. 1 root root 4142 Jun 15 08:07 test2.file
-rw-r--r--. 1 root root 4142 Jun 27 2005 test3.file


An older CentOS 6 box spits out (broken behavior):
curl 7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
* About to connect() to schemas.xmlsoap.org port 80 (#0)
* Trying 65.52.103.126... connected
* Connected to schemas.xmlsoap.org (65.52.103.126) port 80 (#0)
> GET /soap/encoding/ HTTP/1.1
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: schemas.xmlsoap.org
> Accept: */*
> Accept-Encoding: deflate, gzip
>
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/xml
< Last-Modified: Mon, 27 Jun 2005 16:39:47 GMT
< Accept-Ranges: bytes
< ETag: "60d3d0d4367bc51:0"
< Server: Microsoft-IIS/7.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-Powered-By: ASP.NET
< Date: Wed, 14 Jun 2017 19:26:08 GMT
< ntCoent-Length: 19204
< Content-Encoding: gzip
< Content-Length: 4142
< Age: 71586
< X-Cache: HIT from 71.216.5.113
< X-Cache-Lookup: HIT from 71.216.5.113:3128
< Via: 1.0 71.216.5.113 (squid/3.1.23)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
<
{ [data not shown]
101 4142 101 4142 0 0 810 0 0:00:05 0:00:05 --:--:-- 505k* Connection #0 to host schemas.xmlsoap.org left intact

* Closing connection #0
* About to connect() to schemas.xmlsoap.org port 80 (#0)
* Trying 65.52.103.126... connected
* Connected to schemas.xmlsoap.org (65.52.103.126) port 80 (#0)
> GET /soap/encoding/ HTTP/1.1
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: schemas.xmlsoap.org
> Accept: */*
>
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/xml
< Last-Modified: Mon, 27 Jun 2005 16:39:47 GMT
< Accept-Ranges: bytes
< ETag: "60d3d0d4367bc51:0"
< Server: Microsoft-IIS/7.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-Powered-By: ASP.NET
< Date: Wed, 14 Jun 2017 19:26:08 GMT
< ntCoent-Length: 19204
< Content-Encoding: gzip
< Content-Length: 4142
< Age: 71591
< X-Cache: HIT from 71.216.5.113
< X-Cache-Lookup: HIT from 71.216.5.113:3128
< Via: 1.0 71.216.5.113 (squid/3.1.23)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
<
{ [data not shown]
101 4142 101 4142 0 0 820 0 0:00:05 0:00:05 --:--:-- 674k* Connection #0 to host schemas.xmlsoap.org left intact

* Closing connection #0
--2017-06-15 08:19:19-- http://schemas.xmlsoap.org/soap/encoding/
Resolving schemas.xmlsoap.org... 65.52.103.126
Connecting to schemas.xmlsoap.org|65.52.103.126|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4142 (4.0K) [text/xml]
Saving to: âtest3.fileâ

100%[=====================================================================================================================================================================================================================================>] 4,142 --.-K/s in 0s

2017-06-15 08:19:24 (427 MB/s) - âtest3.fileâ

-rw-r--r--. 1 root root 19204 Jun 15 08:19 test1.file
-rw-r--r--. 1 root root 4142 Jun 15 08:19 test2.file
-rw-r--r--. 1 root root 4142 Jun 27 2005 test3.file


An older CentOS 5 box spits out (expected behavior):
curl 7.15.5 (i386-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
* About to connect() to schemas.xmlsoap.org port 80
* Trying 65.52.103.126... connected
* Connected to schemas.xmlsoap.org (65.52.103.126) port 80
> GET /soap/encoding/ HTTP/1.1
> User-Agent: curl/7.15.5 (i386-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: schemas.xmlsoap.org
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Content-Type: text/xml
< Content-Encoding: gzip
< Last-Modified: Mon, 27 Jun 2005 16:39:47 GMT
< Accept-Ranges: bytes
< ETag: "80b4ad4367bc51:0"
< Vary: Accept-Encoding
< Server: Microsoft-IIS/7.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-Powered-By: ASP.NET
< Date: Thu, 15 Jun 2017 15:15:48 GMT
< Content-Length: 3382
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 3382 100 3382 0 0 13481 0 --:--:-- --:--:-- --:--:-- 75225Connection #0 to host schemas.xmlsoap.org left intact

* Closing connection #0
* About to connect() to schemas.xmlsoap.org port 80
* Trying 65.54.226.188... connected
* Connected to schemas.xmlsoap.org (65.54.226.188) port 80
> GET /soap/encoding/ HTTP/1.1
> User-Agent: curl/7.15.5 (i386-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: schemas.xmlsoap.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/xml
< Last-Modified: Mon, 27 Jun 2005 16:39:47 GMT
< Accept-Ranges: bytes
< ETag: "60d3d0d4367bc51:0"
< Server: Microsoft-IIS/7.5
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< X-Powered-By: ASP.NET
< Date: Thu, 15 Jun 2017 15:15:48 GMT
< Content-Length: 19204
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 19204 100 19204 0 0 97260 0 --:--:-- --:--:-- --:--:-- 232k* Connection #0 to host schemas.xmlsoap.org left intact

* Closing connection #0
--08:15:48-- http://schemas.xmlsoap.org/soap/encoding/
Resolving schemas.xmlsoap.org... 65.54.226.188
Connecting to schemas.xmlsoap.org|65.54.226.188|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19204 (19K) [text/xml]
Saving to: `test3.file'

100%[====================================================================================================================================================================================================>] 19,204 --.-K/s in 0.06s

08:15:48 (299 KB/s) - `test3.file' saved [19204/19204]

-rw-r--r-- 1 root root 19204 Jun 15 08:15 test1.file
-rw-r--r-- 1 root root 19204 Jun 15 08:15 test2.file
-rw-r--r-- 1 root root 19204 Jun 27 2005 test3.file
Tags7.3-QA
abrt_hash
URL
Attached Files

-Relationships
+Relationships

-Notes

~0029496

robertbuchanan (reporter)

Apparently part of it is a malfunctioning proxy - the server isn't compressing it, the proxy was; the unaffected server was set to bypass the proxy. Still, server-side compression is a very common thing, and the fact that the data was sent compressed is clearly in the headers, so there's still a problem.
+Notes

-Issue History
Date Modified Username Field Change
2017-06-15 15:23 robertbuchanan New Issue
2017-06-15 15:23 robertbuchanan Tag Attached: 7.3-QA
2017-06-15 18:59 robertbuchanan Note Added: 0029496
+Issue History