View Issue Details

IDProjectCategoryView StatusLast Update
0016151CentOS-7httpdpublic2019-06-05 15:22
Reporterderekmpage 
PriorityimmediateSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0016151: HEAD request with a 404 and custom ErrorPage causes corrupt and mixed-up responses
DescriptionNote we are unable to reproduce this behavior in 2.4.39 <-

We have the latest httpd package from Oficial CentOS repo.
httpd-2.4.6-88.el7.centos.x86_64

https://bz.apache.org/bugzilla/show_bug.cgi?id=63461

The setup uses mod_proxy and custom 404 ErrorPage that is served by Tomcat via http. When HEAD request is made to Apache and results in 404, Apache makes GET request to the custom ErrorPage url on Tomcat that is returning a fairly large html page. At this point the headers Apache returns to the original caller are all mixed-up, it seems to serve data from some other requests and it all feels like some buffer overflow, it also corrupts other http requests that are executing concurrently on the Apache instance and they start returning garbage as well. When 404 HEAD requests are stopped it all goes back to normal.
Steps To ReproduceCreate a errorDocument 404 and serve a custom Error page via mod_proxy.

Generate a bunch of HEAD requests to pages that do not exist.

[dpage@chupacabra ~]$ curl --head -i http://apache01.mydomain.com/ssss
HTTP/1.1 404
Date: Wed, 05 Jun 2019 14:59:24 GMT
Server: KAYAK/1.0
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin
cache-control: no-store
vary: accept-encoding
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Transfer-Encoding: chunked

[dpage@chupacabra ~]$ curl --head -i http://apache01.mydomain.com/ssss
HTTP/1.1 200 OK
Date: Wed, 05 Jun 2019 14:59:24 GMT
Server: Apache
Vary: Host
Content-Length: 4

Additional InformationThe reason for putting the severity and Priority high is my concern of potential DDoS. We have already blocked this on our site by converting all HEAD requests to GET requests. *** I have hidden the Identity of our website for security reasons.

In a nutshell:

We started seeing gibberish / file download dialogs / broken pages in our environment.

To show our users slick heavily branded error pages we don't just throw a 404 when something is missing but generate a full-blown document instead inside Java, e.g., https://www.example.com/something/run/errors/404

This is done using "ErrorDocument 404" in Apache. Now the odd part: This mechanism sends a GET response to a HEAD, it omits the "Content-Length" header by doing though, and this seems to mess things up inside Apache so badly that wires get crossed and different HTTP streams get arbitrarily mixed up.

We first noticed this when we we enabled a job that run a high rate of HEAD requests to verify image existence on out site. This resulted in a large scale web site being brought down one one machine running a python script.
TagsNo tags attached.
abrt_hash
URL

Activities

TrevorH

TrevorH

2019-06-05 15:12

manager   ~0034606

Please update to the latest httpd package - currently httpd-2.4.6-89.el7.centos.x86_64

There is one fix listed in -89 over -88:

* Fri Mar 15 2019 Joe Orton <jorton@redhat.com> - 2.4.6-89
- fix per-request leak of bucket brigade structure (#1583218)

If that does not fix it then you need to report this on bugzilla.redhat.com as CentOS does not fix upstream bugs (we aim to rebuild bug for bug wrt RHEL).
derekmpage

derekmpage

2019-06-05 15:22

reporter   ~0034607

I can still reproduce in

httpd-2.4.6-89.el7.centos.x86_64 - I will report to redhat.

Issue History

Date Modified Username Field Change
2019-06-05 15:03 derekmpage New Issue
2019-06-05 15:12 TrevorH Note Added: 0034606
2019-06-05 15:22 derekmpage Note Added: 0034607