Hi folks,
I am experiencing an issue which I believe is an already reported bug in
HTTPd, but I am not 100% sure. I first observed this with py-requests
which does not support the Expect header, but was also able to reproduce
with curl too.
My setup is running several Tomcat instances behind HTTPd via
mod_proxy_http:
VersionLoggerListener.log Server version name: Apache Tomcat/8.5.53
VersionLoggerListener.log Server built: Mar 11 2020 10:01:39 UTC
VersionLoggerListener.log Server version number: 8.5.53.0
VersionLoggerListener.log OS Name: FreeBSD
VersionLoggerListener.log OS Version: 12.1-STABLE
VersionLoggerListener.log Architecture: amd64
VersionLoggerListener.log Java Home: /usr/local/openjdk8/jre
VersionLoggerListener.log JVM Version: 1.8.0_242-b07
VersionLoggerListener.log JVM Vendor: Oracle Corporation
and
Apache HTTPd 2.4.43
(Log output will be replaced with ellipsis for brevity)
Find catalina.out and httpd.log at
http://home.apache.org/~michaelo/issues/502-broken-pipe/
Now consider the following requests, 443 via HTTP forward proxy, 11111
directly to Tomcat:
$ curl -X POST --anyauth -u : --upload-file inputs.zip -H "Content-Type: application/zip"
https://<hostname>/content-dev/api/documents --verbose -H "Expect:"
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com
.siemens.de'
* Trying <ip>:443...
* Connected to <hostname> (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=DE; O=Siemens; OU=LDA DW; CN=<hostname>
* start date: Mar 12 09:50:16 2020 GMT
* expire date: Mar 19 13:10:13 2021 GMT
* subjectAltName: host "<hostname>" matched cert's "<hostname>"
* issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7;
OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
* SSL certificate verify ok.
POST /content-dev/api/documents HTTP/1.1
Host: <hostname>
User-Agent: curl/7.70.0
Accept: */*
Content-Type: application/zip
Content-Length: 7664149
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< Date: Wed, 20 May 2020 08:27:45 GMT
< Server: Apache/2.4.43 (FreeBSD) OpenSSL/1.1.1d-freebsd mod_auth_gssapi/1.6.1
< X-Frame-Options: SAMEORIGIN
< Correlation-Id: XsTqAS8xDsem5B3tl8pp0wAAAJI
< Content-Length: 374
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Bad Gateway</title>
</head><body>
<h1>Bad Gateway</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
</p>
<hr>
<address>Apache/2.4.43 (FreeBSD) OpenSSL/1.1.1d-freebsd mod_auth_gssapi/1.6.1 Server at
<hostname> Port 443</address>
</body></html>
* Connection #0 to host <hostname> left intact
and
$ curl -X POST --anyauth -u : --upload-file inputs.zip -H "Content-Type: application/zip"
https://<hostname>:11111/content-dev/api/documents --verbose -H "Expect:"
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com
.siemens.de'
* Trying <ip>:11111...
* Connected to <hostname> (<ip>) port 11111 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; O=Siemens; OU=LDA DW; CN=<hostname>
* start date: Mar 12 09:50:16 2020 GMT
* expire date: Mar 19 13:10:13 2021 GMT
* subjectAltName: host "<hostname>" matched cert's "<hostname>"
* issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7;
OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
* SSL certificate verify ok.
POST /content-dev/api/documents HTTP/1.1
Host: <hostname>:11111
User-Agent: curl/7.70.0
Accept: */*
Content-Type: application/zip
Content-Length: 7664149
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 437
< Date: Wed, 20 May 2020 08:28:16 GMT
* NEGOTIATE send, close instead of sending 7598613 bytes
<
* Excess found: excess = 437 url = /content-dev/api/documents (zero-length body)
* Closing connection 0
* Issue another request to this URL:
'https://<hostname>:11111/content-dev/api/documents'
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com
.siemens.de'
* Hostname <hostname> was found in DNS cache
* Trying <ip>:11111...
* Connected to <hostname> (<ip>) port 11111 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* SSL re-using session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; O=Siemens; OU=LDA DW; CN=<hostname>
* start date: Mar 12 09:50:16 2020 GMT
* expire date: Mar 19 13:10:13 2021 GMT
* subjectAltName: host "<hostname>" matched cert's "<hostname>"
* issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7;
OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
* SSL certificate verify ok.
* Server auth using Negotiate with user ''
POST /content-dev/api/documents HTTP/1.1
Host: <hostname>:11111
Authorization: Negotiate YIIRjQYG^...
User-Agent: curl/7.70.0
Accept: */*
Content-Type: application/zip
Content-Length: 7664149
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 201
< WWW-Authenticate: Negotiate oYG3MIG0o...
< vary: Origin
< Access-Control-Allow-Credentials: true
< Access-Control-Expose-Headers:
Correlation-Id,Content-Disposition,Content-Length,Location
< Location:
https://<hostname>:11111/content-dev/api/documents/68552d0c-8f0b-4631-a14a-614d7bd7469d
< Content-Length: 0
< Date: Wed, 20 May 2020 08:28:17 GMT
<
* Connection #1 to host <hostname> left intact
As you can see the request is failing via mod_proxy_http with a 502.
After a few hours of investigation and reading log files it came to me
that mod_proxy_http does not read the response from Tomcat while it
sends the request body because Tomcat has already decided to abort the
request with 401.
See log files: What suprises me that
Http11InputBuffer.parseRequestLine() not only prints the request
headers, but also the entire body (misbehavior of mod_proxy_http?!). As
soon as I increase maxSwallowSize the symptom is gone until the new
threshold.
Similar issues:
* https://bz.apache.org/bugzilla/show_bug.cgi?id=64016
* https://bz.apache.org/bugzilla/show_bug.cgi?id=61090
* https://bz.apache.org/bugzilla/show_bug.cgi?id=60717
* https://github.com/eclipse/jetty.project/issues/651
My questions are:
* Why is parseRequestLine printing the request body?
* Can someone confirm the erratic behavior of HTTPd's module?
* Is there anyting we need to improve on the Tomcat side?
The more I use HTTPd with mod_proxy_http the more I notice that the has
been focus on websites and not real API usages.
Regards,
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
