> Did I get it right, according to the spec, the "Connection"-header is
> forbidden ("MUST NOT"), still, firefox does send it? This leads to the
> described issue.
I think it indeed might be the root cause, also for failed GET requests (which
only seems to happen sometimes?) but it’s really visible with PUT and POST
requests.
I’ve opened https://bugzilla.mozilla.org/show_bug.cgi?id=1427256 - so if by any
chance you can go comment with a “I face the same issue”, then Firefox might
pick it up faster.
> Firefox sends "Connection: keep-alive" while Chrome does not.
Correct, however – it seems like Safari also sends it, so in fact I have to
open a bug report to Safari as well (
Best Regards,
Lucas Rolff
On 28/12/2017, 12.08, "Maximilian Böhm" <[email protected]> wrote:
Sorry, for my long absence. Thank you, Lucas, for perfectly describing and
digging into the issue. I'll be here if there is any further assistance
required.
Did I get it right, according to the spec, the "Connection"-header is
forbidden ("MUST NOT"), still, firefox does send it? This leads to the
described issue.
Just checked it on https://http2.golang.org/.
Firefox sends "Connection: keep-alive" while Chrome does not.
>> I'd rather not fall into such idiocies, you see.
Thanks 😊 whereby, I'd rather prefer such idiocies instead installing
plugins without asking users (well, that's another topic, I guess..
https://www.theverge.com/2017/12/16/16784628/mozilla-mr-robot-arg-plugin-firefox-looking-glass
)
-----Ursprüngliche Nachricht-----
Von: Lucas Rolff [mailto:[email protected]]
Gesendet: Donnerstag, 28. Dezember 2017 11:27
An: Willy Tarreau <[email protected]>
Cc: [email protected]
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum
> It's normal then, as it's mandated by the HTTP/2 spec to reject
> requests containing any connection-specific header fields
In that case, haproxy should be consistent in it’s way of handling clients
sending connection-specific headers:
$ curl 'https://dashboard.domain.com/js/app.js?v=1' -H 'User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101
Firefox/57.0' --compressed -H 'Connection: keep-alive' -o /dev/null -vvv
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0* Trying 178.63.183.40...
* TCP_NODELAY set
* Connected to dashboard.domain.com (178.63.183.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [93 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
0* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3000 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=Domain Control Validated; CN=*.domain.com
* start date: Jan 3 11:17:55 2017 GMT
* expire date: Jan 4 11:17:55 2018 GMT
* subjectAltName: host "dashboard.domain.com" matched cert's "*.domain.com"
* issuer: C=BE; O=GlobalSign nv-sa; CN=AlphaSSL CA - SHA256 - G2
* SSL certificate verify ok.
> GET /js/app.js?v=1 HTTP/1.1
> Host: dashboard.domain.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0)
> Gecko/20100101 Firefox/57.0
> Connection: keep-alive
>
< HTTP/1.1 200 OK
< Server: nginx/1.13.5
< Date: Thu, 28 Dec 2017 10:11:34 GMT
< Content-Type: application/javascript; charset=utf-8 < Last-Modified: Sun,
25 Jun 2017 17:17:05 GMT < Transfer-Encoding: chunked < Vary: Accept-Encoding <
ETag: W/"594ff011-7b7"
< Content-Encoding: gzip
<
{ [695 bytes data]
100 683 0 683 0 0 3936 0 --:--:-- --:--:-- --:--:--
3925
* Connection #0 to host dashboard.domain.com left intact
Making a GET request (and sending Connection: keep-alive) continues to
work, and haproxy doesn’t handle it as malformed according to the specification.
So not sure if POST/PUT is handled differently within haproxy when it comes
to the connection header compared to GET requests.
I agree that if the specification says it cannot be done, it shouldn’t, but
then at least it should be consistent.
Fyi, in that case – nginx isn’t compliant with the http2 specification.
I’m creating a bug report with Mozilla to see if they can change the
behaviour of their browser to not send a Connection header at all, maybe this
will resolve the issue.
> I still have no idea what this "quantum" is by the way ;-)
It’s the 2017 version of “We released a better version of Firefox that is
faster”.. Let’s call it Firefox Quantum.
Haproxy 2.0 could be named “haproxy Quantum 1.5” just for the giggles.
Best Regards,
Lucas R
On 28/12/2017, 10.27, "Willy Tarreau" <[email protected]> wrote:
Hi Lucas,
On Thu, Dec 28, 2017 at 08:38:52AM +0000, Lucas Rolff wrote:
> It worked as it should, so I started adding more and more headers,
until I
> hit the culprit: -H "Connection: keep-alive" or -H "Connection:
close" (or
> even "Connection: test")
(...)
It's normal then, as it's mandated by the HTTP/2 spec to reject requests
containing any connection-specific header fields (Connection being the
first one) :
8.1.2.2. Connection-Specific Header Fields
HTTP/2 does not use the Connection header field to indicate
connection-specific header fields; in this protocol, connection-
specific metadata is conveyed by other means. An endpoint MUST NOT
generate an HTTP/2 message containing connection-specific header
fields; any message containing connection-specific header fields MUST
be treated as malformed (Section 8.1.2.6).
> I tried to replicate the issue in haproxy version 1.8.1, 1.8.2 and
latest
> commit from master - all with the same result, I also tried playing
around
> with the options of forceclose, http-server-close etc on both the
frontend
> and backend in haproxy, none of them seem to "fix" the issue.
That's normal, you don't even reach this step, as it dies while
decompressing
the request (right after hpack decoding just before conversion from H2
to H1).
> However in 1.8.2 I have 100% chance of replicating it using post
requests in
> Firefox and nghttp, where in 1.8.1 the issue in the majority of the
time
> works in Firefox and only have the few percentage failure rate.
There were so many state issues with 1.8.1 that it's not much
surprizing, it's
possible that some of them would fail differently.
> I haven't been able to replicate the issue in other than Firefox and
nghttp -
Clearly this means that this has to be reported to the Firefox team, as
it's
expected to break basically everywhere (or to help detect non-compliant
servers). I still have no idea what this "quantum" is by the way ;-)
> Also, sorry for the lengthy email
Quite the opposite, it was extremely helpful in spotting the problem's
origin.
Thanks!
Willy
