Hi there,
thanks for releasing support for http/2! Sadly, we are facing issues since
enabling it.
We are using HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04 on Debian 8. On the
backend, jetty 9.3.11.v20160721 with http/1.1 answers requests.
Since I've enabled http/2 ("alpn h2,http/1.1"), we are f
❦ 21 décembre 2017 09:00 GMT, Maximilian Böhm :
> We are using HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04 on Debian 8. On the
> backend, jetty 9.3.11.v20160721 with http/1.1 answers requests.
>
> Since I've enabled http/2 ("alpn h2,http/1.1"), we are facing issues with
> Firefox Quantum both,
at [mailto:ber...@luffy.cx]
Gesendet: Donnerstag, 21. Dezember 2017 11:04
An: Maximilian Böhm
Cc: haproxy@formilux.org
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum
❦ 21 décembre 2017 09:00 GMT, Maximilian Böhm :
> We are using HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04 on Debian 8. On the
I tried enabling “option httplog” within my frontend, I do have the same issue
wit
Can't even compose an email correctly..
So:
I experience the same issue however with nginx as a backend.
I tried enabling “option httplog” within my frontend, it's rather easy for me
to replicate, it affects a few percent of the traffic.
I have a site, with a total of 3 requests being perform
Hello Lucas,
On Wed, Dec 27, 2017 at 9:24 PM, Lucas Rolff wrote:
> Can't even compose an email correctly..
>
> So:
>
> I experience the same issue however with nginx as a backend.
>
> I tried enabling “option httplog” within my frontend, it's rather easy for
> me to replicate, it affects a few
aproxy@formilux.org
Subject: Re: HTTP/2 Termination vs. Firefox Quantum
Hello Lucas,
On Wed, Dec 27, 2017 at 9:24 PM, Lucas Rolff wrote:
> Can't even compose an email correctly..
>
> So:
>
> I experience the same issue however with nginx as a backend.
>
> I tried enabling “op
ure if it will actually fix
it there as well.
Also, sorry for the lengthy email
Best Regards,
Lucas Rolff
From: Lucas Rolff
Date: Wednesday, 27 December 2017 at 23.08
To: Lukas Tribus
Cc: "haproxy@formilux.org"
Subject: Re: HTTP/2 Termination vs. Firefox Quantum
My small site is basica
Hi Lucas,
On Thu, Dec 28, 2017 at 08:38:52AM +, Lucas Rolff wrote:
> It worked as it should, so I started adding more and more headers, until I
> hit the culprit: -H "Connection: keep-alive" or -H "Connection: close" (or
> even "Connection: test")
(...)
It's normal then, as it's mandated by
> It's normal then, as it's mandated by the HTTP/2 spec to reject requests
> containing any connection-specific header fields
In that case, haproxy should be consistent in it’s way of handling clients
sending connection-specific headers:
$ curl 'https://dashboard.domain.com/js/app.js?v=1' -H '
On Thu, Dec 28, 2017 at 10:27:28AM +, Lucas Rolff wrote:
> In that case, haproxy should be consistent in it's way of handling clients
> sending connection-specific headers:
>
> $ curl 'https://dashboard.domain.com/js/app.js?v=1' -H 'User-Agent:
> Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13;
Sorry regarding my previous curl – I didn’t use --http2 in my curl request, but
result is the same (with negotiated http2 protocol), I’ve removed the TLSv1.2
output since it’s useless in this case:
===
$ curl 'https://dashboard.domain.com/js/app.js?v=1' -H 'User-
r 2017 11:27
An: Willy Tarreau
Cc: haproxy@formilux.org
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum
> It's normal then, as it's mandated by the HTTP/2 spec to reject
> requests containing any connection-specific header fields
In that case, haproxy should be consistent in it
s (well, that's another topic, I guess..
https://www.theverge.com/2017/12/16/16784628/mozilla-mr-robot-arg-plugin-firefox-looking-glass
)
-Ursprüngliche Nachricht-
Von: Lucas Rolff [mailto:lu...@lucasrolff.com]
Gesendet: Donnerstag, 28. Dezember 2017 11:27
Hello,
> But in this example, you're using HTTP/1.1, The "Connection" header is
> perfectly valid for 1.1. It's HTTP/2 which forbids it. There is no
> inconsistency here.
For me a request like this:
$ curl -kv --http2 https://localhost/111 -H "Connection: keep-alive"
-d "bla=bla"
Fired multiple
Hello,
On Thu, Dec 28, 2017 at 12:29 PM, Lukas Tribus wrote:
> Hello,
>
>
>> But in this example, you're using HTTP/1.1, The "Connection" header is
>> perfectly valid for 1.1. It's HTTP/2 which forbids it. There is no
>> inconsistency here.
>
> For me a request like this:
> $ curl -kv --http2 h
> the output of the http2 golang test and can you please both clarify which OS
> you reproduce this on?
If I visit http2 golang test, I also don’t see it, and I saw it in developer
tools (Because dev tools shouldn’t put headers that isn’t requested/received) –
however based on your findings, th
Hello,
On Thu, Dec 28, 2017 at 10:26 PM, Lucas Rolff wrote:
>> the output of the http2 golang test and can you please both clarify which OS
>> you reproduce this on?
>
> If I visit http2 golang test, I also don’t see it, and I saw it in developer
> tools (Because dev tools shouldn’t put header
Hi Lukas,
On Thu, Dec 28, 2017 at 09:19:24PM +0100, Lukas Tribus wrote:
> On Thu, Dec 28, 2017 at 12:29 PM, Lukas Tribus wrote:
> > Hello,
> >
> >
> >> But in this example, you're using HTTP/1.1, The "Connection" header is
> >> perfectly valid for 1.1. It's HTTP/2 which forbids it. There is no
>
Hi Willy,
> In fact it's a race between the GOAWAY frame caused by the invalid request,
> and the HEADERS frame being sent in response to the stream being closed
> I agree that it's quite confusing, but we're talking about responses to
> conditions that are explicitly forbidden in the spec, so I
Hi Lucas,
On Fri, Dec 29, 2017 at 06:06:49AM +, Lucas Rolff wrote:
> As much as I agree about that specs should be followed, I realized that even
> if there's people that want to follow the spec 100%, there will always be
> implementations used in large scale that won't be following the spec 1
> Yep. For what it's worth, it's been enabled for about one month on
> haproxy.org and till now we didn't get any bad report, which is pretty
> encouraging.
Can I ask where? The negotiated protocol I get on https://haproxy.org/ is
http/1.1 in both Google Chrome and Firefox as an example.
If I
On Fri, Dec 29, 2017 at 08:46:18AM +, Lucas Rolff wrote:
> > Yep. For what it's worth, it's been enabled for about one month on
> > haproxy.org and till now we didn't get any bad report, which is pretty
> > encouraging.
>
> Can I ask where? The negotiated protocol I get on https://haproxy.or
Hello,
On Fri, Dec 29, 2017 at 8:13 AM, Willy Tarreau wrote:
> Yep. For what it's worth, it's been enabled for about one month on haproxy.org
> and till now we didn't get any bad report, which is pretty encouraging.
It appears to run 1.7.5 though:
http://demo.haproxy.org/
>> For now, I'll p
On Fri, Dec 29, 2017 at 11:22:56AM +0100, Lukas Tribus wrote:
> Hello,
>
>
> On Fri, Dec 29, 2017 at 8:13 AM, Willy Tarreau wrote:
> > Yep. For what it's worth, it's been enabled for about one month on
> > haproxy.org
> > and till now we didn't get any bad report, which is pretty encouraging.
>
> Actually that's not the case and that may explain the situation. The machine
> runs OpenSSL 1.0.1 so only NPN is used, ALPN isn't. I'll try with a static
> build of openssl 1.0.2 to see if the ratio increases.
That might very well be the case, I know for sure that Chrome dropped support
NPN a
Hello,
On Fri, Dec 29, 2017 at 11:22 AM, Lukas Tribus wrote:
> It's that:
> - when sending the POST request to the backend server, haproxy sends a
> FIN before the server responds
> - nginx doesn't like that and closes the request (you will see nginx
> error code 499 in nginx server logs)
> - as
> Lucas, can you check my previous mail and see if you can enable ignoring
> client aborts in your backend, assuming you are using nginx?
I can confirm that ignoring client aborts in my backend using
fastcgi_ignore_client_abort “resolves” the issue regarding POST requests.
Best Regards,
Lucas R
Hi Lukas,
On Fri, Dec 29, 2017 at 11:45:55AM +0100, Lukas Tribus wrote:
> On Fri, Dec 29, 2017 at 11:22 AM, Lukas Tribus wrote:
> > It's that:
> > - when sending the POST request to the backend server, haproxy sends a
> > FIN before the server responds
> > - nginx doesn't like that and closes the
On Fri, Dec 29, 2017 at 11:45:55AM +0100, Lukas Tribus wrote:
> The FIN behavior comes from a48c141f4 ("BUG/MAJOR: connection: refine
> the situations where we don't send shutw()"), which also hit 1.8.2, so
> that explains the change in behavior between 1.8.1 and 1.8.2.
For me it happens only when
Hello,
On Fri, Dec 29, 2017 at 2:31 PM, Willy Tarreau wrote:
> On Fri, Dec 29, 2017 at 11:45:55AM +0100, Lukas Tribus wrote:
>> The FIN behavior comes from a48c141f4 ("BUG/MAJOR: connection: refine
>> the situations where we don't send shutw()"), which also hit 1.8.2, so
>> that explains the ch
On Fri, Dec 29, 2017 at 02:52:39PM +0100, Lukas Tribus wrote:
> > For me it happens only when I have "option httpclose" in the configuration,
> > ie we end up in tunnel mode. I can't reproduce it with either keep-alive,
> > http-server-close nor forceclose. At least abortonclose is now safe
> > reg
Hello,
On Fri, Dec 29, 2017 at 3:05 PM, Willy Tarreau wrote:
>> Haproxy calls shutdown() after the HTTP payload was transmitted, nginx
>> in the default configuration or nc for that matter closes the
>> connection (we see recvfrom = 0) and then we close():
>
> I can't reproduce this one for now.
On Fri, Dec 29, 2017 at 03:26:44PM +0100, Lukas Tribus wrote:
> Indeed when the frontend connection is H1, a single send() call
> contains both headers and payload and the issue does not occur. But
> when the frontend connection is H2, then header and payload are in 2
> distinct send calls and the
On Fri, Dec 29, 2017 at 03:30:41PM +0100, Willy Tarreau wrote:
> On Fri, Dec 29, 2017 at 03:26:44PM +0100, Lukas Tribus wrote:
> > Indeed when the frontend connection is H1, a single send() call
> > contains both headers and payload and the issue does not occur. But
> > when the frontend connection
On Fri, Dec 29, 2017 at 03:42:30PM +0100, Willy Tarreau wrote:
> OK I managed to reproduce it with nghttp using --expect-continue to
> force it to leave a pause before sending the data. And indeed there
> the data are immediately followed by a shutdown. Getting closer...
So here's what I found : w
Both in Firefox and Chrome my POST requests in 1.8.2 with the supplied patch,
seem to do the trick (did about 300 post requests in each browser with no
fails).
Best Regards,
On 29/12/2017, 15.58, "Willy Tarreau" wrote:
On Fri, Dec 29, 2017 at 03:42:30PM +0100, Willy Tarreau wrote:
> O
Hi Willy,
On Fri, Dec 29, 2017 at 3:58 PM, Willy Tarreau wrote:
> On Fri, Dec 29, 2017 at 03:42:30PM +0100, Willy Tarreau wrote:
>> OK I managed to reproduce it with nghttp using --expect-continue to
>> force it to leave a pause before sending the data. And indeed there
>> the data are immediate
On Fri, Dec 29, 2017 at 04:46:57PM +0100, Lukas Tribus wrote:
> On Fri, Dec 29, 2017 at 3:58 PM, Willy Tarreau wrote:
> > On Fri, Dec 29, 2017 at 03:42:30PM +0100, Willy Tarreau wrote:
> >> OK I managed to reproduce it with nghttp using --expect-continue to
> >> force it to leave a pause before se
> Lucas, Maximilian, can you check the situation with this patch? The POST
> issue should definitely be gone, please also verify the GET issue with this
> patch (as I was unable to reproduce it).
Sadly didn’t fix the GET request issue for me in Firefox:
https://snaps.hcdn.dk/h1Oz3G950oepEb30AEo
On Fri, Dec 29, 2017 at 04:10:12PM +, Lucas Rolff wrote:
> > Lucas, Maximilian, can you check the situation with this patch? The POST
> > issue should definitely be gone, please also verify the GET issue with this
> > patch (as I was unable to reproduce it).
>
> Sadly didn't fix the GET requ
> If you're willing to run another test, I can prepare a debugging patch which
> will try to report every single error path in the H2 and HPACK code so that
> we can try to understand where the code was upset
I’d love to run another test or 10 – in the end, we’ll all benefit from it
(hopefully)
On Fri, Dec 29, 2017 at 04:48:13PM +, Lucas Rolff wrote:
> > If you're willing to run another test, I can prepare a debugging patch
> > which will try to report every single error path in the H2 and HPACK code
> > so that we can try to understand where the code was upset
>
> I'd love to run
Working page load (total of 4 requests), and we see 4x 3180|3067
POST Request (to website):
h2s_frt_make_resp_data:3180
h2s_frt_make_resp_data:3067
-
GET Request (to website):
h2s_frt_make_resp_data:3180
h2s_frt_make_resp_data:3067
-
GET Request (app.css)
h2s_frt_make_resp_data:3180
h2s_frt_make_r
On Fri, Dec 29, 2017 at 06:02:15PM +, Lucas Rolff wrote:
> POST Request (to website):
> h2s_frt_make_resp_data:3180
> h2s_frt_make_resp_data:3067
> -
> GET Request (to website):
> h2s_frt_make_resp_data:3180
> h2s_frt_make_resp_data:3067
> -
> Get Request (app.css)
> h2_frt_decode_headers:2621
I think you forgot to attach the patch
I did try to add the continue patch for the “blacklisted” headers, and same
result (now in this case, it happened on the website request itself:
POST Request to site:
h2s_frt_make_resp_data:3180
h2s_frt_make_resp_data:3067
GET Request to site:
h2_frt_decod
On Fri, Dec 29, 2017 at 06:18:00PM +, Lucas Rolff wrote:
> I think you forgot to attach the patch
Grrr common mistake, sorry.
> I did try to add the continue patch for the "blacklisted" headers, and same
> result (now in this case, it happened on the website request itself:
Thanks, I'll tak
h2_make_h1_request:153
h2_frt_decode_headers:2621
h2_frt_decode_headers:2643
/* this can be any type of header */
/* RFC7540#8.1.2: upper case not allowed in header field names */
for (i = 0; i < list[idx].n.len; i++)
if ((uint8_t)(list[idx].n.ptr[i] - 'A') < 'Z' - 'A')
got
On Fri, Dec 29, 2017 at 06:56:36PM +, Lucas Rolff wrote:
> h2_make_h1_request:153
> h2_frt_decode_headers:2621
> h2_frt_decode_headers:2643
>
> /* this can be any type of header */
> /* RFC7540#8.1.2: upper case not allowed in header field names */
> for (i = 0; i < list[idx].n.len; i++)
>
On Fri, Dec 29, 2017 at 06:56:36PM +, Lucas Rolff wrote:
> h2_make_h1_request:153
> h2_frt_decode_headers:2621
> h2_frt_decode_headers:2643
>
> /* this can be any type of header */
> /* RFC7540#8.1.2: upper case not allowed in header field names */
> for (i = 0; i < list[idx].n.len; i++)
>
<<
:authority: dashboard.domain.com
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0)
Gecko/20100101 Firefox/59.0
accept: text/css,*/*;q=0.1
accept-language: da,en-US;q=0.8,en;q=0.6,es;q=0.4,tr;q=0.2
accept-encoding: gzip, deflate, br
referer: https://dashboard.dom
On Fri, Dec 29, 2017 at 08:45:57PM +, Lucas Rolff wrote:
> So, this Wx1ZSI usually should be "cookie" - however it's somehow turned into
> garbage.
Ah, this is what I was wondering.
> Repeated - now it's s3U2JV - but still supposed to be "cookie":
Great, so the number of characters is correc
Just a quick update for all those following this thread. Thanks to Lucas'
traces, we could finally fix the problem in the hpack decoder. I'm now
releasing 1.8.3 which should be much more usable with H2.
Willy
proxy@formilux.org
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum
Just a quick update for all those following this thread. Thanks to Lucas'
traces, we could finally fix the problem in the hpack decoder. I'm now
releasing 1.8.3 which should be much more usable with H2.
Willy
I’ve tested the 1.8.3 build, and I can indeed confirm it works like charm!
@Willy, thanks for the extensive time you spend on debugging and investigating
this as well!
Best Regards,
Lucas Rolff
On Sun, Dec 31, 2017 at 07:22:26AM +, Lucas Rolff wrote:
> I've tested the 1.8.3 build, and I can indeed confirm it works like charm!
Great, thank you for confirming. We're making progress :-)
> @Willy, thanks for the extensive time you spend on debugging and
> investigating this as well!
Yo
56 matches
Mail list logo
