Re: H2: interoperability issue due to lack of CONTINUATION frame support
Hello,
On Sat, 1 Sep 2018 at 20:02, Lukas Tribus wrote:
>
> Hi Willy,
>
>
> haproxy is currently unable to handle CONTINUATION [1] frames (see
> commit 61290ec77 - [2]).
>
> If a client emits a CONTINUATION frame, we will break the connection
> and send GOAWAY due to INTERNAL_ERROR. This of course leads to
> interoperability issues.
>
> Notably, older Chrome/Chromium releases (50.0.2645.4 and older),
> lacking Chromium commit a8bff211e ("Increase sent control frame
> fragment size to 16 kB") [3], will cap the maximum control frame size
> at 1024 bytes, triggering CONTINUATION, when large headers are used
> (long URIs, or large cookies).
For the record/archives:
CONTINUATION is supported since v1.9.2 and it does fix this issue.
lukas
Re: H2: interoperability issue due to lack of CONTINUATION frame support
On Sun, Sep 02, 2018 at 09:24:33PM +0200, Lukas Tribus wrote:
> Hello,
>
>
> On Sun, 2 Sep 2018 at 17:24, Willy Tarreau wrote:
> >
> > Hi Lukas,
> >
> > On Sun, Sep 02, 2018 at 11:55:29AM +0200, Lukas Tribus wrote:
> > > Ok. I think with OpenSSL 1.1.1 we may be able to configure ALPN
> > > differently for RSA vs ECC certificates (of the same hostname), so by
> > > not enabling h2 on RSA certificates, we basically disable H2 for
> > > Chrome on Windows XP (Chrome using Microsoft's schannel supporting
> > > only RSA on XP). Chrome on Windows Vista would still be broken (as
> > > schannel on Vista supports ECC certificates), but the market share of
> > > Vista is probably negligible. This should help those that cannot break
> > > this unsupported browser/OS combination and still want to use H2. It's
> > > just a theory though at the moment, I need to test it.
> >
> > I like the idea very much! That's indeed something that could be
> > interesting to study. I even think there's nothing about it that
> > cannot be done with 1.0.2, it would deserve a test!
>
> I tried it and it works fine, crt-list looking like this:
> /etc/private/ssl/sitecert-rsa.pem [alpn http/1.1]
> /etc/private/ssl/sitecert-ecc.pem [alpn h2,http/1.1]
>
> However, openssl 1.1.1 (or boringssl) is required for this, also see
> commit 84e417d85934 ("MINOR: ssl: support Openssl 1.1.1 early callback
> for switchctx"). I doubt this can be done with older openssl.
Ah, I guess it's related to the fact that both carry the same name and
only differ by the key algorithm. Anyway that's a very interesting
approach, I think it would be worth writing an article about it
somewhere.
Cheers,
Willy
Re: H2: interoperability issue due to lack of CONTINUATION frame support
Hello,
On Sun, 2 Sep 2018 at 17:24, Willy Tarreau wrote:
>
> Hi Lukas,
>
> On Sun, Sep 02, 2018 at 11:55:29AM +0200, Lukas Tribus wrote:
> > Ok. I think with OpenSSL 1.1.1 we may be able to configure ALPN
> > differently for RSA vs ECC certificates (of the same hostname), so by
> > not enabling h2 on RSA certificates, we basically disable H2 for
> > Chrome on Windows XP (Chrome using Microsoft's schannel supporting
> > only RSA on XP). Chrome on Windows Vista would still be broken (as
> > schannel on Vista supports ECC certificates), but the market share of
> > Vista is probably negligible. This should help those that cannot break
> > this unsupported browser/OS combination and still want to use H2. It's
> > just a theory though at the moment, I need to test it.
>
> I like the idea very much! That's indeed something that could be
> interesting to study. I even think there's nothing about it that
> cannot be done with 1.0.2, it would deserve a test!
I tried it and it works fine, crt-list looking like this:
/etc/private/ssl/sitecert-rsa.pem [alpn http/1.1]
/etc/private/ssl/sitecert-ecc.pem [alpn h2,http/1.1]
However, openssl 1.1.1 (or boringssl) is required for this, also see
commit 84e417d85934 ("MINOR: ssl: support Openssl 1.1.1 early callback
for switchctx"). I doubt this can be done with older openssl.
Regards,
Lukas
Re: H2: interoperability issue due to lack of CONTINUATION frame support
Hi Lukas, On Sun, Sep 02, 2018 at 11:55:29AM +0200, Lukas Tribus wrote: > Ok. I think with OpenSSL 1.1.1 we may be able to configure ALPN > differently for RSA vs ECC certificates (of the same hostname), so by > not enabling h2 on RSA certificates, we basically disable H2 for > Chrome on Windows XP (Chrome using Microsoft's schannel supporting > only RSA on XP). Chrome on Windows Vista would still be broken (as > schannel on Vista supports ECC certificates), but the market share of > Vista is probably negligible. This should help those that cannot break > this unsupported browser/OS combination and still want to use H2. It's > just a theory though at the moment, I need to test it. I like the idea very much! That's indeed something that could be interesting to study. I even think there's nothing about it that cannot be done with 1.0.2, it would deserve a test! Cheers, Willy
Re: H2: interoperability issue due to lack of CONTINUATION frame support
Hello Willy, On Sat, 1 Sep 2018 at 21:00, Willy Tarreau wrote: > I wanted to address it but the CONTINUATION frame is the worst design > mistake of the H2 protocol and results in layering violations which > make it particularly problematic to implement. In short, while all > frames are independant on each other, this one must absolutely appear > after an existing HEADERS frame and requires to maintain a huge state > of unparsed data to feed to the HPACK decompressor at once after > reassembly. It's even documented in the spec that it can be the source > of a DoS... I see, that's about the same nightmare as IPv4 fragmentation, where supposedly completely stateful routers have to reassemble the IPv4 fragments on MTU mismatched interfaces or to make L4 ACL's actually match the packets. Rethinking about this I don't actually think there a lot of other corner cases triggering CONTINUATION, other than the Chrome 49 case. > And that's sad. It would have been better if most didn't implement this > crap, to get rid of the non-legitimate use cases once for all :-/ > > I will probably revisit this point after 1.9, at least for protocol > completeness, maybe with an option such as > "waste-memory-and-cpu-to-support-stupid-h2-continuation-frames" or > something as explicit to save people from enabling them by accident :-/ Ok. I think with OpenSSL 1.1.1 we may be able to configure ALPN differently for RSA vs ECC certificates (of the same hostname), so by not enabling h2 on RSA certificates, we basically disable H2 for Chrome on Windows XP (Chrome using Microsoft's schannel supporting only RSA on XP). Chrome on Windows Vista would still be broken (as schannel on Vista supports ECC certificates), but the market share of Vista is probably negligible. This should help those that cannot break this unsupported browser/OS combination and still want to use H2. It's just a theory though at the moment, I need to test it. Regards, Lukas
Re: H2: interoperability issue due to lack of CONTINUATION frame support
Hi Lukas,
On Sat, Sep 01, 2018 at 08:02:45PM +0200, Lukas Tribus wrote:
> Hi Willy,
>
>
> haproxy is currently unable to handle CONTINUATION [1] frames (see
> commit 61290ec77 - [2]).
>
> If a client emits a CONTINUATION frame, we will break the connection
> and send GOAWAY due to INTERNAL_ERROR. This of course leads to
> interoperability issues.
Yes, but arguably these issues should be rare because a client is
expected to fill its frames, so all requests smaller than 16 kB
compressed make no sense to be sent with CONTINUATION.
> Notably, older Chrome/Chromium releases (50.0.2645.4 and older),
> lacking Chromium commit a8bff211e ("Increase sent control frame
> fragment size to 16 kB") [3], will cap the maximum control frame size
> at 1024 bytes, triggering CONTINUATION, when large headers are used
> (long URIs, or large cookies).
Hmmm this is a particularly abusive behaviour considering that the
sole purpose of the CONTINUATION frames was to support extremely
large headers (and these ones were debated a lot) :-(
> How to reproduce?
> In Chrome/Chromium 50.0.2645.4 or older, access a very long URI on a
> h2 enabled haproxy instance. For example on cdn.haproxy.com [4].
>
> Why are old Chrome/Chromium relevant?
> Chrome 49 is the latest release that runs under Windows XP. Yes, I
> know, but apparently there are large market segments out there where
> this has a real business impact.
Sure!
> Chrome 49 and Windows XP support
> situation a side, this is not a Chrome bug, but a lack of support of a
> H2 feature in haproxy. Other corner cases may trigger this issue as
> well.
Definitely.
> At least 4 different people reported this as an actual
> interoperability problem with H2 on discourse [5], [6], additionally
> we have 2 reports of results of a RFC7540 test [7], [8], which also
> fails due to at least lack of CONTINUATION support.
>
> While this issue is arguably not major - there would have been more
> reports otherwise, I do think we should address this issue.
I wanted to address it but the CONTINUATION frame is the worst design
mistake of the H2 protocol and results in layering violations which
make it particularly problematic to implement. In short, while all
frames are independant on each other, this one must absolutely appear
after an existing HEADERS frame and requires to maintain a huge state
of unparsed data to feed to the HPACK decompressor at once after
reassembly. It's even documented in the spec that it can be the source
of a DoS... I'm not saying I don't want to implement them at all, it's
just that in the current state of things, I don't have a clean solution
to propose. We'll have more options once the connection layers are
reversed because we'll have the option to try to parse both HEADERS
and CONTINUATION at once into a temporary buffer and try to decompress
the result. But in any case, CONTINUATION frames support will always
be limited (ie too large HEADERS+CONTINUATION may lead to connection
aborts by placing the decompressor in an unrecoverable state).
> As far as I can see, other major h2 implementations do implement
> CONTINUATION support, including nginx and nghttp2.
And that's sad. It would have been better if most didn't implement this
crap, to get rid of the non-legitimate use cases once for all :-/
I will probably revisit this point after 1.9, at least for protocol
completeness, maybe with an option such as
"waste-memory-and-cpu-to-support-stupid-h2-continuation-frames" or
something as explicit to save people from enabling them by accident :-/
Cheers,
Willy
H2: interoperability issue due to lack of CONTINUATION frame support
Hi Willy,
haproxy is currently unable to handle CONTINUATION [1] frames (see
commit 61290ec77 - [2]).
If a client emits a CONTINUATION frame, we will break the connection
and send GOAWAY due to INTERNAL_ERROR. This of course leads to
interoperability issues.
Notably, older Chrome/Chromium releases (50.0.2645.4 and older),
lacking Chromium commit a8bff211e ("Increase sent control frame
fragment size to 16 kB") [3], will cap the maximum control frame size
at 1024 bytes, triggering CONTINUATION, when large headers are used
(long URIs, or large cookies).
How to reproduce?
In Chrome/Chromium 50.0.2645.4 or older, access a very long URI on a
h2 enabled haproxy instance. For example on cdn.haproxy.com [4].
Why are old Chrome/Chromium relevant?
Chrome 49 is the latest release that runs under Windows XP. Yes, I
know, but apparently there are large market segments out there where
this has a real business impact. Chrome 49 and Windows XP support
situation a side, this is not a Chrome bug, but a lack of support of a
H2 feature in haproxy. Other corner cases may trigger this issue as
well.
At least 4 different people reported this as an actual
interoperability problem with H2 on discourse [5], [6], additionally
we have 2 reports of results of a RFC7540 test [7], [8], which also
fails due to at least lack of CONTINUATION support.
While this issue is arguably not major - there would have been more
reports otherwise, I do think we should address this issue.
As far as I can see, other major h2 implementations do implement
CONTINUATION support, including nginx and nghttp2.
Best regards,
Lukas
[1] https://tools.ietf.org/html/rfc7540#section-6.10
[2]
http://git.haproxy.org/?p=haproxy.git;a=commit;h=61290ec774b311edad0285b90fab979a420d252c
[3]
https://github.com/chromium/chromium/commit/a8bff211e9cc3c64f2e6516186131dcdeed59eae
[4]
https://cdn.haproxy.com/wp-includes/css/dashicons.min.css?aa
[5]
https://discourse.haproxy.org/t/http-2-not-compatible-with-filrefox-on-haproxy-1-8-3/2075/32
[6] https://discourse.haproxy.org/t/http-2-chrome-49-on-windows-xp-problems/2931
[7] https://discourse.haproxy.org/t/http2-rfc7540-and-rfc754-test/2042
[8] https://www.mail-archive.com/haproxy@formilux.org/msg28010.html
