RE: HAProxy returns a 502 error when ssl offload and response has a large header
Thanks for your precious answer, I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. For information, we use IIS10 on windows2016 for the backend server. And Jarno reproduce the same problem with his go http server implementation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. We have dump the exchange beetween haproxy and the backend. The response seems not to be aborted by the backend. We are installing h2c to investigate. Regards, Steve De : Willy Tarreau Envoyé : jeudi 7 février 2019 17:03 À : Jarno Huuskonen; Steve GIRAUD; haproxy@formilux.org Objet : Re: HAProxy returns a 502 error when ssl offload and response has a large header Hi, On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > Hi, > > On Thu, Feb 07, Steve GIRAUD wrote: > > Thanks Jarno for the investigation. > > No problem. > > > The large header is only on response and there is only one large header > > (18k). > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > Did you check with curl or chrome if you get the same framing error > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > request is ok. > > I'm CC:ing Willy, in case this is something that a config option can fix > or possibly a incompatability/bug with http2 implementation. I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. You could install h2c between haproxy and your server, in wiretap mode, it's very convenient to see what is exchanged : h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: Then you configure haproxy to communicate to 127.0.0.1: to join the server while your server in fact listens on :. Depending on what you see, we may have the possibility to work around it by advertising a larger max-frame-size in the settings frame when the buffers are larger. Regards, Willy
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, Thanks Jarno for the investigation. The large header is only on response and there is only one large header (18k). haproxy + ssl + http2+ tune.bufsize:32768 --> request fails haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok If I request my backend directly in h2 + ssl but without haproxy, the request is ok. Regards, -- Steve Giraud Hi, On Wed, Feb 06, Steve GIRAUD wrote: > Effectively, the header size is 17 556 bytes. Is the large header(s) only on response (and not on request) ? (Is it one large header 17k header ?) > If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the > request failed. For me (tested with current 2.0dev) increasing global tune.bufsize to 32768 allowed larger response header. With my limited testing http/https on frontend didn't make difference. (Does my test config work for you (you'll need to comment option htx with haprox-1.8) ?) But if I use curl --http2 to haproxy+ssl frontend and my silly httpsrv.go sends x-dummy larger than 16309 then curl --http2 fails with curl: (16) Error in the HTTP2 framing layer (chrome reports ERR_SPDY_FRAME_SIZE_ERROR). Is haproxy trying / sending a larger http2 frame than clients are willing to receive (SETTINGS_MAX_FRAME_SIZE?) ? (Same request with --http1.1 to haproxy+ssl frontend works). I'm attaching my test config and the httpsrv.go that I used as a backend server. Maybe http2 gurus can take a look and see if the frame size error is expected or not ? -Jarno > De : Jarno Huuskonen > Envoyé : mercredi 6 février 2019 09:36 > À : Steve GIRAUD > Cc : haproxy@formilux.org > Objet : Re: HAProxy returns a 502 error when ssl offload and response has a > large header > > Hi, > > On Wed, Feb 06, Steve GIRAUD wrote: > > Hello everybody, > > Has anyone ever found that HAProxy returns a 502 error when ssl offload is > > enabled and the http response contains a very long header. > > If I turn off SSL offload , all is OK with the same header. > > What's the size of the (very long) headers (how many bytes) ? > Is it by any chance larger than the bufsize or maxrewrite ? > > > Default settings : > > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > -Jarno > > -- > Jarno Huuskonen -- Jarno Huuskonen
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Effectively, the header size is 17 556 bytes. If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the request failed. -- Steve Giraud De : Jarno Huuskonen Envoyé : mercredi 6 février 2019 09:36 À : Steve GIRAUD Cc : haproxy@formilux.org Objet : Re: HAProxy returns a 502 error when ssl offload and response has a large header Hi, On Wed, Feb 06, Steve GIRAUD wrote: > Hello everybody, > Has anyone ever found that HAProxy returns a 502 error when ssl offload is > enabled and the http response contains a very long header. > If I turn off SSL offload , all is OK with the same header. What's the size of the (very long) headers (how many bytes) ? Is it by any chance larger than the bufsize or maxrewrite ? > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 -Jarno -- Jarno Huuskonen
HAProxy returns a 502 error when ssl offload and response has a large header
Hello everybody,
Has anyone ever found that HAProxy returns a 502 error when ssl offload is
enabled and the http response contains a very long header.
If I turn off SSL offload , all is OK with the same header.
I Use HaProxy 1.8.17 on PFSense 2.4.4p1
Regards,
uname -a
FreeBSD PFSHA3.statim.fr 11.2-RELEASE-p4 FreeBSD 11.2-RELEASE-p4 #2
b00c407ba5d(RELENG_2_4_4): Mon Nov 26 11:41:48 EST 2018
r...@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense
amd64
haproxy -vv
HA-Proxy version 1.8.17 2019/01/08
Copyright 2000-2019 Willy Tarreau
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -O2 -pipe -fstack-protector -fno-strict-aliasing
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label
-DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1
USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2o-freebsd 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.0.2o-freebsd 27 Mar 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
--
Steve Giraud
