Re: HAProxy returns a 502 error when ssl offload and response has a large header
On Thu, Feb 07, 2019 at 06:37:28PM +0100, Willy Tarreau wrote: > > I'll try with h2c and see if I can put it between client and haproxy. > > Then I suspect that you'll see haproxy either emit RST_STREAM or emit > too large a frame and this frame get rejected. So after checking the code, I can confirm that you'll get the huge frame at once. I see how it would be possible to refragment this into one or several CONTINUATION frames with a limited extra cost. It will not be very pretty nor easy but I'm seeing a solution. Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
On Thu, Feb 07, 2019 at 06:44:01PM +0200, Jarno Huuskonen wrote: > At least on my test case haproxy listens http2 and uses http/1.1 > to backend server > (example config and example backend server (in go) are in earlier > mail: https://www.mail-archive.com/haproxy@formilux.org/msg32727.html > (just increase the header size (l in the go server) > 16309 and the > http2 connection between client <-> haproxy fails with frame error). OK, I initially misunderstood the setup then. It would be possible that the client never advertised support for a larger frame and that we don't have a solution to deliver something that large. In fact the only option would be to fragment the output and build a CONTINUATION frame. I don't know how difficult it could be in this case :-/ > So the test setup is something like: > client(curl/chrome) -> http2/haproxy -> http/1.1/go server port 8081 > > I'll try with h2c and see if I can put it between client and haproxy. Then I suspect that you'll see haproxy either emit RST_STREAM or emit too large a frame and this frame get rejected. Thanks for the explanation! Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Thu, Feb 07, Willy Tarreau wrote: > On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > > Hi, > > > > On Thu, Feb 07, Steve GIRAUD wrote: > > > Thanks Jarno for the investigation. > > > > No problem. > > > > > The large header is only on response and there is only one large header > > > (18k). > > > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > > > Did you check with curl or chrome if you get the same framing error > > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > > request is ok. > > > > I'm CC:ing Willy, in case this is something that a config option can fix > > or possibly a incompatability/bug with http2 implementation. > > I might have an idea. The default H2 max-frame-size is 16kB (by the > spec). It is possible that your server ignores the setting and tries > to push a frame size that is larger than the agreed limit, which > becomes a protocol violation. Or it is possible as well that the > server doesn't know how to send such a large header with this frame > size and simply aborts the response. At least on my test case haproxy listens http2 and uses http/1.1 to backend server (example config and example backend server (in go) are in earlier mail: https://www.mail-archive.com/haproxy@formilux.org/msg32727.html (just increase the header size (l in the go server) > 16309 and the http2 connection between client <-> haproxy fails with frame error). So the test setup is something like: client(curl/chrome) -> http2/haproxy -> http/1.1/go server port 8081 I'll try with h2c and see if I can put it between client and haproxy. -Jarno > You could install h2c between haproxy and your server, in wiretap mode, > it's very convenient to see what is exchanged : > >h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: > > Then you configure haproxy to communicate to 127.0.0.1: to join the > server while your server in fact listens on :. > > Depending on what you see, we may have the possibility to work around > it by advertising a larger max-frame-size in the settings frame when > the buffers are larger. > > Regards, > Willy > -- Jarno Huuskonen
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Thanks for your precious answer, I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. For information, we use IIS10 on windows2016 for the backend server. And Jarno reproduce the same problem with his go http server implementation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. We have dump the exchange beetween haproxy and the backend. The response seems not to be aborted by the backend. We are installing h2c to investigate. Regards, Steve De : Willy Tarreau Envoyé : jeudi 7 février 2019 17:03 À : Jarno Huuskonen; Steve GIRAUD; haproxy@formilux.org Objet : Re: HAProxy returns a 502 error when ssl offload and response has a large header Hi, On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > Hi, > > On Thu, Feb 07, Steve GIRAUD wrote: > > Thanks Jarno for the investigation. > > No problem. > > > The large header is only on response and there is only one large header > > (18k). > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > Did you check with curl or chrome if you get the same framing error > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > request is ok. > > I'm CC:ing Willy, in case this is something that a config option can fix > or possibly a incompatability/bug with http2 implementation. I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. You could install h2c between haproxy and your server, in wiretap mode, it's very convenient to see what is exchanged : h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: Then you configure haproxy to communicate to 127.0.0.1: to join the server while your server in fact listens on :. Depending on what you see, we may have the possibility to work around it by advertising a larger max-frame-size in the settings frame when the buffers are larger. Regards, Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > Hi, > > On Thu, Feb 07, Steve GIRAUD wrote: > > Thanks Jarno for the investigation. > > No problem. > > > The large header is only on response and there is only one large header > > (18k). > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > Did you check with curl or chrome if you get the same framing error > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > request is ok. > > I'm CC:ing Willy, in case this is something that a config option can fix > or possibly a incompatability/bug with http2 implementation. I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. You could install h2c between haproxy and your server, in wiretap mode, it's very convenient to see what is exchanged : h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: Then you configure haproxy to communicate to 127.0.0.1: to join the server while your server in fact listens on :. Depending on what you see, we may have the possibility to work around it by advertising a larger max-frame-size in the settings frame when the buffers are larger. Regards, Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Thu, Feb 07, Steve GIRAUD wrote: > Thanks Jarno for the investigation. No problem. > The large header is only on response and there is only one large header (18k). > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails Did you check with curl or chrome if you get the same framing error that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > If I request my backend directly in h2 + ssl but without haproxy, the request > is ok. I'm CC:ing Willy, in case this is something that a config option can fix or possibly a incompatability/bug with http2 implementation. -Jarno > Hi, > > On Wed, Feb 06, Steve GIRAUD wrote: > > Effectively, the header size is 17 556 bytes. > > Is the large header(s) only on response (and not on request) ? > (Is it one large header 17k header ?) > > > If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the > > request failed. > > For me (tested with current 2.0dev) increasing global tune.bufsize to > 32768 allowed larger response header. With my limited testing http/https on > frontend didn't make difference. > (Does my test config work for you (you'll need to comment option htx > with haprox-1.8) ?) > > But if I use curl --http2 to haproxy+ssl frontend and my silly > httpsrv.go sends x-dummy larger than 16309 then curl --http2 fails > with curl: (16) Error in the HTTP2 framing layer > (chrome reports ERR_SPDY_FRAME_SIZE_ERROR). > > Is haproxy trying / sending a larger http2 frame than clients are > willing to receive (SETTINGS_MAX_FRAME_SIZE?) ? > > (Same request with --http1.1 to haproxy+ssl frontend works). > > I'm attaching my test config and the httpsrv.go that I used as a > backend server. > Maybe http2 gurus can take a look and see if the frame size error is > expected or not ? > > -Jarno > > > De : Jarno Huuskonen > > Envoyé : mercredi 6 février 2019 09:36 > > À : Steve GIRAUD > > Cc : haproxy@formilux.org > > Objet : Re: HAProxy returns a 502 error when ssl offload and response has a > > large header > > > > Hi, > > > > On Wed, Feb 06, Steve GIRAUD wrote: > > > Hello everybody, > > > Has anyone ever found that HAProxy returns a 502 error when ssl offload > > > is enabled and the http response contains a very long header. > > > If I turn off SSL offload , all is OK with the same header. > > > > What's the size of the (very long) headers (how many bytes) ? > > Is it by any chance larger than the bufsize or maxrewrite ? > > > > > Default settings : > > > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 -- Jarno Huuskonen
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, Thanks Jarno for the investigation. The large header is only on response and there is only one large header (18k). haproxy + ssl + http2+ tune.bufsize:32768 --> request fails haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok If I request my backend directly in h2 + ssl but without haproxy, the request is ok. Regards, -- Steve Giraud Hi, On Wed, Feb 06, Steve GIRAUD wrote: > Effectively, the header size is 17 556 bytes. Is the large header(s) only on response (and not on request) ? (Is it one large header 17k header ?) > If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the > request failed. For me (tested with current 2.0dev) increasing global tune.bufsize to 32768 allowed larger response header. With my limited testing http/https on frontend didn't make difference. (Does my test config work for you (you'll need to comment option htx with haprox-1.8) ?) But if I use curl --http2 to haproxy+ssl frontend and my silly httpsrv.go sends x-dummy larger than 16309 then curl --http2 fails with curl: (16) Error in the HTTP2 framing layer (chrome reports ERR_SPDY_FRAME_SIZE_ERROR). Is haproxy trying / sending a larger http2 frame than clients are willing to receive (SETTINGS_MAX_FRAME_SIZE?) ? (Same request with --http1.1 to haproxy+ssl frontend works). I'm attaching my test config and the httpsrv.go that I used as a backend server. Maybe http2 gurus can take a look and see if the frame size error is expected or not ? -Jarno > De : Jarno Huuskonen > Envoyé : mercredi 6 février 2019 09:36 > À : Steve GIRAUD > Cc : haproxy@formilux.org > Objet : Re: HAProxy returns a 502 error when ssl offload and response has a > large header > > Hi, > > On Wed, Feb 06, Steve GIRAUD wrote: > > Hello everybody, > > Has anyone ever found that HAProxy returns a 502 error when ssl offload is > > enabled and the http response contains a very long header. > > If I turn off SSL offload , all is OK with the same header. > > What's the size of the (very long) headers (how many bytes) ? > Is it by any chance larger than the bufsize or maxrewrite ? > > > Default settings : > > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > -Jarno > > -- > Jarno Huuskonen -- Jarno Huuskonen
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi,
On Wed, Feb 06, Steve GIRAUD wrote:
> Effectively, the header size is 17 556 bytes.
Is the large header(s) only on response (and not on request) ?
(Is it one large header 17k header ?)
> If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the
> request failed.
For me (tested with current 2.0dev) increasing global tune.bufsize to
32768 allowed larger response header. With my limited testing http/https on
frontend didn't make difference.
(Does my test config work for you (you'll need to comment option htx
with haprox-1.8) ?)
But if I use curl --http2 to haproxy+ssl frontend and my silly
httpsrv.go sends x-dummy larger than 16309 then curl --http2 fails
with curl: (16) Error in the HTTP2 framing layer
(chrome reports ERR_SPDY_FRAME_SIZE_ERROR).
Is haproxy trying / sending a larger http2 frame than clients are
willing to receive (SETTINGS_MAX_FRAME_SIZE?) ?
(Same request with --http1.1 to haproxy+ssl frontend works).
I'm attaching my test config and the httpsrv.go that I used as a
backend server.
Maybe http2 gurus can take a look and see if the frame size error is
expected or not ?
-Jarno
> De : Jarno Huuskonen
> Envoyé : mercredi 6 février 2019 09:36
> À : Steve GIRAUD
> Cc : haproxy@formilux.org
> Objet : Re: HAProxy returns a 502 error when ssl offload and response has a
> large header
>
> Hi,
>
> On Wed, Feb 06, Steve GIRAUD wrote:
> > Hello everybody,
> > Has anyone ever found that HAProxy returns a 502 error when ssl offload is
> > enabled and the http response contains a very long header.
> > If I turn off SSL offload , all is OK with the same header.
>
> What's the size of the (very long) headers (how many bytes) ?
> Is it by any chance larger than the bufsize or maxrewrite ?
>
> > Default settings :
> > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> -Jarno
>
> --
> Jarno Huuskonen
--
Jarno Huuskonen
global
tune.bufsize 32678
defaults
mode http
option http-use-htx
timeout connect 1s
timeout client 2s
timeout server 4s
timeout tarpit 3s
listen HTTPS_in
mode http
bind 127.0.0.1:8443 ssl crt common.pem alpn h2,http/1.1
bind 127.0.0.1:8080
server go-http 127.0.0.1:8081
package main
import (
"fmt"
"math/rand"
"net/http"
)
const letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
func handler(w http.ResponseWriter, r *http.Request) {
l := 16309 // >= 16310 breaks haproxy http2 FE (curl: (16) Error in the
HTTP2 framing layer) / chrome also reports ERR_SPDY_FRAME_SIZE_ERROR
b := make([]byte, l)
for i := range b {
b[i] = letterBytes[rand.Int63()%int64(len(letterBytes))]
}
s := string(b[:l])
w.Header().Set("X-Dummy", s)
fmt.Fprintf(w, "Howdy neighbour!\n")
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8081", nil)
}
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Effectively, the header size is 17 556 bytes. If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the request failed. -- Steve Giraud De : Jarno Huuskonen Envoyé : mercredi 6 février 2019 09:36 À : Steve GIRAUD Cc : haproxy@formilux.org Objet : Re: HAProxy returns a 502 error when ssl offload and response has a large header Hi, On Wed, Feb 06, Steve GIRAUD wrote: > Hello everybody, > Has anyone ever found that HAProxy returns a 502 error when ssl offload is > enabled and the http response contains a very long header. > If I turn off SSL offload , all is OK with the same header. What's the size of the (very long) headers (how many bytes) ? Is it by any chance larger than the bufsize or maxrewrite ? > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 -Jarno -- Jarno Huuskonen
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Wed, Feb 06, Steve GIRAUD wrote: > Hello everybody, > Has anyone ever found that HAProxy returns a 502 error when ssl offload is > enabled and the http response contains a very long header. > If I turn off SSL offload , all is OK with the same header. What's the size of the (very long) headers (how many bytes) ? Is it by any chance larger than the bufsize or maxrewrite ? > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 -Jarno -- Jarno Huuskonen
