Hi Max,
On Thu, Mar 21, Maximilian Böhm wrote:
> thanks for your suggestions. It was not successful.
>
> However, I managed to make it reproductible. I would be really happy, if
> someone more experienced would take a look on this.
>
> Setup
> Client (Chrome) -> Haproxy (Docker) -> Jetty (Docker)
>
> The client executes following script, it can be saved on the local disk, we
> can ignore the CORS logging. Do not change the '30000', otherwise it will not
> occur.
> <script type="text/javascript">
> function loopMe () {
> setTimeout(function () {
> var xhr = new XMLHttpRequest();
> xhr.open('POST', 'https://[DOMAIN]/app/docker-jetty.json');
> xhr.send();
> loopMe();
> }, 30000)
> }
> loopMe();
> </script>
>
Can you test with 2.0-dev2 afaik it contains H2 abortonclose fixes that might
help with this ?
And somewhat out of curiosity does changing jetty's default?
jetty.http.idleTimeout=30000 make it easier to reproduce ?
-Jarno
> Haproxy.cfg
> global
> daemon
> tune.ssl.default-dh-param 2048
> stats socket /var/run/haproxy.stat
>
> defaults
> mode http
> option httplog
> log stdout format raw daemon
> timeout connect 5m
> timeout client 5m
> timeout server 5m
>
> frontend frontend_h2
> bind *:443 ssl crt /usr/local/etc/haproxy/ssl/ alpn h2,http/1.1
> use_backend backend_jetty
>
> backend backend_jetty
> server web850 127.0.0.1:81
>
> Commands for starting the container
> 1) docker run -p 443:443 -v [LOCAL_DIR_HAPROXY]:/usr/local/etc/haproxy/ -it
> haproxy:1.9.4
> 2) docker run -d --name jetty -v [LOCAL_DIR_JETTY]:/var/lib/jetty/webapps -p
> 81:8080 jetty
>
> Substitute the following variables:
> 1) [DOMAIN]: Domain you have a certificate for or generate one
> 2) [LOCAL_DIR_HAPROXY]: Local directory where you need to put the
> "haproxy.cfg" and your certificate (subdirectory "ssl")
> 3) [LOCAL_DIR_JETTY]: Local directory, create a subdirectory called "app" and
> create an empty file named "docker-jetty.json").
>
> Substitute the variables, start the container and open the script in the
> browser. After 10-15 requests you should get a SC 400....
>
> At first sight, it looks like jetty is doing something terribly wrong. But,
> and that's the problem, it does not occur if I have just http/1.1 enabled
> between the client and haproxy. Any ideas?
>
> Thanks,
> Max
>
> -----Ursprüngliche Nachricht-----
> Von: Jarno Huuskonen <jarno.huusko...@uef.fi>
> Gesendet: Mittwoch, 20. März 2019 12:59
> An: Maximilian Böhm <maximilian.bo...@auctores.de>
> Cc: haproxy@formilux.org
> Betreff: Re: 400 SC on h2 xhr post
>
> Hi Max,
>
> On Wed, Mar 20, Maximilian Böhm wrote:
> > >> If the 400 errors happen within 3mins, have you tried changing
> > >> client/keep-alive timeouts to see if anything changes ?
> > They do most often happen in the first 3 mins. But that's not always the
> > case. And if it's really a timeout, shouldn't it be more clearly recurring?
> > Like every tenth request fails. But that's also not the case. Sometimes
> > it's the 3rd request, sometimes the 20th or even later.
> > However, I did increase the previously set timeouts (40min). But this did
> > not change anything at all. Is there another timeout which explicitly only
> > affects h2 on the client side?
>
> I'm not aware of any more timeouts to test. I think possible timeouts are in
> https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.1
>
> Have you tested different values for http-reuse (never to always) ?
> (https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.2-http-reuse)
> (Probably doesn't make any difference).
>
> This could be related to
> https://www.mail-archive.com/haproxy@formilux.org/msg32959.html
> that test case also returns 400 error with state CH-- with http2.
>
> -Jarno
>
> > -----Ursprüngliche Nachricht-----
> > Von: Jarno Huuskonen <jarno.huusko...@uef.fi>
> > Gesendet: Dienstag, 19. März 2019 17:34
> > An: Maximilian Böhm <maximilian.bo...@auctores.de>
> > Cc: haproxy@formilux.org
> > Betreff: Re: 400 SC on h2 xhr post
> >
> > Hi,
> >
> > On Tue, Mar 19, Maximilian Böhm wrote:
> > > The problem I experience is within a legacy javascript application which
> > > periodically checks if the user is still logged in. It does so by sending
> > > an xhr request every 30 seconds (I said, it's a legacy app, right? It
> > > does so by POST not GET...). As you may guess, this behavior works using
> > > http1.1 quasi infinitely. But as soon as I activate HTTP/2, I'll get the
> > > following output (sooner or later):
> > > 172.17.0.1:46372 [19/Mar/2019:12:10:13.465] [fntnd] [bknd] 0/0/0/14/14
> > > 200 368 - - ---- 1/1/0/1/0 0/0 "POST [URL] HTTP/1.1"
> > > 172.17.0.1:46372 [19/Mar/2019:12:10:43.465] [fntnd] [bknd] 0/0/0/-1/8 400
> > > 187 - - CH-- 1/1/0/0/0 0/0 "POST [URL] HTTP/1.1"
> > >
> > > Which means, the developer toolbar announces a response code "400" and
> > > "<html><body><h1>400 Bad request</h1>Your browser sent an invalid
> > > request.</body></html>". I was not yet successful reproduce this behavior
> > > with OkHttp (java http2-capable library). Jetty - on the backend site -
> > > does not report any requests in its ncsa request log.
> >
> > I've seen some(very few (maybe one-two a day)) 400 bad requests with
> > haproxy 1.9.4 (http2) to apache+php (http/1.1) backend. These requests alos
> > have CH.. state in logs.
> > (400 errors have also happened for GET requests).
> >
> > > It is not directly reproducible (like every second time) but it usually
> > > happens with the first 3 minutes. I experienced this behavior in Chrome
> > > (73.0.3683.75), Firefox (65.0.2 (32-Bit)) and Edge (44.17763.1.0). I also
> > > tried with different networks and different internet connections.
> > >
> > > Any ideas? Maybe a similar bug is known? What shall/can I do next?
> > > Setting up Wireshark with MITM and comparing the requests? Right now, I
> > > can't imagine the error is on side of the client nor on the backend (the
> > > backend is not changed).
> >
> > If the 400 errors happen within 3mins, have you tried changing
> > client/keep-alive timeouts to see if anything changes ?
> >
> > > timeout queue 2m
> > > timeout client 2m
> > > timeout http-keep-alive 2m
--
Jarno Huuskonen
