Haproxy 1.8 http/2 mode does not pass the h2spec conformance test
See the tool here https://github.com/summerwind/h2spec Generic tests for HTTP/2 server 1. Starting HTTP/2 × 1: Sends a client connection preface -> The endpoint MUST accept client connection preface. Expected: SETTINGS Frame (flags:0x00) Actual: Error: unexpected EOF 2. Streams and Multiplexing × 1: Sends a PRIORITY frame on idle stream Error: unexpected EOF × 2: Sends a WINDOW_UPDATE frame on half-closed (remote) stream Error: unexpected EOF × 3: Sends a PRIORITY frame on half-closed (remote) stream Error: unexpected EOF × 4: Sends a RST_STREAM frame on half-closed (remote) stream Error: unexpected EOF × 5: Sends a PRIORITY frame on closed stream Error: unexpected EOF 3. Frame Definitions 3.1. DATA × 1: Sends a DATA frame Error: unexpected EOF × 2: Sends multiple DATA frames Error: unexpected EOF × 3: Sends a DATA frame with padding Error: unexpected EOF 3.2. HEADERS × 1: Sends a HEADERS frame Error: unexpected EOF × 2: Sends a HEADERS frame with padding Error: unexpected EOF × 3: Sends a HEADERS frame with priority Error: unexpected EOF 3.3. PRIORITY × 1: Sends a PRIORITY frame with priority 1 Error: unexpected EOF × 2: Sends a PRIORITY frame with priority 256 Error: unexpected EOF × 3: Sends a PRIORITY frame with stream dependency Error: unexpected EOF × 4: Sends a PRIORITY frame with exclusive Error: unexpected EOF × 5: Sends a PRIORITY frame for an idle stream, then send a HEADER frame for a lower stream ID Error: unexpected EOF 3.4. RST_STREAM × 1: Sends a RST_STREAM frame Error: unexpected EOF 3.5. SETTINGS × 1: Sends a SETTINGS frame Error: unexpected EOF 3.7. PING × 1: Sends a PING frame Error: unexpected EOF 3.8. GOAWAY × 1: Sends a GOAWAY frame Error: unexpected EOF 3.9. WINDOW_UPDATE × 1: Sends a WINDOW_UPDATE frame with stream ID 0 Error: unexpected EOF × 2: Sends a WINDOW_UPDATE frame with stream ID 1 Error: unexpected EOF 3.10. CONTINUATION × 1: Sends a CONTINUATION frame Error: unexpected EOF × 2: Sends multiple CONTINUATION frames Error: unexpected EOF 4. HTTP Message Exchanges × 1: Sends a GET request Error: unexpected EOF × 2: Sends a HEAD request Error: unexpected EOF × 3: Sends a POST request Error: unexpected EOF × 4: Sends a POST request with trailers Error: unexpected EOF 5. HPACK × 1: Sends a indexed header field representation Error: unexpected EOF × 2: Sends a literal header field with incremental indexing - indexed name Error: unexpected EOF × 3: Sends a literal header field with incremental indexing - indexed name (with Huffman coding) Error: unexpected EOF × 4: Sends a literal header field with incremental indexing - new name Error: unexpected EOF × 5: Sends a literal header field with incremental indexing - new name (with Huffman coding) Error: unexpected EOF × 6: Sends a literal header field without indexing - indexed name Error: unexpected EOF × 7: Sends a literal header field without indexing - indexed name (with Huffman coding) Error: unexpected EOF × 8: Sends a literal header field without indexing - new name Error: unexpected EOF × 9: Sends a literal header field without indexing - new name (huffman encoded) Error: unexpected EOF × 10: Sends a literal header field never indexed - indexed name Error: unexpected EOF × 11: Sends a literal header field never indexed - indexed name (huffman encoded) Error: unexpected EOF × 12: Sends a literal header field never indexed - new name Error: unexpected EOF × 13: Sends a literal header field never indexed - new name (huffman encoded) Error: unexpected EOF × 14: Sends a dynamic table size update Error: unexpected EOF × 15: Sends multiple dynamic table size update Error: unexpected EOF Hypertext Transfer Protocol Version 2 (HTTP/2) 3. Starting HTTP/2 3.5. HTTP/2 Connection Preface × 1: Sends client connection preface -> The server connection preface MUST be the first frame the server sends in the HTTP/2 connection. Expected: SETTINGS Frame (flags:0x00) Actual: Error: unexpected EOF ✔ 2: Sends invalid connection preface 4. HTTP Frames 4.1. Frame Format × 1: Sends a frame with unknown type Error: unexpected EOF × 2: Sends a frame with undefined flag Error: unexpected EOF × 3: Sends a frame with reserved field bit Error: unexpected EOF 4.2. Frame Size × 1: Sends a DATA frame with 2^14 octets in length Error: unexpected EOF × 2: Sends a large size DATA frame that exceeds the SETTINGS_MAX_FRAME_SIZE Error: rea
Re: Haproxy 1.8 http/2 mode does not pass the h2spec conformance test
A very stripped down version of config global log 127.0.0.1local0 log 127.0.0.1local1 notice maxconn 4096 pidfile /var/run/haproxy.pid stats socket /var/run/haproxy.stat mode 600 level admin expose-fd listeners daemon tune.ssl.default-dh-param 1024 tune.ssl.cachesize 10 ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets tune.bufsize 16384 tune.maxrewrite 1024 ssl-engine rdrand ssl-mode-async nbthread 4 defaults log global mode http compression algo gzip compression type text/html text/plain text/css application/javascript application/octet-stream application/json option httplog option dontlognull option redispatch option tcp-smart-accept option tcp-smart-connect option forwardfor timeout check 5s timeout client 50s timeout tunnel 6s timeout connect 20s timeout http-keep-alive 15s timeout http-request 30s timeout queue 20s timeout server 50s hash-balance-factor 125 balance hdr(Cookie) hash-type consistent djb2 stats enable stats hide-version stats auth : stats uri /statz default-server inter 5s fall 3 rise 1 frontend http bind :::80 v4v6 redirect scheme https if !{ ssl_fc } default_backend assets_backend frontend https bind :::443 v4v6 ssl crt no-tls-tickets alpn h2,http/1.1 maxconn 4096 default_backend assets_backend On Sun, Nov 26, 2017 at 11:04 PM Willy Tarreau wrote: On Mon, Nov 27, 2017 at 03:08:30AM +, Robin Anil wrote: > > See the tool here https://github.com/summerwind/h2spec > > Thanks for the link, but could you also post your config ? Given that 100% > of your tests ended up in "Error: unexpected EOF", I guess there's > something > completely wrong somewhere! If the config is not working for whatever > reason, > I would at least like to be able to emit a warning or an error when this > condition is detected. > > Willy >
Re: Haproxy 1.8 http/2 mode does not pass the h2spec conformance test
I am running HaProxy 1.8 within docker container using a fork of the official build and pointed them to the latest 1.8.0 instead of rc-4. So it is built using openssl1.1 Docker hub link: https://hub.docker.com/_/haproxy/ See the Dockerfile https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/alpine/Dockerfile Just replaced these two lines ENV HAPROXY_VERSION 1.8.0 ENV HAPROXY_MD5 6ccea4619b7183fbcc8c98bae1f9823d On Sun, Nov 26, 2017 at 11:29 PM Willy Tarreau wrote: > On Mon, Nov 27, 2017 at 06:15:48AM +0100, Willy Tarreau wrote: > > On Mon, Nov 27, 2017 at 05:10:16AM +0000, Robin Anil wrote: > > > A very stripped down version of config > > > > Thank you, I'll check if anything there can explain this. > > So with your config I'm getting this : > > Finished in 52.3802 seconds > 145 tests, 110 passed, 4 skipped, 31 failed > > I had to disable ssl-mode-async as my openssl version doesn't support it. > > Of the 31, I'm seeing real bugs compared to what is expected to work, such > as : > > 3.8. GOAWAY > Ã 1: Sends a GOAWAY frame > -> The endpoint MUST accept GOAWAY frame. >Expected: Connection closed > Actual: Timeout > > Others are expected for now : > > 3.10. CONTINUATION > Ã 1: Sends a CONTINUATION frame > -> The endpoint MUST accept CONTINUATION frame. >Expected: HEADERS Frame (stream_id:1) > Actual: Connection closed > > 4. HTTP Message Exchanges > Ã 4: Sends a POST request with trailers > -> The endpoint MUST respond to the request. > Expected: HEADERS Frame (stream_id:1) >Actual: Connection closed > > And the other ones are very useful as they likely indicate missing checks. > So I'll take a look. Thanks! > > Willy >
Re: Haproxy 1.8 http/2 mode does not pass the h2spec conformance test
Sorry, this one, not the alpine build https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/Dockerfile On Sun, Nov 26, 2017 at 11:36 PM Robin Anil wrote: > I am running HaProxy 1.8 within docker container using a fork of the > official build and pointed them to the latest 1.8.0 instead of rc-4. So it > is built using openssl1.1 > > Docker hub link: https://hub.docker.com/_/haproxy/ > See the Dockerfile > https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/alpine/Dockerfile > > Just replaced these two lines > ENV HAPROXY_VERSION 1.8.0 > ENV HAPROXY_MD5 6ccea4619b7183fbcc8c98bae1f9823d > > On Sun, Nov 26, 2017 at 11:29 PM Willy Tarreau wrote: > >> On Mon, Nov 27, 2017 at 06:15:48AM +0100, Willy Tarreau wrote: >> > On Mon, Nov 27, 2017 at 05:10:16AM +, Robin Anil wrote: >> > > A very stripped down version of config >> > >> > Thank you, I'll check if anything there can explain this. >> >> So with your config I'm getting this : >> >> Finished in 52.3802 seconds >> 145 tests, 110 passed, 4 skipped, 31 failed >> >> I had to disable ssl-mode-async as my openssl version doesn't support it. >> >> Of the 31, I'm seeing real bugs compared to what is expected to work, such >> as : >> >> 3.8. GOAWAY >> Ã 1: Sends a GOAWAY frame >> -> The endpoint MUST accept GOAWAY frame. >>Expected: Connection closed >> Actual: Timeout >> >> Others are expected for now : >> >> 3.10. CONTINUATION >> Ã 1: Sends a CONTINUATION frame >> -> The endpoint MUST accept CONTINUATION frame. >>Expected: HEADERS Frame (stream_id:1) >> Actual: Connection closed >> >> 4. HTTP Message Exchanges >> Ã 4: Sends a POST request with trailers >> -> The endpoint MUST respond to the request. >> Expected: HEADERS Frame (stream_id:1) >>Actual: Connection closed >> >> And the other ones are very useful as they likely indicate missing checks. >> So I'll take a look. Thanks! >> >> Willy >> >
Re: Haproxy 1.8 http/2 mode does not pass the h2spec conformance test
I can get the tests to match yours now. Looks like it was a connection issue. Finished in 70.5089 seconds 145 tests, 116 passed, 0 skipped, 29 failed Also, If you haven't noticed the --strict option, in that mode I can get the test to hang on 4.2. Frame Size ✔ 1: Sends a DATA frame with 2^14 octets in length 2: Sends a large size DATA frame that exceeds the SETTINGS_MAX_FRAME_SIZE On Sun, Nov 26, 2017 at 11:39 PM Robin Anil wrote: > Sorry, this one, not the alpine build > > > https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/Dockerfile > > On Sun, Nov 26, 2017 at 11:36 PM Robin Anil wrote: > >> I am running HaProxy 1.8 within docker container using a fork of the >> official build and pointed them to the latest 1.8.0 instead of rc-4. So it >> is built using openssl1.1 >> >> Docker hub link: https://hub.docker.com/_/haproxy/ >> See the Dockerfile >> https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/alpine/Dockerfile >> >> Just replaced these two lines >> ENV HAPROXY_VERSION 1.8.0 >> ENV HAPROXY_MD5 6ccea4619b7183fbcc8c98bae1f9823d >> >> On Sun, Nov 26, 2017 at 11:29 PM Willy Tarreau wrote: >> >>> On Mon, Nov 27, 2017 at 06:15:48AM +0100, Willy Tarreau wrote: >>> > On Mon, Nov 27, 2017 at 05:10:16AM +, Robin Anil wrote: >>> > > A very stripped down version of config >>> > >>> > Thank you, I'll check if anything there can explain this. >>> >>> So with your config I'm getting this : >>> >>> Finished in 52.3802 seconds >>> 145 tests, 110 passed, 4 skipped, 31 failed >>> >>> I had to disable ssl-mode-async as my openssl version doesn't support it. >>> >>> Of the 31, I'm seeing real bugs compared to what is expected to work, >>> such >>> as : >>> >>> 3.8. GOAWAY >>> Ã 1: Sends a GOAWAY frame >>> -> The endpoint MUST accept GOAWAY frame. >>>Expected: Connection closed >>> Actual: Timeout >>> >>> Others are expected for now : >>> >>> 3.10. CONTINUATION >>> Ã 1: Sends a CONTINUATION frame >>> -> The endpoint MUST accept CONTINUATION frame. >>>Expected: HEADERS Frame (stream_id:1) >>> Actual: Connection closed >>> >>> 4. HTTP Message Exchanges >>> Ã 4: Sends a POST request with trailers >>> -> The endpoint MUST respond to the request. >>> Expected: HEADERS Frame (stream_id:1) >>>Actual: Connection closed >>> >>> And the other ones are very useful as they likely indicate missing >>> checks. >>> So I'll take a look. Thanks! >>> >>> Willy >>> >>
Re: Haproxy 1.8 http/2 mode does not pass the h2spec conformance test
Hi WIlly, Most of the tests pass now in 1.8.2. However, the following test still hangs in the --strict mode. 2: Sends a large size DATA frame that exceeds the SETTINGS_MAX_FRAME_SIZE On Sun, Nov 26, 2017 at 11:54 PM Robin Anil wrote: > I can get the tests to match yours now. Looks like it was a connection > issue. > Finished in 70.5089 seconds > > 145 tests, 116 passed, 0 skipped, 29 failed > > > > Also, If you haven't noticed the --strict option, in that mode I can get > the test to hang on > >4.2. Frame Size > > ✔ 1: Sends a DATA frame with 2^14 octets in length > > 2: Sends a large size DATA frame that exceeds the > SETTINGS_MAX_FRAME_SIZE > > On Sun, Nov 26, 2017 at 11:39 PM Robin Anil wrote: > >> Sorry, this one, not the alpine build >> >> >> https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/Dockerfile >> >> On Sun, Nov 26, 2017 at 11:36 PM Robin Anil wrote: >> >>> I am running HaProxy 1.8 within docker container using a fork of the >>> official build and pointed them to the latest 1.8.0 instead of rc-4. So it >>> is built using openssl1.1 >>> >>> Docker hub link: https://hub.docker.com/_/haproxy/ >>> See the Dockerfile >>> https://github.com/docker-library/haproxy/blob/0c9c27713bfca8505331e0da2664a9454755c7b9/1.8-rc/alpine/Dockerfile >>> >>> Just replaced these two lines >>> ENV HAPROXY_VERSION 1.8.0 >>> ENV HAPROXY_MD5 6ccea4619b7183fbcc8c98bae1f9823d >>> >>> On Sun, Nov 26, 2017 at 11:29 PM Willy Tarreau wrote: >>> >>>> On Mon, Nov 27, 2017 at 06:15:48AM +0100, Willy Tarreau wrote: >>>> > On Mon, Nov 27, 2017 at 05:10:16AM +, Robin Anil wrote: >>>> > > A very stripped down version of config >>>> > >>>> > Thank you, I'll check if anything there can explain this. >>>> >>>> So with your config I'm getting this : >>>> >>>> Finished in 52.3802 seconds >>>> 145 tests, 110 passed, 4 skipped, 31 failed >>>> >>>> I had to disable ssl-mode-async as my openssl version doesn't support >>>> it. >>>> >>>> Of the 31, I'm seeing real bugs compared to what is expected to work, >>>> such >>>> as : >>>> >>>> 3.8. GOAWAY >>>> Ã 1: Sends a GOAWAY frame >>>> -> The endpoint MUST accept GOAWAY frame. >>>>Expected: Connection closed >>>> Actual: Timeout >>>> >>>> Others are expected for now : >>>> >>>> 3.10. CONTINUATION >>>> Ã 1: Sends a CONTINUATION frame >>>> -> The endpoint MUST accept CONTINUATION frame. >>>>Expected: HEADERS Frame (stream_id:1) >>>> Actual: Connection closed >>>> >>>> 4. HTTP Message Exchanges >>>> Ã 4: Sends a POST request with trailers >>>> -> The endpoint MUST respond to the request. >>>> Expected: HEADERS Frame (stream_id:1) >>>>Actual: Connection closed >>>> >>>> And the other ones are very useful as they likely indicate missing >>>> checks. >>>> So I'll take a look. Thanks! >>>> >>>> Willy >>>> >>>
HAProxy 1.8 takes too long to update new config
In Http/2 mode with threads enabled, updating the config on a live serving haproxy server takes several minutes. If I turn this off, the update of config is near instantaneous. The config change itself is just adding a newline in the file. So it feels like haproxy is waiting for connections to close down or something... My question is, is this a known behavior? Definitely not desired, not unless the stats page has some indicator that update is happening and connections are being migrated over or something. Robin
Re: HAProxy 1.8 takes too long to update new config
I can isolate that to http2, not threads at v1.8.2 Separately, I am seeing a lot of intermittent failures with POST/PUT requests, I see the following returned from haproxy. 400 Bad request Your browser sent an invalid request. I can verify these requests never hit the backend. On Thu, Dec 28, 2017 at 1:47 AM Robin Anil wrote: > In Http/2 mode with threads enabled, updating the config on a live serving > haproxy server takes several minutes. > > If I turn this off, the update of config is near instantaneous. > > The config change itself is just adding a newline in the file. So it feels > like haproxy is waiting for connections to close down or something... > > My question is, is this a known behavior? Definitely not desired, not > unless the stats page has some indicator that update is happening and > connections are being migrated over or something. > > Robin >
Re: HAProxy 1.8 takes too long to update new config
I just saw. since I don't know how to add to a thread which I am not part of (I still haven't subscribed to the haproxy mailing list). I will have to indicate here that the issue I am having is not in firefox but on the latest chrome stable. I can consistently make it happen in our discourse instance behind haproxy 1.8.2 when http2 is enabled. The config update issue is the main purpose of this thread though. On Thu, Dec 28, 2017 at 2:55 AM Lucas Rolff wrote: > Robin, there’s also an ongoing thread with Firefox which has the same > issues, especially with post/put requests in 1.8.2, you might wanna keep an > eye on that one as well > > Get Outlook for iOS <https://aka.ms/o0ukef> > ------ > *From:* Robin Anil > *Sent:* Thursday, December 28, 2017 9:36:56 AM > *To:* haproxy@formilux.org > *Subject:* Re: HAProxy 1.8 takes too long to update new config > > I can isolate that to http2, not threads at v1.8.2 > > Separately, I am seeing a lot of intermittent failures with POST/PUT > requests, I see the following returned from haproxy. > 400 Bad request > Your browser sent an invalid request. > > > I can verify these requests never hit the backend. > > > > > > > On Thu, Dec 28, 2017 at 1:47 AM Robin Anil wrote: > >> In Http/2 mode with threads enabled, updating the config on a live >> serving haproxy server takes several minutes. >> >> If I turn this off, the update of config is near instantaneous. >> >> The config change itself is just adding a newline in the file. So it >> feels like haproxy is waiting for connections to close down or something... >> >> My question is, is this a known behavior? Definitely not desired, not >> unless the stats page has some indicator that update is happening and >> connections are being migrated over or something. >> >> Robin >> >
Re: HAProxy 1.8 takes too long to update new config
- from a phone, xquz typoz On Dec 28, 2017 3:31 AM, "Willy Tarreau" wrote: Hi Robert, On Thu, Dec 28, 2017 at 07:47:03AM +0000, Robin Anil wrote: > In Http/2 mode with threads enabled, updating the config on a live serving > haproxy server takes several minutes. > > If I turn this off, the update of config is near instantaneous. I'm sorry but I don't understand what you call "this" above nor what you mean by "updating the config". If the server is running in http2 mode, and servicing connections, updating the config as shown below is no longer instantaneous. Takes over 5 minutes. Even when the update is a no-op like adding a new line at the end of the existing config. If the server is running in non http2 mode then any config updates are near instantaneous haproxy -f new_haproxy.cfg -p haproxy.pid -sf $(cat haproxy.pid) > The config change itself is just adding a newline in the file. So it feels > like haproxy is waiting for connections to close down or something... How do you proceed for the update and how do you measure this "time" ? For me "updating" means reloading, where a new process parses the new file. Since I hardly see how it could take time to parse a file in which you only added a few lines, it's unclear to me what exact problem you're facing. Could you maybe explain the steps you're taking and what you're observing ? Thanks, Willy
Re: HAProxy 1.8 takes too long to update new config
I see, not only is the command stuck, the stats page does not load beyond top header part and that state continues to persist for about 5-6 minutes. Here is my configuration. defaults log global mode http compression algo gzip compression type text/html text/plain text/css application/javascript application/octet-stream application/json option httplog option dontlognull option redispatch option tcp-smart-accept option tcp-smart-connect option forwardfor timeout check 5s timeout client 50s timeout tunnel 6s timeout connect 20s timeout http-keep-alive 15s timeout http-request 30s timeout queue 20s timeout server 50s hash-balance-factor 125 balance hdr(Cookie) hash-type consistent djb2 stats enable stats hide-version stats uri /statz default-server inter 5s fall 3 rise 1 On Thu, Dec 28, 2017 at 4:27 AM Lukas Tribus wrote: > Hello, > > > > I'm sorry but I don't understand what you call "this" above nor what you > > mean by "updating the config". > > > > If the server is running in http2 mode, and servicing connections, > updating > > the config as shown below is no longer instantaneous. Takes over 5 > minutes. > > So what you are saying is that *the command* to reload haproxy takes 5 > minutes. I assume that within that timeframe, both old and new > requests continue to be served. > > H2 timeouts are different than H1. Please share you configuration, > especially everything timeout related. I assume "timeout client" is > more than a few seconds in your configuration? > > > You may want to lower "timeout client" and configure hard-stop-after > according to your expectation: > > http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#hard-stop-after > > The different timeout behavior in H2 is already documented: > > http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=75df9d7a7acb741e8413a6c2f3d6b6fe07b44bb8 > > > > Regarding email threads, you can look for the post on mail-archive.org > and hit that Reply-to button at the bottom (it will use the msg-id as > in the in-reply-to header, at least in thunderbird) - or you use do it > on your own, the msg-id of the email is at in the bottom line: > https://www.mail-archive.com/haproxy@formilux.org/ > > > lukas >
Re: HAProxy 1.8 takes too long to update new config
hard-stop-after is indeed helping, now it waits for exactly 30s (as configured in the hard-stop-after). Thanks for clarifying. On Thu, Dec 28, 2017 at 11:01 AM Robin Anil wrote: > I see, not only is the command stuck, the stats page does not load beyond > top header part and that state continues to persist for about 5-6 minutes. > > Here is my configuration. > > > defaults > log global > mode http > compression algo gzip > compression type text/html text/plain text/css application/javascript > application/octet-stream application/json > option httplog > option dontlognull > option redispatch > option tcp-smart-accept > option tcp-smart-connect > option forwardfor > timeout check 5s > timeout client 50s > timeout tunnel 6s > timeout connect 20s > timeout http-keep-alive 15s > timeout http-request 30s > timeout queue 20s > timeout server 50s > hash-balance-factor 125 > balance hdr(Cookie) > hash-type consistent djb2 > stats enable > stats hide-version > stats uri /statz > > default-server inter 5s fall 3 rise 1 > > On Thu, Dec 28, 2017 at 4:27 AM Lukas Tribus wrote: > >> Hello, >> >> >> > I'm sorry but I don't understand what you call "this" above nor what you >> > mean by "updating the config". >> > >> > If the server is running in http2 mode, and servicing connections, >> updating >> > the config as shown below is no longer instantaneous. Takes over 5 >> minutes. >> >> So what you are saying is that *the command* to reload haproxy takes 5 >> minutes. I assume that within that timeframe, both old and new >> requests continue to be served. >> >> H2 timeouts are different than H1. Please share you configuration, >> especially everything timeout related. I assume "timeout client" is >> more than a few seconds in your configuration? >> >> >> You may want to lower "timeout client" and configure hard-stop-after >> according to your expectation: >> >> http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#hard-stop-after >> >> The different timeout behavior in H2 is already documented: >> >> http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=75df9d7a7acb741e8413a6c2f3d6b6fe07b44bb8 >> >> >> >> Regarding email threads, you can look for the post on mail-archive.org >> and hit that Reply-to button at the bottom (it will use the msg-id as >> in the in-reply-to header, at least in thunderbird) - or you use do it >> on your own, the msg-id of the email is at in the bottom line: >> https://www.mail-archive.com/haproxy@formilux.org/ >> >> >> lukas >> >
Intermittent 500 errors with http/2 and 1.8.4
After seeing the 1.8.4 release, I decided to upgrade and try to run http/2 again Unfortunately as soon I enable it, I see intermittent 500 errors from our fleet. I know a lot of work went in between 1.8.2 and 1.8.3 to reduce these errors, yes its very few but a customer did notice, so we had to roll it back. Let me know how I can help diagnose this. [image: Screen Shot 2018-02-10 at 12.45.14 AM.png] All I need is to enable * alpn h2,http/1.1 *to see 500 errors happening. global log 127.0.0.1local0 log 127.0.0.1local1 notice maxconn 4096 pidfile /var/run/haproxy.pid stats socket /var/run/haproxy.stat mode 600 level admin daemon hard-stop-after 30s tune.ssl.default-dh-param 1024 tune.ssl.cachesize 10 ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets tune.bufsize 16384 tune.maxrewrite 1024 ssl-engine rdrand ssl-mode-async nbthread 4 defaults log global mode http compression algo gzip compression type text/html text/plain text/css application/javascript application/octet-stream application/json option httplog option dontlognull option redispatch option tcp-smart-accept option tcp-smart-connect option forwardfor timeout check 5s timeout client 50s timeout tunnel 6s timeout connect 20s timeout http-keep-alive 15s timeout http-request 30s timeout queue 20s timeout server 50s hash-balance-factor 125 balance hdr(Cookie) hash-type consistent djb2 stats enable stats hide-version stats auth redacted:redacted stats uri /redacted default-server inter 5s fall 3 rise 1 frontend health_check bind *:8081 timeout client 5s monitor-uri / frontend http bind :::80 v4v6 maxconn 4096 redirect scheme https if !{ ssl_fc } default_backend assets_backend frontend https bind :::443 v4v6 ssl crt /etc/ssl/www.ourdomain.com.crt no-sslv3 no-tls-tickets *# alpn h2,http/1.1* maxconn 4096 -