Re: graceful shutdown maximum time
On Fri, Aug 18, 2017, at 02:53 PM, William Lallemand wrote: > On Fri, Aug 18, 2017 at 02:26:29PM -0700, Justin Karneges wrote: > > Is there a way to configure HAProxy to have a maximum graceful shutdown > > time? For example it would stop listening for new connections > > immediately, and then after a specified amount of time it would close > > all connections and terminate, regardless of what kind of activity might > > be going on within those connections. > > The "hard-stop-after" keyword do what you want: > > https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#hard-stop-after Cool! Looks like a relatively new feature. I'll check it out. Thanks, Justin
Re: graceful shutdown maximum time
On Fri, Aug 18, 2017 at 02:26:29PM -0700, Justin Karneges wrote: > Hi, > > When HAProxy is shut down gracefully, my understanding is that it waits > for all open connections to be closed before it will terminate. However, > if the connections don't ever close then HAProxy may never shut down (or > perhaps it takes a very long time, I'm not sure). This is mainly a > problem with long-lived connections that remain continuously active, so > timeouts won't trigger either. > > Is there a way to configure HAProxy to have a maximum graceful shutdown > time? For example it would stop listening for new connections > immediately, and then after a specified amount of time it would close > all connections and terminate, regardless of what kind of activity might > be going on within those connections. > > Otherwise my plan is to create a cron job that kills old haproxy > processes on a delay. Would be nice if there were a better option > though. > > Thanks for any suggestion. > > Justin > The "hard-stop-after" keyword do what you want: https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#hard-stop-after -- William Lallemand
graceful shutdown maximum time
Hi, When HAProxy is shut down gracefully, my understanding is that it waits for all open connections to be closed before it will terminate. However, if the connections don't ever close then HAProxy may never shut down (or perhaps it takes a very long time, I'm not sure). This is mainly a problem with long-lived connections that remain continuously active, so timeouts won't trigger either. Is there a way to configure HAProxy to have a maximum graceful shutdown time? For example it would stop listening for new connections immediately, and then after a specified amount of time it would close all connections and terminate, regardless of what kind of activity might be going on within those connections. Otherwise my plan is to create a cron job that kills old haproxy processes on a delay. Would be nice if there were a better option though. Thanks for any suggestion. Justin
RE: Experimental / broken HTTP/2 support
king about having some optional fields that are supposed
to be easy to use. Note that we had the same issue with SSL long
ago, ending with "~" after the frontend's name in the logs...
Better avoid this for H2. Ideas are welcome.
There are some good points as well, proving we're not too far. For example,
the stats work over H2, even when enabling compression!
I'm pretty sure you'll find a ton of bugs there, and don't look at the code
too closely, sometimes I really had to put a few lines just to plug a hole
through which water was leaking.
I'm more interested in observations, like "oh too bad you didn't do this" or
"why not report the version this way" etc. You'll all see that it's easierto
think about it when playing with it.
Currently I'm running it locally on my laptop, listening on 127.0.0.2 for
traffic that my browser sends there when I want to access h2.haproxy.org or
h2.haproxy.com (then the Host header gets rewritten).
I could notice already that loading haproxy.com through this is a bit faster
thanks to the higher parallelism allowing more objects to be loaded at once.
Those who want to test will have to retrieve branch 20170818-h2-2 from the
git development repository :
git clone -b 20170818-h2-2 http://git.haproxy.org/git/haproxy.git/
It builds as usual. For the config, you have to specify "alpn h2" on the
"bind" line. This requires openssl-1.0.2. If you have an older version you
can try with "npn h2", which some browsers support as well.
The first failed connection in the logs above was made with ALPN then a
fallback to NPN was done. Or maybe both were attempted in parallel, I
haven't checked much.
The working config I'm using here is the following :
global
log 127.0.0.1:5514 local0
stats socket /tmp/sock1 mode 666 level admin
stats timeout 1h
tune.ssl.default-dh-param 1024
ssl-server-verify none
tune.bufsize 16384
defaults
mode http
timeout client 20s
timeout server 20s
timeout connect 1s
listen www
log global
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC
%CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs http=%[http_major] %{+Q}r"
bind 127.0.0.2:443 ssl crt rsa+dh2048.pem alpn h2
server www www.haproxy.org:443 ssl verify none
http-request set-header host www.haproxy.org
stats uri /stat
And in my /etc/hosts, I have :
127.0.0.2 h2.haproxy.org
Then I can aim by browser at h2.haproxy.org, get over the cert warning, and
continue from there. It's interesting to see how certain sites with many
objects react, even those which are far away, because the browser visibly
allows much more parallelism than on HTTP/1. If some want to try on
www.haproxy.com (many more objects, it's a modern site :-)) you'll need two
such instances, one for "h2." replacing "www" and another one for "cdn."
running on a different address. And you'll have to modify your /etc/hosts
and later complain that you can't access the site anymore after you forget
your test (it happened to me already). So it's better if you find another
site where most of the objects are on the same host matching the URL bar,
it's much easier to configure and to test.
I'm not really asking for bug reports at this step, I expect a lot. However
if you manage to find 100% reproducible cases for connection hangs, timeouts
or stuff like this, it can still indicate I missed something, so do not
hesitate. Just don't be sad if I only respond "yeah I know".
I'm still getting hopes for something working (a bit better) by mid-end of
october, which means we could have a very fresh H2 support in haproxy 1.8,
very likely still tagged experimental.
Have fun,
Willy
---
This email has been checked for viruses by AVG.
http://www.avg.com
Experimental / broken HTTP/2 support
ams from the incoming connection, translating them into H1 and processing H1 requests just like before. That's why the logs still report "HTTP/1.1", it's what is presented into the version string. Among the things that are still limitations that could possibly be overcome before the 1.8 release, I can cite : - header field names don't have any single upper case letter (as is the case in H2), so it might be possible that some bogus hard-coded products don't match "Host" when "host:" is present for example. It's not very hard to place an upper case after each "-" but for now it's not done ; I'm interested in interoperability issue reports if any. - no server-side keep-alive for now. The cause is simple : the streams in HTTP/2 only transport a single request so there is no reuse by the same stream and for now we have nowhere to place our idle connection, meaning that they have to be closed all the time ; we definitely need to address this by having per-session lists, but it will not be trivial so I preferred to focus on protocol processing for now, which already isn't the easiest thing to implement ; - no data upload yet, DATA frames are simply ignoread, so POST, PUT and CONNECT will not work. It's not a huge work to get them to work now but doing it in an architecture that's going to die is pointless. - CONTINUATION frames are not supported for now, it's due to the current architecture making this very complicated. - it's mandatory that the buffer size (tune.bufsize) is at least 16384 bytes. There's no such control for now during the config parsing so it starts and H2 connections are simply aborted when it's discovered that the buffer is too small and the browser retries using HTTP/1. - trailers are not forwarded from the server to the client yet. Not critical for most tests. - logs also report the H2 front connection and bad requests. This will have to be addressed later. - request/response aborts are not translated to RST_STREAM frames yet, so I guess it's the browser which will detect some inconsistency and break the connection, or we'll send a GOAWAY frame to break it. - no graceful shutdown of the connection (will probably not exist for the release, we'll see). - the HTTP/1 to 2 upgrade method is not supported yet, though I'm not much worried about it at the moment. - no trivial way to report HTTP/2 in the logs. I'm using a sample fetch function reporting the on-wire format as 1 or 2 for now. I considered replacing "HTTP/1.1" with "HTTP/2.0" in the logs but that's inaccurate since we really process "1.1" so it might be confusing to those dealing with regex which don't seem to match, and in addition "HTTP/2.0" is not the correct version string, the correct one is "HTTP/2". But writing this without the dot and the minor version is going to break some log processing tools. Thus I was thinking about having some optional fields that are supposed to be easy to use. Note that we had the same issue with SSL long ago, ending with "~" after the frontend's name in the logs... Better avoid this for H2. Ideas are welcome. There are some good points as well, proving we're not too far. For example, the stats work over H2, even when enabling compression! I'm pretty sure you'll find a ton of bugs there, and don't look at the code too closely, sometimes I really had to put a few lines just to plug a hole through which water was leaking. I'm more interested in observations, like "oh too bad you didn't do this" or "why not report the version this way" etc. You'll all see that it's easierto think about it when playing with it. Currently I'm running it locally on my laptop, listening on 127.0.0.2 for traffic that my browser sends there when I want to access h2.haproxy.org or h2.haproxy.com (then the Host header gets rewritten). I could notice already that loading haproxy.com through this is a bit faster thanks to the higher parallelism allowing more objects to be loaded at once. Those who want to test will have to retrieve branch 20170818-h2-2 from the git development repository : git clone -b 20170818-h2-2 http://git.haproxy.org/git/haproxy.git/ It builds as usual. For the config, you have to specify "alpn h2" on the "bind" line. This requires openssl-1.0.2. If you have an older version you can try with "npn h2", which some browsers support as well. The first failed connection in the logs above was made with ALPN then a fallback to NPN was done. Or maybe both were attempted in parallel, I haven't checked much. The working config I'm using here is the following : global log 127.0.0.1:5514 local0 stats socket /tmp/sock1 mode 666 level admin stats timeout 1h tune.ss
[ANNOUNCE] haproxy-1.7.9
Hi, HAProxy 1.7.9 was released on 2017/08/18. It added 19 new commits after version 1.7.8. Among the fixes, there is the resolution of the wrong termination state which was introduced in 1.7.5 and a lot of fixes for lua. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Sources : http://www.haproxy.org/download/1.7/src/ Git repository : http://git.haproxy.org/git/haproxy-1.7.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-1.7.git Changelog: http://www.haproxy.org/download/1.7/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Adis Nezirovic (1): BUG/MINOR: lua: Fix bitwise logic for hlua_server_check_* functions. Christopher Faulet (5): BUG/MINOR: http: Set the response error state in http_sync_res_state MINOR: http: Reorder/rewrite checks in http_resync_states MINOR: http: Switch requests/responses in TUNNEL mode only by checking txn flags BUG/MEDIUM: http: Switch HTTP responses in TUNNEL mode when body length is undefined BUG/MAJOR: http: Fix possible infinity loop in http_sync_(req|res)_state Frédéric Lécaille (1): BUG/MINOR: peers: peer synchronization issue (with several peers sections). Nenad Merdanovic (2): BUG/MINOR: lua: Fix Server.get_addr() port values BUG/MINOR: lua: Correctly use INET6_ADDRSTRLEN in Server.get_addr() Thierry FOURNIER (4): BUG/MINOR: lua: In error case, the safe mode is not removed BUG/MINOR: lua: executes the function destroying the Lua session in safe mode BUG/MAJOR: lua/socket: resources not detroyed when the socket is aborted BUG/MEDIUM: lua: bad memory access Willy Tarreau (5): DOC: update CONTRIBUTING regarding optional parts and message format DOC: update the list of OpenSSL versions in the README MINOR: tools: add a portable timegm() alternative BUILD: lua: replace timegm() with my_timegm() to fix build on Solaris 10 BUG/MINOR: lua: always detach the tcp/http tasks before freeing them ben51degrees (1): DOC: Updated 51Degrees git URL to point to a stable version. -- William Lallemand
