websockets termination?
Is there an equivalent to ssl termination mode for websockets? a websockets aware backend connecting to pure tcp server, haproxy negotiating the connection upgrade but only then connecting to the backend server. Is this possible? Thanks, Gerardo
Re: [PATCH] BUG/MAJOR: stream: fix tcp half connection expire causes cpu 100%
Hi again, CCing Richard who reported client-fin not always working in the past. Updates below. On Wed, Mar 08, 2017 at 09:21:37PM +0100, Willy Tarreau wrote: > Hi, > > On Wed, Mar 08, 2017 at 10:09:25PM +0800, longhb wrote: > > [PATCH] BUG/MAJOR: stream: fix tcp half connection expire causes cpu 100% > > > > Repetition condition: > > haproxy config: > > global: > > tune.bufsize 10485760 > > defaults > > timeout server-fin 90s > > timeout client-fin 90s > > backend node2 > > mode tcp > > timeout server 900s > > timeout connect 10s > > server def 127.0.0.1: > > frontend fe_api > > mode tcp > > timeout client 900s > > bind :1990 > > use_backend node2 > > timeout server-fin shorter than timeout server, the backend server > > sends data, this package is left in the cache of haproxy, the backend > > server continue sending fin package, haproxy recv fin package. this > > time the session information is as follows: > > 0x2373470: proto=tcpv4 src=127.0.0.1:39513 fe=fe_api be=node2 > > srv=def ts=08 age=1s calls=3 > > rq[f=848000h,i=0,an=00h,rx=14m58s,wx=,ax=] > > rp[f=8004c020h,i=0,an=00h,rx=,wx=14m58s,ax=] s0=[7,0h,fd=6,ex=] > > s1=[7,18h,fd=7,ex=] exp=14m58s > > rp has set the CF_SHUTR state, next, the client sends the fin package, > > session information is as follows: > > 0x2373470: proto=tcpv4 src=127.0.0.1:39513 fe=fe_api be=node2 > > srv=def ts=08 age=38s calls=4 rq[f=84a020h,i=0,an=00h,rx=,wx=,ax=] > > rp[f=8004c020h,i=0,an=00h,rx=1m11s,wx=14m21s,ax=] s0=[7,0h,fd=6,ex=] > > s1=[9,10h,fd=7,ex=] exp=1m11s > > After waiting 90s, session information is as follows: > > 0x2373470: proto=tcpv4 src=127.0.0.1:39513 fe=fe_api be=node2 > > srv=def ts=04 age=4m11s calls=718074391 > > rq[f=84a020h,i=0,an=00h,rx=,wx=,ax=] > > rp[f=8004c020h,i=0,an=00h,rx=?,wx=10m49s,ax=] s0=[7,0h,fd=6,ex=] > > s1=[9,10h,fd=7,ex=] exp=? run(nice=0) > > cpu information: > > 6899 root 20 0 112224 21408 4260 R 100.0 0.7 3:04.96 > > haproxy > > Buffering is set to ensure that there is data in the haproxy buffer, > > and haproxy can receive the fin package, set the CF_SHUTR flag, If the > > CF_SHUTR > > flag has been set, The following code does not clear the timeout > > message, > > causing cpu 100%: > > stream.c:process_stream: > > if (unlikely((res->flags & (CF_SHUTR|CF_READ_TIMEOUT)) == > > CF_READ_TIMEOUT)) { > > if (si_b->flags & SI_FL_NOHALF) > > si_b->flags |= SI_FL_NOLINGER; > > si_shutr(si_b); > > } > >If you have closed the read, set the read timeout does not make sense. > >With or without cf_shutr, read timeout is set: > >if (tick_isset(s->be->timeout.serverfin)) { > >res->rto = s->be->timeout.serverfin; > >res->rex = tick_add(now_ms, res->rto); > >} > > --- > > src/stream.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/src/stream.c b/src/stream.c > > index b333dec..ab07505 100644 > > --- a/src/stream.c > > +++ b/src/stream.c > > @@ -2095,7 +2095,7 @@ struct task *process_stream(struct task *t) > > if (req->flags & CF_READ_ERROR) > > si_b->flags |= SI_FL_NOLINGER; > > si_shutw(si_b); > > - if (tick_isset(s->be->timeout.serverfin)) { > > + if (tick_isset(s->be->timeout.serverfin) && !(res->flags & > > CF_SHUTR)) { > > res->rto = s->be->timeout.serverfin; > > res->rex = tick_add(now_ms, res->rto); > > } > > @@ -2278,7 +2278,7 @@ struct task *process_stream(struct task *t) > > if (unlikely((res->flags & (CF_SHUTW|CF_SHUTW_NOW)) == CF_SHUTW_NOW && > > channel_is_empty(res))) { > > si_shutw(si_f); > > - if (tick_isset(sess->fe->timeout.clientfin)) { > > + if (tick_isset(sess->fe->timeout.clientfin) && !(req->flags & > > CF_SHUTR)) { > > req->rto = sess->fe->timeout.clientfin; > > req->rex = tick_add(now_ms, req->rto); > > } > > I think the patch is fine but I'm worried because I think we changed > this recently (in 1.6 or so), so I'd like to investigate the cause of > the change (if any) and see what the motives were because I suspect > we possibly broke something else at the same time. So I looked at process_stream() and in fact I don't like the way we handle client-fin and server-fin at the moment. The issue you found is real but is not unique. We have the same in the "timeout tunnel" handling in combination with client-fin/server-fin because the timeouts
Does reqrep work?
Hi, I wonder if reqrep works? I need to replace request /ABC/collection/XXX/ to /collection/collection_XXX/ So I have the following line in my haproxy.cfg: reqrep ^([^\ :]*)\ /ABC/collection/(.*)/ \1\ /collection/collection_\2/ But instead of substitution I got nothing but error 400: Mar 9 17:12:46 localhost haproxy[16670]: 185.94.108.37:41710 [09/Mar/2017:17:12:46.373] fe~ be/srv1 518/0/0/1/519 400 2764 - - 446/432/26/13/0 0/0 "GET /ABC/collection/1592/ HTTP/1.1" What is wrong, please? haproxy -vv: HA-Proxy version 1.5.2 2014/07/12 Copyright 2000-2014 Willy TarreauBuild options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll.
Re: [PATCH] [MEDIUM] Improve "no free ports" error case
Hi Willy, Excellent, I will try this idea, it should definitely help! Thanks for the explanations. Regards, - Krishna On Thu, Mar 9, 2017 at 1:37 PM, Willy Tarreauwrote: > On Thu, Mar 09, 2017 at 12:50:16PM +0530, Krishna Kumar (Engineering) > wrote: > > 1. About 'retries', I am not sure if it works for connect() failing > > synchronously on the > > local system (as opposed to getting a timeout/refused via callback). > > Yes it normally does. I've been using it for the same purpose in certain > situations (eg: binding to a source port range while some daemons are > later bound into that range). > > > The > > document > > on retries says: > > > > " is the number of times a connection attempt should be > retried > > on > > a server when a connection either is refused or times out. > The > > default value is 3. > > " > > > > The two conditions above don't fall in our use case. > > It's still a refused connection :-) > > > The way I understood was that > > retries happens during the callback handler. Also I am not sure if there > is > > any way to circumvent the "1 second" gap for a retry. > > Hmmm I have to check. In fact when the LB algorithm is not determinist > we immediately retry on another server. If we're supposed to end up only > on the same server we indeed apply the delay. But if it's a synchronous > error, I don't know. And I think it's one case (especially -EADDRNOTAVAIL) > where we should immediately retry. > > > 2. For nolinger, it was not recommended in the document, > > It's indeed strongly recommended against, mainly because we've started > to see it in configs copy-pasted from blogs without understanding the > impacts. > > > and also I wonder if any data > > loss can happen if the socket is not lingered for some time beyond the > FIN > > packet that > > the remote server sent for doing the close(), delayed data packets, etc. > > The data loss happens only with outgoing data, so for HTTP it's data > sent to the client which are at risk. Data coming from the server are > properly consumed. In fact, when you configure "http-server-close", > the nolinger is automatically enabled in your back so that haproxy > can close the server connection without accumulating time-waits. > > > 3. Ports: Actually each HAProxy process has 400 ports limitation to a > > single backend, > > and there are many haproxy processes on this and other servers. The ports > > are split per > > process and per system. E.g. system1 has 'n' processes and each have a > > separate port > > range from each other, system2 has 'n' processes and a completely > different > > port range. > > For infra reasons, we are restricting the total port range. The unique > > ports for different > > haproxy processes running on same system is to avoid attempting to use > the > > same port > > (first port# in the range) by two processes and failing in connect, when > > attempting to > > connect to the same remote server. Hope I explained that clearly. > > Yep I clearly see the use case. That's one of the rare cases where it's > interesting to use SNAT between your haproxy nodes and the internet. This > way you'll use a unified ports pool for all your nodes and will not have > to reserve port ranges per system and per process. Each process will then > share the system's local source ports, and each system will have a > different > address. Then the SNAT will convert these IP1..N:port1..N to the public IP > address and an available port. This will offer you more flexibility to add > or remove nodes/processes etc. Maybe your total traffic cannot pass through > a single SNAT box though in which case I understand that you don't have > much choice. However you could then at least not force each process' port > range and instead fix the system's local port range so that you know that > all processes of a single machine share a same port range. That's already > better because you won't be forcing to assign ports from unfinished > connections. > > Willy >
Re: [PATCH] [MEDIUM] Improve "no free ports" error case
On Thu, Mar 09, 2017 at 12:50:16PM +0530, Krishna Kumar (Engineering) wrote: > 1. About 'retries', I am not sure if it works for connect() failing > synchronously on the > local system (as opposed to getting a timeout/refused via callback). Yes it normally does. I've been using it for the same purpose in certain situations (eg: binding to a source port range while some daemons are later bound into that range). > The > document > on retries says: > > " is the number of times a connection attempt should be retried > on > a server when a connection either is refused or times out. The > default value is 3. > " > > The two conditions above don't fall in our use case. It's still a refused connection :-) > The way I understood was that > retries happens during the callback handler. Also I am not sure if there is > any way to circumvent the "1 second" gap for a retry. Hmmm I have to check. In fact when the LB algorithm is not determinist we immediately retry on another server. If we're supposed to end up only on the same server we indeed apply the delay. But if it's a synchronous error, I don't know. And I think it's one case (especially -EADDRNOTAVAIL) where we should immediately retry. > 2. For nolinger, it was not recommended in the document, It's indeed strongly recommended against, mainly because we've started to see it in configs copy-pasted from blogs without understanding the impacts. > and also I wonder if any data > loss can happen if the socket is not lingered for some time beyond the FIN > packet that > the remote server sent for doing the close(), delayed data packets, etc. The data loss happens only with outgoing data, so for HTTP it's data sent to the client which are at risk. Data coming from the server are properly consumed. In fact, when you configure "http-server-close", the nolinger is automatically enabled in your back so that haproxy can close the server connection without accumulating time-waits. > 3. Ports: Actually each HAProxy process has 400 ports limitation to a > single backend, > and there are many haproxy processes on this and other servers. The ports > are split per > process and per system. E.g. system1 has 'n' processes and each have a > separate port > range from each other, system2 has 'n' processes and a completely different > port range. > For infra reasons, we are restricting the total port range. The unique > ports for different > haproxy processes running on same system is to avoid attempting to use the > same port > (first port# in the range) by two processes and failing in connect, when > attempting to > connect to the same remote server. Hope I explained that clearly. Yep I clearly see the use case. That's one of the rare cases where it's interesting to use SNAT between your haproxy nodes and the internet. This way you'll use a unified ports pool for all your nodes and will not have to reserve port ranges per system and per process. Each process will then share the system's local source ports, and each system will have a different address. Then the SNAT will convert these IP1..N:port1..N to the public IP address and an available port. This will offer you more flexibility to add or remove nodes/processes etc. Maybe your total traffic cannot pass through a single SNAT box though in which case I understand that you don't have much choice. However you could then at least not force each process' port range and instead fix the system's local port range so that you know that all processes of a single machine share a same port range. That's already better because you won't be forcing to assign ports from unfinished connections. Willy