Patch for ALPN compatibility with OpenSSL development
Hi all, At GitHub we’ve worked on a patch to make HAProxy’s ALPN code compatible with the patches for it that have landed in OpenSSL: http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6f017a8f9db3a79f3a3406cf8d493ccd346db691 This final version is slightly different from what HAProxy currently expects, which is based on some custom OpenSSL patches. Let me know if this is a good approach towards fixing this problem or that it should be done differently. - Dirkjan Bussink 0001-Use-ALPN-support-as-it-will-be-available-in-OpenSSL-.patch Description: Binary data
check works on one backend but not another
I'm running HAProxy version 1.5-dev21-6b07bf7 and have a strange behaviour issue with the check command. Here's my config: backend nginx modehttp balance leastconn server app1app1.prod:81check inter 3s port 9700 rise 1 fall 2 maxconn 2000 server app2app2.prod:81check inter 3s port 9700 rise 1 fall 2 maxconn 2000 server downlocalhost:81 backup backend nginx-ssl modetcp balance leastconn server app1app1.prod:444 check inter 3s port 9700 rise 1 fall 2 maxconn 2000 server app2app2.prod:444 check inter 3s port 9700 rise 1 fall 2 maxconn 2000 server downlocalhost:81 backup The check talks to a service I have on each app server that confirms all the required backend services are enabled (nginx, php-fpm, varnish) and reports back: # telnet app1.prod 9700 Trying 172.16.10.1... Connected to app1.prod. Escape character is '^]'. HTTP/1.1 503 Service Unavailable Content-Type: text/plain Stack DOWN Now, in testing I've killed one of the three and I get the above answer, which works on the "nginx" backend call, but does not work on the ssl one, as per this screenshot: http://i.imgur.com/oofs3S7.png The thing is, it's the *exact same call* for both. Any explanation for this?
Re: HAProxy as a TCP Fast Open Client
Hi Willy Did some more investigation on the case where the application request is too large to fit within the initial SYN. Here is my test setup: Web clients ——> haproxy ——> long-thin-pipe —> haproxy--—> web servers TFO Client TFO Server Client sends an HTTP request larger than MSS, the client side haproxy uses TFO and puts as much data as possible within the initial SYN. When SYN ACK is returned, the remaining request data is sent. On closer inspection although the correct number of octets are sent, the octets in the continuation packet are all NUL. E.g. Debug shows 1500 octets in the call to sendto() and a return value of 1500. Wireshark shows TFO sending 1420 octets in the SYN. After SYN ACK comes back, 80 octets are sent in the next packet, but these 80 octets are all NUL. Looks like something broken in the TFO client, but would be good to see if others can duplicate my results. I’m testing using VMware which I think emulates TCP offload by default, wondering whether that could be the cause? Regarding default values for the TFO backlog - I was concerned that if this is maxconn then is there a DoS vulnerability? Eg if a TFO client streams SYNs with random data at you, each of these ties up an haproxy connection for a while, starving other clients? ATB -David
Re: Patch for ALPN compatibility with OpenSSL development
It would be great to see this patch find it’s way into the next dev release. Let us know if any changes need to be made. -- Joe Williams williams@gmail.com WILLI567-ARIN On February 13, 2014 at 4:32:42 AM, Dirkjan Bussink (d.buss...@gmail.com) wrote: Hi all, At GitHub we’ve worked on a patch to make HAProxy’s ALPN code compatible with the patches for it that have landed in OpenSSL: http://git.openssl.org/gitweb/?p=openssl.git;a=commit;h=6f017a8f9db3a79f3a3406cf8d493ccd346db691 This final version is slightly different from what HAProxy currently expects, which is based on some custom OpenSSL patches. Let me know if this is a good approach towards fixing this problem or that it should be done differently. - Dirkjan Bussink - 0001-Use-ALPN-support-as-it-will-be-available-in-OpenSSL-.patch, 4.2 KB
Re: check works on one backend but not another
Hi Jeremy, You're checking with a TCP handshake only, which is not aware of HTTP status response. add "option httpchk" in your backend and add the keyword "check-ssl" on your server lines to tell HAProxy the health check must be ciphered. Baptiste On Thu, Feb 13, 2014 at 2:42 PM, Jeremy Wilson wrote: > I'm running HAProxy version 1.5-dev21-6b07bf7 and have a strange behaviour > issue with the check command. > > Here's my config: > > backend nginx > modehttp > balance leastconn > > server app1app1.prod:81check inter 3s port 9700 rise 1 fall > 2 maxconn 2000 > server app2app2.prod:81check inter 3s port 9700 rise 1 fall > 2 maxconn 2000 > > server downlocalhost:81 backup > > backend nginx-ssl > modetcp > balance leastconn > > server app1app1.prod:444 check inter 3s port 9700 rise 1 fall > 2 maxconn 2000 > server app2app2.prod:444 check inter 3s port 9700 rise 1 fall > 2 maxconn 2000 > > server downlocalhost:81 backup > > The check talks to a service I have on each app server that confirms all the > required backend services are enabled (nginx, php-fpm, varnish) and reports > back: > > # telnet app1.prod 9700 > Trying 172.16.10.1... > Connected to app1.prod. > Escape character is '^]'. > HTTP/1.1 503 Service Unavailable > Content-Type: text/plain > > Stack DOWN > > Now, in testing I've killed one of the three and I get the above answer, > which works on the "nginx" backend call, but does not work on the ssl one, as > per this screenshot: > > http://i.imgur.com/oofs3S7.png > > The thing is, it's the *exact same call* for both. > > Any explanation for this? > >
Re: HAProxy as a TCP Fast Open Client
Hi David, On Thu, Feb 13, 2014 at 01:50:16PM +, David Harrold wrote: > Hi Willy > > Did some more investigation on the case where the application request is too > large to > fit within the initial SYN. > > Here is my test setup: > > Web clients ??> haproxy ??> long-thin-pipe ?> haproxy--?> web > servers > TFO Client > TFO Server > > Client sends an HTTP request larger than MSS, the client side haproxy uses > TFO and puts as much data > as possible within the initial SYN. When SYN ACK is returned, the remaining > request data is sent. > On closer inspection although the correct number of octets are sent, the > octets in the continuation packet are all NUL. > > E.g. Debug shows 1500 octets in the call to sendto() and a return value of > 1500. > Wireshark shows TFO sending 1420 octets in the SYN. After SYN ACK comes back, > 80 octets are sent in the next packet, > but these 80 octets are all NUL. OK so that's clearly a bug below. > Looks like something broken in the TFO client, but would be good to see if > others can duplicate my results. > > I?m testing using VMware which I think emulates TCP offload by default, > wondering whether that could be the cause? Could be indeed, we've got a few issues with GRO/GSO in the past as well. I'll have to run some tests with your patch to see with different kernels if I can reproduce the same issue. It is also possible that it was a known old bug that's already fixed but not backported to some stable branches. > Regarding default values for the TFO backlog - I was concerned that if this > is maxconn then is > there a DoS vulnerability? Eg if a TFO client streams SYNs with random data > at you, each of these ties up > an haproxy connection for a while, starving other clients? But it's the same with real connections in practice, because even when the connection is accepted, we still need to parse it. This is also the reason for having a short http-request timeout. For example, if you support 100k concurrent connections on the frontend and wait for a request for 5 seconds, a client will have to send 20k conns/s to keep you full. In practice, even at this rate, you'll accept 100k extra conns in the backlog which will get served but will have to wait 0 to 5s on average. The worst thing to do is to reduce the accept rate, which lowers the bar for attackers. The higher the limit, the more information we have for dealing with pending data. One of the easy things we are already able to do is count the number of concurrent connections per source address for example. Something we cannot do if we refrain from accepting these connections. I also have some memories about the network stack defending itself when a SYN queue overflows, it would reject TFO or accelerate the regeneration of cookies, I don't remember exactly. Cheers, Willy
Re: check works on one backend but not another
On 2014-02-13, at 11:11 AM, Baptiste wrote: > You're checking with a TCP handshake only, which is not aware of HTTP > status response. > add "option httpchk" in your backend and add the keyword "check-ssl" > on your server lines to tell HAProxy the health check must be > ciphered. But the check itself is not SSL - it connects to port 9700 of the server, which just echoes out a HTTP response.
Re: speeding up failover
Read the manual about `rise` and `fall` parameters. These allow you to control how many successive checks must pass or fail before the server transitions up or down (rises / falls). The check interval is used as the check timeout unless you specify a check timeout. See "timeout check" in the manual. http://haproxy.1wt.eu/download/1.5/doc/configuration.txt
Re: check works on one backend but not another
While this does not answer your question per se you can use the track option to eliminate the duplicate check. In other words, the SSL backend can track the checks done by the non-SSL backend. backend nginx-ssl modetcp balance leastconn server app1app1.prod:444 track nginx/app1 server app2app2.prod:444 track nginx/app2 server downlocalhost:81 backup
Re: check works on one backend but not another
Baptiste gave you the proper answer already. The SSL backend is using TCP mode, so the check is a TCP check without the `option httpchk` defined on the backend, which just checks that the port is open. Add the httpchk option without check-ssl and you will be all set. Or you can use track to skip the duplicate check against port 9700.
job offer
Join International Programs For Charity by working directly from your computer as Charity Donation Officer with $3,000 monthly. Send Your Full Name, Address, Date of Birth and phone number by email: unicefaustra...@in.com or Click here to Apply online: http://www.keeclinic.com/bbs//data/employ/ Kelly Scott Mullen HR Manager International Programs For Charity Unicef Donations
