Re: http-send-name-header buffer corruption (1.4.25)
Hi Cyril, On Sun, Sep 07, 2014 at 06:30:40PM +0200, Cyril Bonté wrote: Hi all, Le 06/09/2014 08:28, Willy Tarreau a écrit : To be honnest, I think it is necessary, and I hope it will be enough. We had this issue in 1.5-dev as well a long time ago and the problem was the same, the beginning of the message was not moved. Now 1.5's buffer semantics have changed to make all this much simpler. There is one thing to test to confirm it's enough : make a farm with two untested servers, one which does not respond and a valid one so that there's a retry and redispatch. Haproxy will first pick server1, prepend the header, and try to send the request there. After the failure it will redispatch to server2 and rewind the buffer to change the header. On server2 you should see a valid request if the fix is OK. That's how I managed to fix it in 1.5-dev. The config below should be enough to reproduce it : listen test :10002 balance roundrobin mode http option http-server-close http-send-name-header Host retries 1 option redispatch server bad 127.0.0.1:12345 server good 127.0.0.1:80 I also remember that rewinding POST requests is harder than rewinding GET because the body was already partially parsed. I remember something in 1.4 which we did to take care of that, either we refrain from parsing the body before connecting when there's the send-name-header, or we could correctly rewind. So sending a POST request through the config above will tell you if that's OK :-) I was optimistic during my tests unless I tried sending POST requests with a body longer than the buffer size. In certain conditions, haproxy segfaults. For example, this is the case with the bad server, no need to redispatch. I need some time to remember how buffers work in 1.4, in order to fix the segfault. Argh, crap! I fear that we're opening a new can of worms :-( At least we have working code in 1.5 so we know there's a fix, but the two versions are so much different that we cannot port 1.5 changes back to 1.4. It's possible however that we'll have to add a WAIT_CONN flag to prevent the body from being parsed/forwarded until the connection is opened. Willy
Re: HAProxy 1.5 incorrectly marks servers as DOWN
I'm more and more suspecting that this is a new bug introduced on the 1.5.x
branch. We don't have these issues on our 1.4 deployment and it seems that
the check.c file has undergone a lot of changes and refactoring between
1.4.18 and 1.5.4.
I started to look how the check subsystem works but the learning curve is
quite high on this one. I'm hoping that maybe Willy (it seems that your
name is all over the changelog on the check.c) might have some clues which
I could then pursue further. The biggest problem is that this bug seems to
be nondeterministic, but on the good side is that I can run modified
haproxy binarines in my environment so that I could trace this further.
On Mon, Sep 8, 2014 at 11:30 AM, Juho Mäkinen j...@unity3d.com wrote:
On Thu, Sep 4, 2014 at 11:35 PM, Pavlos Parissis
pavlos.paris...@gmail.com wrote:
On 04/09/2014 08:55 πμ, Juho Mäkinen wrote:
I'm upgrading my old 1.4.18 haproxies to 1.5.4 and I have a mysterious
problem where haproxy marks some backend servers as being DOWN with a
message L4TOUT in 2000ms.
Are you sure that you haven't reached any sort of limits on your backend
servers? Number of open files and etc...
Quite sure because I can always use curl from the haproxy machine to the
backend machine and I get the response to the check command always without
any delays.
Are you sure that backend servers return a response with HTTP status 200
on healtchecks?
Yes. I also ran strace on a single haproxy process when the haproxy marked
multiple backends as being down. Here's an example output:
08:06:07.302582 connect(30, {sa_family=AF_INET, sin_port=htons(3500),
sin_addr=inet_addr(172.16.6.102)}, 16) = -1 EINPROGRESS (Operation now in
progress)
08:06:07.303024 recvfrom(30, 0x1305494, 16384, 0, 0, 0) = -1 EAGAIN
(Resource temporarily unavailable)
08:06:07.303097 getsockopt(30, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
08:06:07.303167 sendto(30, GET /check HTTP/1.0\r\n\r\n, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
08:06:07.304522 recvfrom(30, HTTP/1.1 200 OK\r\nX-Powered-By:
Express\r\nAccess-Control-Allow-Origin: *\r\nAccess-Control-Allow-Methods:
GET, HEAD, POST, PUT, DELE..., 16384, 0, NULL, NULL) = 503
08:06:07.304603 setsockopt(30, SOL_SOCKET, SO_LINGER, {onoff=1, linger=0},
8) = 0
08:06:07.304666 close(30) = 0
So the server clearly sends an HTTP 200 OK response, in just 1.9 ms. I
analysed around 20 different checks via the strace to the same backend
(which is marked down by haproxy) and none of them was over one second.
Here's an example from haproxy logging what happens when the problem
starts:
Sep 8 07:22:25 localhost haproxy[24282]: [08/Sep/2014:07:22:24.615] https
comet-getcampaigns/comet-172.16.2.97:3500 423/0/1/3/427 200 502 - -
1577/1577/3/1/0 0/0 GET /mobile HTTP/1.1
Sep 8 07:22:25 localhost haproxy[24284]: [08/Sep/2014:07:22:24.280]
https~ comet-getcampaigns/comet-172.16.2.97:3500 771/0/2/346/1121 200 40370
- - 2769/2769/6/0/0 0/0 GET /mobile HTTP/1.1Sep 8 07:22:25
localhost haproxy[24284]: [08/Sep/2014:07:22:25.090] https~
comet-getcampaigns/comet-172.16.2.97:3500 379/0/2/-1/804 502 204 - - SH--
2733/2733/7/0/0 0/0 GET /mobile HTTP/1.1
Sep 8 07:22:25 localhost haproxy[24280]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 231ms, status: 2/3 UP.
Sep 8 07:22:25 localhost haproxy[24281]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 217ms, status: 2/3 UP.
Sep 8 07:22:25 localhost haproxy[24282]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 137ms, status: 2/3 UP.
Sep 8 07:22:25 localhost haproxy[24284]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 393ms, status: 2/3 UP.Sep
8 07:22:25 localhost haproxy[24284]: [08/Sep/2014:07:22:25.661] https
comet-getcampaigns/comet-172.16.2.97:3500 305/0/1/-1/314 -1 0 - - SD--
2718/2718/5/0/0 0/0 GET /mobile HTTP/1.1
Sep 8 07:22:27 localhost haproxy[24278]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4 connection
problem, info: Connection refused, check duration: 0ms, status: 2/3 UP.
Sep 8 07:22:27 localhost haproxy[24279]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4 connection
problem, info: Connection refused, check duration: 0ms, status: 2/3 UP.
Sep 8 07:22:28 localhost haproxy[24280]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4 connection
problem, info: Connection refused, check duration: 2ms, status: 1/3 UP.
Sep 8 07:22:28 localhost haproxy[24284]: Health check for server
Re: Recommended SSL ciphers and settings
Hi, On 09.09.2014 11:43, pablo platt wrote: I've tried both options and I'm still not getting A+. Unfortunately, I can't ask the user what the error is. If I'll run into this again, I'll try to get this info. To reach A+ you need rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains if ssl-proxy ssl-proxy means here the connection is ssl. and a cipher list like -- EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384: EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 -- Together it should work. As you can see we have no longer RC4 ciphers, cheers thomas Thanks On Mon, Sep 8, 2014 at 9:46 AM, Jarno Huuskonen jarno.huusko...@uef.fi mailto:jarno.huusko...@uef.fi wrote: Hi, On Sun, Sep 07, pablo platt wrote: Hi, I'm using haproxy to terminate SSL and it works for most of my users. I have alphassl wildcard certificate. I'm using SSL to improve WebSockets and RTMP connections of port 443. I don't have sensitive data or e-commerce. I have one user that see a warning in Chrome and can't use my website. Do you know what warning chrome gives to that user ? Is it possible that this the warning is because an antivirus is not happy with the default ciphers or other ssl settings? When running a test https://sslcheck.globalsign.com/en_US I'm getting: Sessions may be vulnerable to BEAST attack Server has not enabled HTTP Strict-Transport-Security Server has SSL v3 enabled Server is using RC4-based ciphersuites which have known vulnerabilities Server configuration does not meet FIPS guidelines Server does not have OCSP stapling configured Server has not yet upgraded to a Extended Validation certificate Server does not have SPDY enabled I found one suggestion: bind 10.0.0.9:443 http://10.0.0.9:443 name https ssl crt /path/to/domain.pem ciphers RC4:HIGH:!aNULL:!MD5 http://blog.haproxy.com/2013/01/21/mitigating-the-ssl-beast-attack-using-the-aloha-load-balancer-haproxy/ And another: bind 0.0.0.0:443 http://0.0.0.0:443 ssl crt /etc/cert.pem nosslv3 prefer-server-ciphers ciphers RC4-SHA:AES128-SHA:AES256-SHA Both gives me other warnings. What other warnings ? (Does haproxy give you warnings/errors or client browsers) ? Perhaps you could try ciphersuite from: https://wiki.mozilla.org/Security/Server_Side_TLS for example in global: ssl-default-bind-ciphers ... or on bind: bind 0.0.0.0:443 http://0.0.0.0:443 ssl crt /path/to/crt ciphers ... To enable ocsp stapling see haproxy config: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.1-crt http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2-set%20ssl%20ocsp-response -Jarno -- Jarno Huuskonen -- Thomas Heil - Email: h...@terminal-consulting.de Tel: 0176 / 44555622 --
Re: [PATCH] [RFC] Linux network namespace support for haproxy
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 9/09/2014 5:19 p.m., Willy Tarreau wrote:
Hi Dave,
On Mon, Sep 08, 2014 at 04:25:22PM -0400, Dave McCowan wrote:
Second patch :
diff --git a/include/types/connection.h
b/include/types/connection.h index 89f4f38..a75d45d 100644
--- a/include/types/connection.h +++
b/include/types/connection.h @@ -334,6 +334,7 @@ struct
proxy_hdr_v2 { #define PP2_TYPE_SSL 0x20 #define
PP2_TYPE_SSL_VERSION 0x21 #define PP2_TYPE_SSL_CN
0x22 +#define PP2_TYPE_NETWORK_NAMESPACE 0x30
Could you please avoid to fragment the values here, I fear the
day we need a contiguous array and we can't find one anymore.
Let's start at 0x23 for now. Note, since we're in development,
we can change these numbers later, so let's not consider the
PPv2 spec as official whenever it comes to what we stuff in
haproxy-dev.
Can I reserve three spots for SSL_SNI, SSL_NPN, and SSL_ALPN? We
can try to keep the SSL block contiguous for now.
I do not think three spots is right for those. Two would suffice for
ALPN and SNI values and I requested 0x01 (for ALPN) and 0x02 (for SNI
authority-URI) earlier. Although I am flexible on the exact values.
NPN is not necessary since ALPN is what got standardized in RFC7301
and contains the NPN value set. PPv2 senders should upgrade NPN
values to ALPN for transmission by just sending them in the ALPN TLV.
Also, please do not irrevocably link ALPN and SNI with SSL/TLS in the
macro name. They are also used by other transport and transfer
protocols. Names like these should suffice:
PP2_TYPE_ALPN
PP2_TYPE_ORIGIN or PP2_TYPE_AUTHORITY_URI
Amos
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.22 (MingW32)
iQEcBAEBAgAGBQJUDtjfAAoJELJo5wb/XPRjE2wH/2+9cnzyyf9Kqbua16cqqPGE
vK8wrYOygzFcO9dGcKWnblUHsGBUOf80WcnIUg7YhhmH3r5afPcoRP7S5uzKtsCA
AY7XUMEZJKwZKR9UhVmgdZIhANZhLw134NnRYBzcPHZdLRVcBR0/4xK5UKvhVIIY
DD4R6CHB5latQ8ODq06lH7eyGoRSep2g3bTARaB6qEFZ+VuxZxbmzu1bOIfoMlPk
57PWUxgVJu2K+9d8laZ34CCUZ+9y3ZnZjbHES2kKsgNIhhEG0JUQ+WsmyCYlCvni
dBc8ZC1nRcky7SYw9RHiQgYiq18alK4tRtVeVpdujiMMhZF79LuE7rFxzs+6Yqw=
=XRC0
-END PGP SIGNATURE-
RFE: Allow logging directly to files
Hi guys, Long story short: We have a need for a new feature for haproxy, which allows logging to normal files (as opposed to the current domain socket/UDP servers). This will of course require adding such an option to the configuration. I am willing to write the whole feature (with reviews from the community, of course). I understood from the README at your github that it might be best to ask here first: is there a specific reason why this wasn't implemented yet? Long Story (and an introduction): I normally work on the OpenStack project (an open source cloud project), and specifically on the networking component called Neutron. A simplified setup in OpenStack looks like one or more machines, and each machine may run a number of VMs and in addition several agents (some of which deal with networking, some of which deal with storage, etc). The main agent which is interesting to this discussion is an LBaaS-Agent (Load Balancing as a Service), which serves mainly as an engine to run one or more load balancing services - for example haproxy. Each service may handle more than one haproxy process. The problem I'm trying to solve: If there are N haproxy processes in one machine (N = 2), and say the setup isn't stable (so VMs constantly goes up and down), many log messages will be written to syslog and it will become ungainly and complicated to differentiate which log message belongs to which haproxy. I spent a few days recently to solve this problem by assigning, for each haproxy process, a separate domain socket, and having LBaaS-Agent read from said sockets. Each time a new log is read, the message is written to a specific (different) file. In other words, say I have N haproxy processes; these are connected to N domain sockets, and the logs are read by a single LBaaS-Agent and finally written to N log files. This is far from optimal (and difficult to maintain because of the long route each log file is now making, and because there is now a single point of failure). Adding the ability for each haproxy to dump its logs to a specific file will solve the long workaround we made in the code (which is currently being reviewed and can be found at [1]). As you might read this, you might exclaim a simple change in the syslog configuration will tell it to redirect the logs to a specific file (see [2]). However, we have two main issues with this implementation: 1) All the logs will be written to one file, not solving to sorting problem. The difficulty of manually sorting each log to a different haproxy is increased exponentially with the number of haproxy processes. 2) This requires changes in the deployment tool, since neutron shouldn't touch syslog.conf. This isn't as simple as it sound since there are quite a few deployment tools for OpenStack, and if someone isn't using a deployment tool for OpenStack, this feature shouldn't be available. In conclusion, I would appreciate any feedback on the proposed patch, and the reasons it hasn't been implemented yet. If you see other alternatives, these will be welcome as well (we are open to discuss any proposal). [1] https://review.openstack.org/#/c/119731/ [2] http://marc.info/?l=haproxym=138643818630142w=2 Thanks a lot for reading this far, -- John Schwarz, Software Engineer, Red Hat.
Re: Recommended SSL ciphers and settings
rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains if ssl-proxy Do I need to add it to the frontend or backend? Will it break raw TLS (not HTTPS)? Thanks On Tue, Sep 9, 2014 at 1:25 PM, Thomas Heil h...@terminal-consulting.de wrote: Hi, On 09.09.2014 11:43, pablo platt wrote: I've tried both options and I'm still not getting A+. Unfortunately, I can't ask the user what the error is. If I'll run into this again, I'll try to get this info. To reach A+ you need rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains if ssl-proxy ssl-proxy means here the connection is ssl. and a cipher list like -- EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384: EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 -- Together it should work. As you can see we have no longer RC4 ciphers, cheers thomas Thanks On Mon, Sep 8, 2014 at 9:46 AM, Jarno Huuskonen jarno.huusko...@uef.fi wrote: Hi, On Sun, Sep 07, pablo platt wrote: Hi, I'm using haproxy to terminate SSL and it works for most of my users. I have alphassl wildcard certificate. I'm using SSL to improve WebSockets and RTMP connections of port 443. I don't have sensitive data or e-commerce. I have one user that see a warning in Chrome and can't use my website. Do you know what warning chrome gives to that user ? Is it possible that this the warning is because an antivirus is not happy with the default ciphers or other ssl settings? When running a test https://sslcheck.globalsign.com/en_US I'm getting: Sessions may be vulnerable to BEAST attack Server has not enabled HTTP Strict-Transport-Security Server has SSL v3 enabled Server is using RC4-based ciphersuites which have known vulnerabilities Server configuration does not meet FIPS guidelines Server does not have OCSP stapling configured Server has not yet upgraded to a Extended Validation certificate Server does not have SPDY enabled I found one suggestion: bind 10.0.0.9:443 name https ssl crt /path/to/domain.pem ciphers RC4:HIGH:!aNULL:!MD5 http://blog.haproxy.com/2013/01/21/mitigating-the-ssl-beast-attack-using-the-aloha-load-balancer-haproxy/ And another: bind 0.0.0.0:443 ssl crt /etc/cert.pem nosslv3 prefer-server-ciphers ciphers RC4-SHA:AES128-SHA:AES256-SHA Both gives me other warnings. What other warnings ? (Does haproxy give you warnings/errors or client browsers) ? Perhaps you could try ciphersuite from: https://wiki.mozilla.org/Security/Server_Side_TLS for example in global: ssl-default-bind-ciphers ... or on bind: bind 0.0.0.0:443 ssl crt /path/to/crt ciphers ... To enable ocsp stapling see haproxy config: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#5.1-crt http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2-set%20ssl%20ocsp-response -Jarno -- Jarno Huuskonen -- Thomas Heil - Email: h...@terminal-consulting.de Tel: 0176 / 44555622 --
Re: RFE: Allow logging directly to files
Hi John, On Tue, Sep 09, 2014 at 02:48:46PM +0300, John Schwarz wrote: Hi guys, Long story short: We have a need for a new feature for haproxy, which allows logging to normal files (as opposed to the current domain socket/UDP servers). This will of course require adding such an option to the configuration. I am willing to write the whole feature (with reviews from the community, of course). I understood from the README at your github that it might be best to ask here first: is there a specific reason why this wasn't implemented yet? Yes, haproxy works in multiplexed-I/O mode around a single poll() loop. You must absolutely not use any blocking I/O there, or you'll pause your process for a long time. That's the first reason why haproxy does no file system access once started. The second reason is that it's supposed to be chrooted and to drop privileges for increased security. However if the file is opened before the fork (like is done for logs over unix sockets), it will not be an issue. Still the performance issue remains. If you have, say, 4ms access time to a file because the machine is running under moderate I/O load, the service will become extremely slow as nothing will be done during these I/O. One option could be to implement async I/O, but that's not portable at all, so probably not really an option. Another option would be to dedicate a process to logs, but that would basically mean that we send logs overthere using a unix socket and it dumps them to disk. That's exactly what a syslog deamon does. (...) The problem I'm trying to solve: If there are N haproxy processes in one machine (N = 2), and say the setup isn't stable (so VMs constantly goes up and down), many log messages will be written to syslog and it will become ungainly and complicated to differentiate which log message belongs to which haproxy. I spent a few days recently to solve this problem by assigning, for each haproxy process, a separate domain socket, and having LBaaS-Agent read from said sockets. Each time a new log is read, the message is written to a specific (different) file. In other words, say I have N haproxy processes; these are connected to N domain sockets, and the logs are read by a single LBaaS-Agent and finally written to N log files. I know people who use many haproxy processes (once per application customer, with many applications running on a server), they have a dedicated syslog daemon, once per application user as well. And in the end it provides really good results, because each application comes with its haproxy and syslogd configs in a very templatable way. The unix socket is always the same, at a place relative to the config and logs, always with the same name, or just running over UDP with a port directly related to the application's or to the customer ID. Also, the benefits of using a regular syslog is that you can continue to use all the syslog facilities such as logs rotation, filtering/alerting/buffering etc... depending on the syslog daemon. This is far from optimal (and difficult to maintain because of the long route each log file is now making, and because there is now a single point of failure). Adding the ability for each haproxy to dump its logs to a specific file will solve the long workaround we made in the code (which is currently being reviewed and can be found at [1]). The method above provides all that without the difficult processing, and ensures that people skilled with each component can still apply their skills to each part of the log chain, which is another important point in general. Don't you think it would solve your trouble ? Regards, Willy
Re: Session stickiness on multi-process haproxy with ssl
On Tue, Sep 9, 2014 at 4:01 PM, e...@cslab.ece.ntua.gr wrote: Hello, I have HAproxy 1.5.4 installed in Debian Wheezy x64. My configuration file is attached. I want session stickiness so i use appsession attribute but I have a serious performance issue with ssl. Initially I didn't use nbproc parameter and haproxy could only serve 50reqs/sec with 100% cpu using only one core in a 8-core virtual machine. This is very low performance for my expectations, so I considered using nbproc=8 but then, as I have read, I can't have correct session stickiness. Is it expected that haproxy has initially (with 1 process) so low performance with ssl? Do I necessarily have to choose between performance and stickiness in my case, because I can't give up on either. Is there an alternative for session stickiness in multi-process haproxy? Kind regards, Evie Hi Evie, how big is your SSL key size??? What type of web application are you load-balancing and what type of clients have access to your application? Can you explain us the reason of the cipher you forced? (ssl-default-bind-ciphers) Also, you're using httpclose mode, maybe using http-keep-alive' would help a bit. can you check if your conntrack table is full? (using dmesg) you can also use log-format and log TLS version, negociated cipher and SSL session ID. If SSL session ID change all the time for a single user, it means you're not resuming SSL session and spend your time computing keys. Baptiste
Performance issue with SSL and keep alive, weird result of ab test
Hi all, I'm just doing some performance test on a ha-proxy 1.5.4 and 'ab' on the client side: * http = OK * https = OK * https + Keep-Alive = NOT OK (really bad performance) Can someone explain this result to me. What did I miss? Here the relevant config: -8 #- # Global settings #- global log /dev/log local0 maxconn 8192 user haproxy group haproxy stats socket/var/run/haproxy/haproxy.sock level admin tune.ssl.default-dh-param 4096 #- # Default settings #- defaults balance roundrobin log global modehttp option httplog option dontlognull retries 3 option redispatch option http-server-close timeout http-request 5s timeout connect 5s timeout server 10s timeout client 60s frontend fe_ssl_static bind XX:80 bind XX:443 ssl crt /etc/pki/tls/certs/certificates/multidomain/ssl.pem/server.pem ciphers ECDHE+aRSA+AES256+GCM+SHA384:ECDHE+aRSA+AES128+GCM+SHA256:ECDHE+aRSA+AES256+SHA384:ECDHE+aRSA+AES128+SHA256:ECDHE+aRSA+RC4+SHA:ECDHE+aRSA+AES256+SHA:ECDHE+aRSA+AES128+SHA:AES256+GCM+SHA384:AES128+GCM+SHA256:AES128+SHA256:AES256+SHA256:DHE+aRSA+AES128+SHA:RC4+SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS default_backend be_bl backend be_bl option httpchk GET /test.html http-check expect string okay option forwardfor acl ssl ssl_fc reqidel ^X-Forwarded-Proto:.* reqadd X-Forwarded-Proto:\ https if ssl reqadd X-Forwarded-Proto:\ http unless ssl server server01 YY:80 check maxconn 1000 weight 100 server server02 YY:80 check maxconn 1000 weight 100 -8 HA-Proxy is connected via 100MBit. The following tests were done: Via http: -8 $ ab -c 30 -n 3000 http://my.domain.de/jquery.js This is ApacheBench, Version 2.3 $Revision: 655654 $ Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking my.domain.de (be patient) Completed 300 requests Completed 600 requests Completed 900 requests Completed 1200 requests Completed 1500 requests Completed 1800 requests Completed 2100 requests Completed 2400 requests Completed 2700 requests Completed 3000 requests Finished 3000 requests Server Software:lighttpd Server Hostname:my.domain.de Server Port:80 Document Path: /jquery.js Document Length:93068 bytes Concurrency Level: 30 Time taken for tests: 26.286 seconds Complete requests: 3000 Failed requests:0 Write errors: 0 Total transferred: 280415082 bytes HTML transferred: 279437600 bytes Requests per second:114.13 [#/sec] (mean) Time per request: 262.859 [ms] (mean) Time per request: 8.762 [ms] (mean, across all concurrent requests) Transfer rate: 10417.87 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:3 49 148.0 271046 Processing:60 213 78.31931439 Waiting:6 29 11.1 28 257 Total: 74 262 168.42221642 Percentage of the requests served within a certain time (ms) 50%222 66%240 75%257 80%267 90%308 95%490 98% 1195 99% 1244 100% 1642 (longest request) -8 vai https -8 $ ab -c 30 -n 3000 https://my.domain.de/jquery.js This is ApacheBench, Version 2.3 $Revision: 655654 $ Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking my.domain.de (be patient) Completed 300 requests Completed 600 requests SSL read failed - closing connection Completed 900 requests Completed 1200 requests Completed 1500 requests Completed 1800 requests Completed 2100 requests Completed 2400 requests Completed 2700 requests Completed 3000 requests Finished 3000 requests Server Software:lighttpd Server Hostname:my.domain.de Server Port:443 SSL/TLS Protocol: TLSv1/SSLv3,ECDHE-RSA-AES128-SHA256,2048,128 Document Path: /jquery.js Document Length:93068 bytes Concurrency Level: 30 Time taken for tests: 33.296 seconds Complete requests: 3000 Failed requests:0 Write errors: 0 Total transferred: 280180493 bytes HTML transferred:
prefer epoll or sepoll on linux?
Hello, I'm running linux Mint 17 qiana. $ uname -a Linux cingarfield-dt 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The packaged haproxy for this distro is version 1.4.24: $ /usr/sbin/haproxy -vv HA-Proxy version 1.4.24 2013/06/17 Copyright 2000-2013 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 OPTIONS = USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. However I would like to use SSL so I built 1.5.3 from source: ~/dev/haproxy/haproxy-1.5.3$ ./haproxy -vv HA-Proxy version 1.5.3 2014/07/25 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014 Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.31 2012-07-06 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. My question is why does the packaged 1.4.24 use sepoll but from source 1.5.3 use epoll? Is there a meaningful difference, eg, in terms of performance? Thank you, Colin Ingarfield
[ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
Hi all, I did the ab test with concurrency = 1 and keep-alive. I found the following log entries written by HAProxy. Sep 9 16:54:20 server haproxy[29183]: :60646 [09/Sep/2014:16:54:20.014] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 93412 - - 10/9/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:25 server haproxy[29183]: :60649 [09/Sep/2014:16:54:25.051] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 93412 - - 12/9/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:30 server haproxy[29183]: :60653 [09/Sep/2014:16:54:30.089] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 93412 - - 14/9/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:35 server haproxy[29183]: :60656 [09/Sep/2014:16:54:35.127] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 93413 - - 11/7/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:40 server haproxy[29183]: :60660 [09/Sep/2014:16:54:40.167] fe_ssl_static~ be_bl/server02 18/0/1/0/28 200 93412 - - 12/4/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:45 server haproxy[29183]: :60663 [09/Sep/2014:16:54:45.205] fe_ssl_static~ be_bl/server01 19/0/0/1/31 200 93413 - - 40/5/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:50 server haproxy[29183]: :60667 [09/Sep/2014:16:54:50.244] fe_ssl_static~ be_bl/server01 19/0/0/1/29 200 93413 - - 29/21/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:55 server haproxy[29183]: :60670 [09/Sep/2014:16:54:55.282] fe_ssl_static~ be_bl/server02 20/0/1/0/34 200 93412 - - 10/3/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:00 server haproxy[29183]: :60675 [09/Sep/2014:16:55:00.324] fe_ssl_static~ be_bl/server02 20/0/0/1/30 200 93412 - - 10/5/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:05 server haproxy[29183]: :60678 [09/Sep/2014:16:55:05.363] fe_ssl_static~ be_bl/server01 20/0/0/1/30 200 93413 - - 21/15/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:10 server haproxy[29183]: :60682 [09/Sep/2014:16:55:10.402] fe_ssl_static~ be_bl/server02 21/0/0/0/30 200 93412 - - 33/22/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:15 server haproxy[29183]: :60685 [09/Sep/2014:16:55:15.442] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 93412 - - 36/2/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:20 server haproxy[29183]: :60689 [09/Sep/2014:16:55:20.480] fe_ssl_static~ be_bl/server01 21/0/0/1/31 200 93413 - - 4/3/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:25 server haproxy[29183]: :60692 [09/Sep/2014:16:55:25.519] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 93413 - - 23/8/0/1/0 0/0 GET /jquery.js HTTP/1.0 Now I guess that the way keep-alive is done with the 'ab-tool' and HAProxy is somewhat incompatible. It seems that the timeout on HAProxy side triggers a new request. Can someone put light on this Keep-Alive behaviour? Best regards Andreas Mock -Ursprüngliche Nachricht- Von: Andreas Mock [mailto:andreas.m...@drumedar.de] Gesendet: Dienstag, 9. September 2014 16:34 An: haproxy Betreff: Performance issue with SSL and keep alive, weird result of ab test Hi all, I'm just doing some performance test on a ha-proxy 1.5.4 and 'ab' on the client side: * http = OK * https = OK * https + Keep-Alive = NOT OK (really bad performance) Can someone explain this result to me. What did I miss? Here the relevant config: -8 #- # Global settings #- global log /dev/log local0 maxconn 8192 user haproxy group haproxy stats socket/var/run/haproxy/haproxy.sock level admin tune.ssl.default-dh-param 4096 #- # Default settings #- defaults balance roundrobin log global modehttp option httplog option dontlognull retries 3 option redispatch option http-server-close timeout http-request 5s timeout connect 5s timeout server 10s timeout client 60s frontend fe_ssl_static bind XX:80 bind XX:443 ssl crt /etc/pki/tls/certs/certificates/multidomain/ssl.pem/server.pem ciphers ECDHE+aRSA+AES256+GCM+SHA384:ECDHE+aRSA+AES128+GCM+SHA256:E CDHE+aRSA+AES256+SHA384:ECDHE+aRSA+AES128+SHA256:ECDHE+aRSA+R C4+SHA:ECDHE+aRSA+AES256+SHA:ECDHE+aRSA+AES128+SHA:AES256+GC M+SHA384:AES128+GCM+SHA256:AES128+SHA256:AES256+SHA256:DHE+aR SA+AES128+SHA:RC4+SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PS K:!SRP:!DSS default_backend be_bl backend be_bl option httpchk GET /test.html http-check expect string okay option forwardfor acl ssl ssl_fc reqidel ^X-Forwarded-Proto:.* reqadd
Re: prefer epoll or sepoll on linux?
Hi Colin, On Tue, 09 Sep 2014, 17:00:37 +0200, Colin Ingarfield wrote: Hello, I'm running linux Mint 17 qiana. $ uname -a Linux cingarfield-dt 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The packaged haproxy for this distro is version 1.4.24: $ /usr/sbin/haproxy -vv HA-Proxy version 1.4.24 2013/06/17 [...] My question is why does the packaged 1.4.24 use sepoll but from source 1.5.3 use epoll? Is there a meaningful difference, eg, in terms of performance? from http://www.haproxy.org/news.html: Nov 22th, 2012 : Development 1.5-dev13 with Compression! 4. Polling o speculative polling was generalized to all pollers, and sepoll disappeared as it was superseded by epoll. The main reason for this important change is the way OpenSSL works and the fact that it can easily get stuck with some data in buffers with no I/O event to unblock them. So we needed to engage into this difficult change. I'd have preferred to delay it to 1.6 if I was offered the choice ! But in the end this is good because it's done and it improves both performance and reliability. Even select() and poll() are now fast. Thank you, Colin Ingarfield HTH, cheers. l8er manfred signature.asc Description: Digital signature
Re: Session stickiness on multi-process haproxy with ssl
On Tue, Sep 9, 2014 at 4:47 PM, e...@cslab.ece.ntua.gr wrote:
On Tue, Sep 9, 2014 at 4:01 PM, e...@cslab.ece.ntua.gr wrote:
Hello,
I have HAproxy 1.5.4 installed in Debian Wheezy x64. My configuration
file
is attached. I want session stickiness so i use appsession attribute but
I
have a serious performance issue with ssl. Initially I didn't use nbproc
parameter and haproxy could only serve 50reqs/sec with 100% cpu using
only
one core in a 8-core virtual machine. This is very low performance for
my
expectations, so I considered using nbproc=8 but then, as I have read, I
can't have correct session stickiness.
Is it expected that haproxy has initially (with 1 process) so low
performance with ssl?
Do I necessarily have to choose between performance and stickiness in my
case, because I can't give up on either. Is there an alternative for
session stickiness in multi-process haproxy?
Kind regards,
Evie
Hi Evie,
how big is your SSL key size???
My key is 2048-bit.
What type of web application are you load-balancing and what type of
clients have access to your application?
Apache2 webservers are used as backends that serve a django-based site
with user authentication.
Can you explain us the reason of the cipher you forced?
(ssl-default-bind-ciphers)
Also, you're using httpclose mode, maybe using http-keep-alive' would
help a bit.
I tested http-keep-alive and a simple cipher such as RC4-SHA suitable for
my key but saw no difference.
can you check if your conntrack table is full? (using dmesg)
you can also use log-format and log TLS version, negociated cipher and
SSL session ID.
If SSL session ID change all the time for a single user, it means
you're not resuming SSL session and spend your time computing keys.
How can I check if ssl session id changes? Can I override this with a
proxy config if it happens?
Thanks
Please keep the ML in Cc :)
You can use the log-format directive below, in your frontend, to log
SSL related informations:
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\
%CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\
{%sslv/%sslc/%[ssl_fc_sni]/%[ssl_fc_session_id]}\
%[capture.req.method]\ %[capture.req.hdr(0)]%[capture.req.uri]\
HTTP/1.1
then try to anonymize logs and post some lines in attachment.
Baptiste
Re: prefer epoll or sepoll on linux?
On 09/09/2014 10:12 AM, Manfred Hollstein wrote: Hi Colin, On Tue, 09 Sep 2014, 17:00:37 +0200, Colin Ingarfield wrote: Hello, I'm running linux Mint 17 qiana. $ uname -a Linux cingarfield-dt 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The packaged haproxy for this distro is version 1.4.24: $ /usr/sbin/haproxy -vv HA-Proxy version 1.4.24 2013/06/17 [...] My question is why does the packaged 1.4.24 use sepoll but from source 1.5.3 use epoll? Is there a meaningful difference, eg, in terms of performance? from http://www.haproxy.org/news.html: Nov 22th, 2012 : Development 1.5-dev13 with Compression! 4. Polling o speculative polling was generalized to all pollers, and sepoll disappeared as it was superseded by epoll. The main reason for this important change is the way OpenSSL works and the fact that it can easily get stuck with some data in buffers with no I/O event to unblock them. So we needed to engage into this difficult change. I'd have preferred to delay it to 1.6 if I was offered the choice ! But in the end this is good because it's done and it improves both performance and reliability. Even select() and poll() are now fast. Thank you, Colin Ingarfield HTH, cheers. l8er manfred This page did not turn up in my brief googling -- I should have tried harder. But it's exactly what I was looking for. Thanks!
SSL handshake failure
I do not think this is a problem with haproxy (running 1.5.4), but I'm hoping haproxy can help me debug it. When I get SSL handshake failure, can haproxy be configured to log debug messages about WHY it failed? We don't have any visibility into the client -- it's at a customer site in Japan, I'm in the US. There is another question, but it's on an unrelated product. I've got the latest version of Wireshark (1.12.0), configured with my certificate's private key for SSL decrypting. The problem is that Wireshark is telling me that there is something wrong with the TLSv1 frames (Ignored Unknown Record). I do not have decrypted responses, only decrypted requests, and I assume that is because of those TLSv1 problems. The question: Is wireshark buggy, or are those TLSv1 frames actually problematic? The program was compiled against openssl-0.9.8e-27.el5_10.1 and it's running on a system with openssl-0.9.8e-7.el5 installed -- the production systems don't have a compiler or dev libraries installed, and when I attempted to install them, yum wouldn't work. If I force haproxy to use sslv3, then wireshark can decrypt the packets properly (when checked with a browser), but then our testing tools can't connect to it. Thanks, Shawn
Re: HAProxy 1.5 incorrectly marks servers as DOWN
On 08/09/2014 10:30 πμ, Juho Mäkinen wrote:
On Thu, Sep 4, 2014 at 11:35 PM, Pavlos Parissis
pavlos.paris...@gmail.com mailto:pavlos.paris...@gmail.com wrote:
On 04/09/2014 08:55 πμ, Juho Mäkinen wrote:
I'm upgrading my old 1.4.18 haproxies to 1.5.4 and I have a mysterious
problem where haproxy marks some backend servers as being DOWN with a
message L4TOUT in 2000ms.
Are you sure that you haven't reached any sort of limits on your backend
servers? Number of open files and etc...
Quite sure because I can always use curl from the haproxy machine to the
backend machine and I get the response to the check command always
without any delays.
Are you sure that backend servers return a response with HTTP status 200
on healtchecks?
Yes. I also ran strace on a single haproxy process when the haproxy
marked multiple backends as being down. Here's an example output:
08:06:07.302582 connect(30, {sa_family=AF_INET, sin_port=htons(3500),
sin_addr=inet_addr(172.16.6.102)}, 16) = -1 EINPROGRESS (Operation now
in progress)
08:06:07.303024 recvfrom(30, 0x1305494, 16384, 0, 0, 0) = -1 EAGAIN
(Resource temporarily unavailable)
08:06:07.303097 getsockopt(30, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
08:06:07.303167 sendto(30, GET /check HTTP/1.0\r\n\r\n, 23,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 23
08:06:07.304522 recvfrom(30, HTTP/1.1 200 OK\r\nX-Powered-By:
Express\r\nAccess-Control-Allow-Origin:
*\r\nAccess-Control-Allow-Methods: GET, HEAD, POST, PUT, DELE...,
16384, 0, NULL, NULL) = 503
08:06:07.304603 setsockopt(30, SOL_SOCKET, SO_LINGER, {onoff=1,
linger=0}, 8) = 0
08:06:07.304666 close(30) = 0
So the server clearly sends an HTTP 200 OK response, in just 1.9 ms. I
analysed around 20 different checks via the strace to the same backend
(which is marked down by haproxy) and none of them was over one second.
Are sure that for the above respone was for a healtcheck that marked the
server down? It is quite difficult to find this, I have been in your
position and took me some time to find the actual problem.
Here's an example from haproxy logging what happens when the problem starts:
Sep 8 07:22:25 localhost haproxy[24282]: [08/Sep/2014:07:22:24.615]
https comet-getcampaigns/comet-172.16.2.97:3500 423/0/1/3/427 200 502 -
- 1577/1577/3/1/0 0/0 GET /mobile HTTP/1.1
Sep 8 07:22:25 localhost haproxy[24284]: [08/Sep/2014:07:22:24.280]
https~ comet-getcampaigns/comet-172.16.2.97:3500 771/0/2/346/1121 200
40370 - - 2769/2769/6/0/0 0/0 GET /mobile HTTP/1.1Sep 8 07:22:25
localhost haproxy[24284]: [08/Sep/2014:07:22:25.090] https~
comet-getcampaigns/comet-172.16.2.97:3500 379/0/2/-1/804 502 204 - -
SH-- 2733/2733/7/0/0 0/0 GET /mobile HTTP/1.1
Sep 8 07:22:25 localhost haproxy[24280]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 231ms, status: 2/3 UP.
Sep 8 07:22:25 localhost haproxy[24281]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 217ms, status: 2/3 UP.
Sep 8 07:22:25 localhost haproxy[24282]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 137ms, status: 2/3 UP.
Sep 8 07:22:25 localhost haproxy[24284]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Socket error,
info: Connection reset by peer, check duration: 393ms, status: 2/3
UP.Sep 8 07:22:25 localhost haproxy[24284]: [08/Sep/2014:07:22:25.661]
https comet-getcampaigns/comet-172.16.2.97:3500 305/0/1/-1/314 -1 0 - -
SD-- 2718/2718/5/0/0 0/0 GET /mobile HTTP/1.1
The above means that the processes received a TCP RST packet on the
opening socket towards the backend. Have you run tcpdump on haproxy to
see if you backends send TCP RST.
Do you have any kind of firewall(network or host based) between haproxy
and backend?
Sep 8 07:22:27 localhost haproxy[24278]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4
connection problem, info: Connection refused, check duration: 0ms,
status: 2/3 UP.
Sep 8 07:22:27 localhost haproxy[24279]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4
connection problem, info: Connection refused, check duration: 0ms,
status: 2/3 UP.
Sep 8 07:22:28 localhost haproxy[24280]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4
connection problem, info: Connection refused, check duration: 2ms,
status: 1/3 UP.
Sep 8 07:22:28 localhost haproxy[24284]: Health check for server
comet-getcampaigns/comet-172.16.2.97:3500 failed, reason: Layer4
connection problem, info: Connection refused, check duration: 1ms,
status: 1/3 UP.
Sep 8 07:22:28 localhost haproxy[24282]:
Re: HAProxy 1.5 incorrectly marks servers as DOWN
Thanks Pavlos for your help. Fortunately (and embarrassedly for me) the mistake was not anywhere near haproxy but instead my haproxy configure template system had a bug which mixed up the backend name and ip address. Because of this haproxy showed different names for those servers which were actually down and that threw me into way off when I investigated this issue, blinded by the actual problem which was always so near of my sight. :( haproxy shows the server name in the server log when it reports health check statuses. Example: Health check for server comet/comet-172.16.4.209:3500 succeeded, reason: Layer7 check passed, code: 200, info: OK, check duration: 2ms, status: 3/3 UP. This could be improved by also showing the actual ip and port in the log. Suggestion: Health check for server comet/comet-172.16.4.209:3500 (172.16.4.209:3500) succeeded, reason: Layer7 check passed, code: 200, info: OK, check duration: 2ms, status: 3/3 UP. As a side question: The documentation was a bit unclear. If I have nbproc 1 and I use the admin socket to turn servers administrative status down or up, do I need to do it to separated admin sockets per haproxy process, or can I just use one admin socket? You need a different socket. Each process can only be managed by a dedicated stats socket. There isn't any kind of aggregation where you issue a command to 1 stats socket and this command is pushed to all processes. Next release will address this kind of issues. Thank you, good to know! - Garo
Re: [ADDENDUM] Performance issue with SSL and keep alive, weird result of ab test
Hi Andreas, On Tue, Sep 09, 2014 at 03:05:36PM +, Andreas Mock wrote: Hi all, I did the ab test with concurrency = 1 and keep-alive. I found the following log entries written by HAProxy. Sep 9 16:54:20 server haproxy[29183]: :60646 [09/Sep/2014:16:54:20.014] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 93412 - - 10/9/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:25 server haproxy[29183]: :60649 [09/Sep/2014:16:54:25.051] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 93412 - - 12/9/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:30 server haproxy[29183]: :60653 [09/Sep/2014:16:54:30.089] fe_ssl_static~ be_bl/server02 19/0/0/1/29 200 93412 - - 14/9/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:35 server haproxy[29183]: :60656 [09/Sep/2014:16:54:35.127] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 93413 - - 11/7/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:40 server haproxy[29183]: :60660 [09/Sep/2014:16:54:40.167] fe_ssl_static~ be_bl/server02 18/0/1/0/28 200 93412 - - 12/4/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:45 server haproxy[29183]: :60663 [09/Sep/2014:16:54:45.205] fe_ssl_static~ be_bl/server01 19/0/0/1/31 200 93413 - - 40/5/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:50 server haproxy[29183]: :60667 [09/Sep/2014:16:54:50.244] fe_ssl_static~ be_bl/server01 19/0/0/1/29 200 93413 - - 29/21/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:54:55 server haproxy[29183]: :60670 [09/Sep/2014:16:54:55.282] fe_ssl_static~ be_bl/server02 20/0/1/0/34 200 93412 - - 10/3/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:00 server haproxy[29183]: :60675 [09/Sep/2014:16:55:00.324] fe_ssl_static~ be_bl/server02 20/0/0/1/30 200 93412 - - 10/5/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:05 server haproxy[29183]: :60678 [09/Sep/2014:16:55:05.363] fe_ssl_static~ be_bl/server01 20/0/0/1/30 200 93413 - - 21/15/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:10 server haproxy[29183]: :60682 [09/Sep/2014:16:55:10.402] fe_ssl_static~ be_bl/server02 21/0/0/0/30 200 93412 - - 33/22/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:15 server haproxy[29183]: :60685 [09/Sep/2014:16:55:15.442] fe_ssl_static~ be_bl/server02 20/0/1/0/30 200 93412 - - 36/2/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:20 server haproxy[29183]: :60689 [09/Sep/2014:16:55:20.480] fe_ssl_static~ be_bl/server01 21/0/0/1/31 200 93413 - - 4/3/0/1/0 0/0 GET /jquery.js HTTP/1.0 Sep 9 16:55:25 server haproxy[29183]: :60692 [09/Sep/2014:16:55:25.519] fe_ssl_static~ be_bl/server01 20/0/1/0/31 200 93413 - - 23/8/0/1/0 0/0 GET /jquery.js HTTP/1.0 Now I guess that the way keep-alive is done with the 'ab-tool' and HAProxy is somewhat incompatible. It seems that the timeout on HAProxy side triggers a new request. Can someone put light on this Keep-Alive behaviour? I suspect that the server returns a response incompatible with keep-alive (eg: no content-length nor transfer-encoding), or that it simply returns a transfer-encoded response that ab doesn't support. From this point, a network capture is needed I guess :-/ Just for the record, ab is also the tool I'm using the most for ssl and for keep-alive tests, so I'm sure that it is supposed to work sometimes :-) Willy
Re: Reset near end of connection for slow transfers
I wanted to follow up and mention that we figured out the problem. The nolinger option was producing this behavior. Disabling it seems to have completely eliminated the issue. On Mon, Sep 8, 2014 at 12:55 PM, Sparr spa...@gmail.com wrote: I've got haproxy and apache on separate VMs in the same environment. When I download slowly (naturally across the internet or artificially with curl --limit-rate) I often get a connection failure near the end of the transfer. Usually at 99%, rarely as early as 87% on especially slow connections. If I make my download request directly to the apache server then there are no failures.
