We are using a Layer 7 style health check against a tcp backend. It's supposedly HTTP, but for whatever reason I couldn't get haproxy http check to work. Anyway, our check "works", but only sometimes. Over the course of day with no traffic to the backends at all (just our layer 7 health check) we'll see many "downs"... but if I hammer the health check using curl to do the soap request, it always returns.
The healthcheck takes milliseconds to respond... but when haproxy "fails" the check it of course waits the full haproxy timeout period. I temporarily worked around the spurious "Downs" by increasing the time between checks (again, testing outside says we need almost no time between checks) to 4s from the default 2s and by changing falls from the default value to 5 times. That doesn't prevent the layer 7 check problems, just keeps haproxy from downing the backend (I figure the combination of delayed inter and number of falls). Is there a problem with health checks and haproxy? Again, using a machine gun approach on the health check service, we see no problems, but for whatever reason, occasionally (maybe 1 out 10, could be more), the haproxy tcp expect fails. Using tcpdump, seems we're getting the right return though. We're using haproxy 1.15 without pcre expressions. Also, both checks fail occasionally, the non-ssl one and the ssl one. We see more check failures on the non-ssl one. But both are pretty high. qsrv1 and qsrv2 are up. We just don't want things failing over to qsrv2 (backup) unless qsrv1 is really down... and right now it's saying qsrv1 is down a lot, and our other tests say that it never went down and is handling healtchecks without issue. Here's the original config (using default inter, etc): #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats # This disables sslv3 (POODLE bug) and removes weak ciphers # For the most secure #ssl-default-bind-options no-sslv3 no-tls-tickets force-tlsv12 # For the most secure #ssl-default-bind-ciphers AES128+EECDH:AES128+EDH ssl-default-bind-options no-sslv3 ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4 tune.ssl.default-dh-param 1024 #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option dontlognull option redispatch retries 3 timeout http-request 10s timeout connect 5000 timeout client 50000 timeout server 50000 maxconn 20000 # # Web stats (temporary?) # listen stats :8080 mode http stats enable stats hide-version stats uri / #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend http *:10080 mode tcp default_backend srv-http option tcplog frontend https mode tcp bind :10443 ssl crt /etc/haproxy-ssl/example-combined.pem option tcplog default_backend srv-https backend srv-http option tcp-check tcp-check send POST\ /services/hcheck\ HTTP/1.1\r\nUser-Agent:\ curl/7.37.0\r\nHost:\ qsrv1.example.com:10080\r\nAccept:\ */*\r\nContent-Length:\ 114\r\nContent-Type:\ text/xml;charset=UTF-8\r\n\r\n<soapenv:Envelope\ xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">\ \ \ <soapenv:Body\ /></soapenv:Envelope> tcp-check expect rstring HTTP/1.1\ 200\ OK.*>OK< server qsrv1 10.1.50.52:10080 check slowstart 5000 server qsrv2 10.1.50.150:10080 check slowstart 5000 backup backend srv-https option tcp-check tcp-check send POST\ /services/hcheck\ HTTP/1.1\r\nUser-Agent:\ curl/7.37.0\r\nHost:\ qsrv1.example.com:10080\r\nAccept:\ */*\r\nContent-Length:\ 114\r\nContent-Type:\ text/xml;charset=UTF-8\r\n\r\n<soapenv:Envelope\ xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">\ \ \ <soapenv:Body\ /></soapenv:Envelope> tcp-check expect rstring HTTP/1.1\ 200\ OK.*>OK< server qsrv1 10.1.50.52:10443 check slowstart 5000 ssl verify none server qsrv2 10.1.50.150:10443 check slowstart 5000 ssl verify none backup