Greetings,
On 03/15/2016 02:54 PM, Zachary Punches wrote:
Hello!
My name is Zack, and I have been in the middle of an on going HAProxy
issue that has me scratching my head.
Here is the setup:
Our setup is hosted by amazon, and our HAProxy (1.6.3) boxes are in
each region in 3 regions. We have 2 HAProxy boxes per region for a
total of 6 proxy boxes.
These boxes are routed information through route 53. Their entire job
is to forward data from one of our clients to our database backend. It
handles this absolutely fine, except between the hours of 7pm PST and
7am PST. During these hours, our route53 health checks time out thus
causing the traffic to switch to the other HAProxy box inside of the
same region.
During the other 12 hours of the day, we receive 0 alerts from our
health checks.
I have noticed that we get a series of SSL handshake failures (though
this happens throughout the entire day) that causes the server to hang
for a second, thus causing the health checks to fail. During the day
our SSL failures do not cause the server to hang long enough to go
fail the checks, they only fail at night. I have attached my HAProxy
config hoping that you guys have an answer for me. Lemme know if you
need any more info.
Before thinking about less obvious potential causes, the CPU of the
instance isn't close to getting capped out during the time in question?
Also, are the connection counts under 15,000 (otherwise I could see it
ending up with a timeout and trying again)?
- Chad
I have done a few tcpdump captures during the SSL handshake failures
(not at night during it failing, but during the day when it still gets
the SSL handshake failures, but doesn’t fail the health check) and it
seems there is a d/c and a reconnect during the handshake.
Here is my config, I will be running a tcpdump tonight to capture the
packets during the failure and will attach it if you guys need more info.
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 30000
user haproxy
group haproxy
daemon
ssl-default-bind-options no-sslv3 no-tls-tickets
tune.ssl.default-dh-param 2048
# turn on stats unix socket
# stats socket /var/lib/haproxy/stats`
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
retries 3
timeout http-request 5s
timeout queue 1m
timeout connect 31s
timeout client 31s
timeout server 31s
maxconn 15000
# Stats
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth $StatsUser:$StatsPass
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend shared_incoming
maxconn 15000
timeout http-request 5s
# Bind ports of incoming traffic
bind *:1025 accept-proxy # http
bind *:1026 accept-proxy ssl crt /path/to/default/ssl/cert.pem ssl
crt /path/to/cert/folder/ # https
bind *:1027 # Health checking port
acl gs_texthtml url_reg \/gstext\.html ## allow gs to do meta
tag verififcation
acl gs_user_agent hdr_sub(User-Agent) -i globalsign ## allow gs to
do meta tag verififcation
# Add headers
http-request set-header $Proxy-Header-Ip %[src]
http-request set-header $Proxy-Header-Proto http if !{ ssl_fc }
http-request set-header $Proxy-Header-Proto https if { ssl_fc }
# Route traffic based on domain
use_backend gs_verify if gs_texthtml or gs_user_agent ## allow
gs meta tag verification
use_backend
%[req.hdr(host),lower,map_dom(/path/to/map/file.map,unknown_domain)]
# Drop unrecognized traffic
default_backend unknown_domain
#---------------------------------------------------------------------
# Backends
#---------------------------------------------------------------------
backend server0 ## added to allow gs ssl meta tag verification
reqrep ^GET\ /.*\ (HTTP/.*) GET\ /GlobalSignVerification\ \1
server server0_http server0.domain.com:80/GlobalSignVerification/
backend server1
server server1_http server1.domain.net:80
backend server2
server server2_http server2.domain.net:80
backend server3
server server3_http server3.domain.net:80
backend server4
server server4_http server4.domain.net:80
backend server5
server server5_http server5.domain.net:80
backend server6
server server6_http server6.domain.net:80
backend server7
server server7_http server7.domain.net:80
backend server8
server server8_http server8.domain.net:80
backend server9
server server9_http server9.domain.net:80
backend unknown_domain
timeout connect 4s
timeout server 4s
errorfile 503 /etc/haproxy-shared/errors/404.html
