Hey,
I'm currently attempting to replace our commercial Loadbalancer with SSL
termination with haproxy. I'm running it on FreeBSD 9.2 Stable.
We have thousands of requests per second and for a while everything runs
extremely smooth. No queues are running full, machine load is at 0.5,
haproxy is at 2-3% CPU per process.
Then after an undetermined amount of time one process after another
(we're running with nbproc) is locked up with 100% CPU and does not
recover. Eventually all listen queues are filled to the max and I have
to kill and restart haproxy.
All I know so far is that it happens more frequently in peak traffic
times. Also I have tried to use the option httpclose because I thought
it was related to keep-alive issues. When I have tried this, instead of
locking up at 100%, haproxy suddenly exits with a core dump.
Below you will find our current haproxy configuration as well as some
sysctls we have tuned. haproxy 1.5-dev24 was installed from the FreeBSD
ports and is using kqueue.
Does anybody have a suggestion to track the issue down? The system
itself does not seem to have any problem until haproxy locks up. Like I
said before, after that, the listen queues are flooded because haproxy
does not accept any more requests.
Any feedback appreciated.
~ John
Machine:
2x Hexacore Xeon
32GB RAM
2x Dual Port Intel i350 Cards
FreeBSD 9.2 Stable
OpenSSL 1.0.1
haproxy:
global
log /var/run/log local0 notice
maxconn 20000
daemon
nbproc 12
defaults
mode http
retries 3
option redispatch
maxconn 20000
contimeout 5000
clitimeout 50000
srvtimeout 50000
log global
option dontlog-normal
option tcplog
option abortonclose
option forwardfor
frontend tracking_http
mode http
bind 0.0.0.0:80
option http-server-close
timeout http-keep-alive 5000
default_backend web-farm
frontend tracking_https
mode http
bind 0.0.0.0:443 ssl crt /usr/local/etc/ssl/combined.pem ciphers
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-RC4-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES128-SHA:AES256-SHA256:AES256-SHA:RC4-SHA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!EDH
reqadd X-Forwarded-Proto:\ https
option http-server-close
timeout http-keep-alive 5000
default_backend web-farm
backend web-farm
mode http
balance roundrobin
server app_1 10.5.1.161:80 cookie app1inst1 check inter 2000
rise 2 fall 5
server app_2 10.5.1.169:80 cookie app1inst2 check inter 2000
rise 2 fall 5
server app_3 10.5.1.177:80 cookie app1inst1 check inter 2000
rise 2 fall 5
server app_4 10.5.1.185:80 cookie app1inst2 check inter 2000
rise 2 fall 5
listen stats :1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth xxx:xxx
FreeBSD Sysctls:
net.inet.tcp.delayed_ack=0
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.msl=5000
kern.ipc.nmbclusters=2039162
kern.ipc.nmbjumbop=1019581
net.inet.icmp.icmplim=1000
kern.ipc.somaxconn=4096
net.inet.tcp.experimental.initcwnd10=1
kern.ipc.soacceptqueue=1024
kern.maxfiles=100000
kern.maxfilesperproc=90000
hw.igb.rx_process_limit=4096
--
John-Paul Bader | Software Development
www.wooga.com
wooga GmbH | Saarbruecker Str. 38 | D-10405 Berlin
Sitz der Gesellschaft: Berlin; HRB 117846 B
Registergericht Berlin-Charlottenburg
Geschaeftsfuehrung: Jens Begemann, Philipp Moeser
