Thanks for the response pelle, The haproxy box is running at about 10-15% cpu.
Looking at the TCP Dump, it seems the client is doing 1 or more TCP_Retransmissions of the incoming request when there is a delay. Anyone know why this would happen? Here is the requested information: root@haproxy:~# sysctl net.core.somaxconn net.core.somaxconn = 128 root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_max_syn_backlog = 2048 I removed the tcp_tw_recycle, but this did not change anything Thank you, Andy On Thu, Oct 10, 2013 at 1:58 PM, Pär Åslund <psl...@gmail.com> wrote: > Hi Andy, > > Can't see anything wrong with your configuration. > How about the server running haproxy? anything about that regarding cpu > etc? > > I wouldn't be using net.ipv4.tcp_tw_recycle = 1 > tw_recycle is a bit dodgy and can give some unwanted side-effects. > tw_reuse = 1 should be sufficient. > > How does a tcpdump look? Since you get this issue on the stats get a dump > on that. > > What setting are there on the box regarding SYN backlog? > net.core.somaxconn & net.ipv4.tcp_max_syn_backlog > > Best regards, > pelle > > > On Thu, Oct 10, 2013 at 7:16 PM, Andy M. <ajm2...@gmail.com> wrote: > >> Hello, >> >> I am trying to configure an HAProxy And seem to be running into a problem >> where the HAProxy will spike. I have a high throughput server cluster, >> which will need to handle about 10-20k QPS. I believe that HAProxy should >> be able to handle that quite easily from what I have read. I am running >> HAProxy on Ubuntu 12.04LTS. I am running an a Xeon 1270, with 8GB of >> RAM(Which i believe is all overkill). I am doing a simple load balance >> between two or three servers(I have two server clusters, one with two >> servers, one with three). I am running 1.4.24. >> >> The reason I believe something is wrong with my haproxy is because I can >> refresh my webpage a few times, and every 4-5 times I will be get a spike >> around 800ms-3sec to resolve. This also happens when not hitting my >> backend server, and instead hitting the HAProxy stats page. None of my >> servers are anywhere close to using up their system resources(all below 20% >> in both memory and CPU). I don't see much in the syslog that seems wrong >> other than this message: >> >> Oct 10 16:48:28 haproxy rsyslogd-2177: imuxsock begins to drop message >> 3912 due to rate-limiting >> >> I believe that has to do with the logging not being able to keep up, >> rather than haproxy dropping packets. >> >> I was wondering if there were any suggestions for me to look into to try >> and fix my problem, or suggest how to configure a HAProxy correctly to be >> able to handle the load I need(preferably higher than the load I need, as >> this may increase over time). I am open to changing operating systems if >> need be as well. >> >> Here are my two config files: >> >> global >> log /dev/log local0 info >> log /dev/log local0 notice >> maxconn 50000 >> user y >> group y >> #debug >> >> defaults >> log global >> mode http >> option httplog >> option dontlognull >> option forwardfor >> retries 3 >> option redispatch >> option http-server-close >> maxconn 30000 >> contimeout 10000 >> clitimeout 50000 >> srvtimeout 50000 >> balance leastconn >> >> listen c_cluster 255.255.255.146:80 >> maxconn 30000 >> server c1 10.101.13.74:80 maxconn 10000 >> server c2 10.101.13.78:80 maxconn 10000 >> server c3 10.101.13.82:80 maxconn 10000 >> listen stats :8888 >> mode http >> stats enable >> stats hide-version >> stats uri / >> >> >> and >> >> # this config needs haproxy-1.1.28 or haproxy-1.2.1 >> global >> log /dev/log local0 info >> log /dev/log local0 notice >> maxconn 20000 >> user y >> group y >> #debug >> >> defaults >> log global >> mode http >> option httplog >> option dontlognull >> option forwardfor >> retries 3 >> option redispatch >> option http-server-close >> maxconn 20000 >> contimeout 10000 >> clitimeout 50000 >> srvtimeout 50000 >> balance leastconn >> >> listen a_cluster 255.255.255.151:80 >> maxconn 20000 >> server a1 10.101.13.68:80 maxconn 10000 >> server a2 10.101.13.66:80 maxconn 10000 >> listen stats :8888 >> mode http >> stats enable >> stats hide-version >> stats uri / >> >> >> My sysctl.conf has the following changes: >> >> fs.file-max = 1000000 >> net.ipv4.tcp_tw_reuse = 1 >> net.ipv4.tcp_tw_recycle = 1 >> >> Thank you, >> >> Andy >> > >
