On 17.10.2014 03:47, Dennis Jacobfeuerborn wrote:
> Hi,
> I'm currently trying to determine the best configuration for haproxy but
> have issues with understanding how the cpu's are used.
>
> The system is a virtual machine configured with 8 cores. The global
> section of the configuration contains these cpu settings:
>
> nbproc 8
> cpu-map all 1-7
> cpu-map 1 0
>
> The idea here is that processes 2-8 are used for ssl-offloading and are
> pinned to the cores 1-7 and that the balancing happens in process 1
> which is pinned to core 0.
>
> What then follows is several instances of this configuration for
> different sites:
>
> listen front-offload-<domain>
> bind-process 2-7
> bind <ip-address>:443 ssl crt <cert> ciphers
> ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!eNULL:!NULL:!DH:!EDH:!AESGCM
> no-sslv3
> http-request set-header X-Forwarded-Proto https
> server clear abns@ssl-proxy-<domain> send-proxy
>
> frontend front-https-<domain>
> bind-process 1
> bind abns@ssl-proxy-<domain> accept-proxy
> default_backend back-<domain>
>
> frontend front-http-<domain>
> bind-process 1
> bind <ip-address>:80
> default_backend back-<domain>
>
> backend back-<domain>
> bind-process 1
> mode http
> balance roundrobin
> option httpchk GET /health HTTP/1.1\r\nHost:\ <domain>
> http-check expect string OK
>
> default-server inter 2s fall 3 rise 1
> server websvr1 10.0.0.10:80 check
> ...
>
> The idea here is that front-offload-X handles the ssl offloading and
> then hands the traffic over to front-https-X. Parallel to that traffic
> on port 80 arrives on front-http-X and then traffic from the last two
> frontends is sent to the backend.
> That way the offloading is handled by processes 2-7 and everything else
> by process 1 so that the statistics only need to be displayed for
> process 1 to get a picture if the traffic coming in for http and https
> and what is happening on the backend.
>
> Now from what I read on the net about haproxy I expected that the
> load-balancing isn't that cpu intensive and that most of the cpu usage
> should be generated by the ssl offloading but what I'm seeing is that
> the two cores 0 and 4 are at 20% idle while the rest is at 80-90% idle
> and that cpu core 4 shows 20% soft interrupt usage while all other cores
> show 0% soft interrupt usage. The statistics page also shows an idle
> time of only 20%.
>
> Can sombody explain why I'm seeing this usage pattern? What I expected
> was that core 0 would see some usage but that the other cores are
> actually seeing much more usage because they are doing the offload
> heavy-lifting.
>
> Is my thinking wrong here and if so can somebody explain how to make
> better sense of how haproxy uses the cpu's?
>
So I went ahead and tested Lukas recommendation and moved everything
over to a CentOS 7 System with a 3.10 kernel. Since the new VM only uses
4 cores I changed the mapping as follows:
nbproc 4
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3
The ssl offload frontend is now bound to process 2-4.
Running a benchmark shows the following CPU load in top:
%Cpu0 : 22,7 us, 70,1 sy, 0,0 ni, 7,2 id, 0,0 hi, 0,0 si
%Cpu1 : 33,0 us, 23,7 sy, 0,0 ni, 43,3 id, 0,0 hi, 0,0 si
%Cpu2 : 36,3 us, 20,9 sy, 0,0 ni, 7,7 id, 0,0 hi, 35,2 si
%Cpu3 : 0,0 us, 0,0 sy, 0,0 ni,100,0 id, 0,0 hi, 0,0 si
(I removed the wa and st columns to improve readability in this email)
As you can see two processes see pretty heavy usage with one of them
spending a lot of time in soft irqs. cpu1 has some work to do but cpu3
is completely idle.
So why am I seeing these numbers? Should I expect that the ssl
offloading is distributed evenly over cpu1-3 and cpu3 shouldn't be
completely idle or does haproxy use up one cpu and only utilize the next
when required? I'm not so much concerned with absolute performance but
with the ability to predict and plan utilization.
Regards,
Dennis
