-----Original Message-----
From: Willy Tarreau
Sent: Friday, April 20, 2012 7:37 AM
To: Baptiste
Cc: Joel Svensson ; haproxy@formilux.org
Subject: Re: Layer4 session concurrency
On Fri, Apr 20, 2012 at 06:38:41AM +0200, Baptiste wrote:
On Thu, Apr 19, 2012 at 3:48 PM, Joel Svensson
<joel__svens...@hotmail.com> wrote:
> Hi!
>
> From the text below I can't figure out if HAProxy will handle more
> sessions
> (than ~20000/GB ram) in Layer4 mode?
>
> The session concurrency
> This factor is tied to the previous one. Generally, the session rate
> will
> drop when the number of concurrent sessions increases (except the epoll
> polling mechanism). The slower the servers, the higher the number of
> concurrent sessions for a same session rate. If a load balancer receives
> 10000 sessions per second and the servers respond in 100 ms, then the
> load
> balancer will have 1000 concurrent sessions. This number is limited by
> the
> amount of memory and the amount of file-descriptors the system can
> handle.
> With 8 kB buffers, HAProxy will need about 16 kB per session, which
> results
> in around 60000 sessions per GB of RAM. In practise, socket buffers in
> the
> system also need some memory and 20000 sessions per GB of RAM is more
> reasonable. Layer 4 load balancers generally announce millions of
> simultaneous sessions because they don't process any data so they don't
> need
> any buffer. Moreover, they are sometimes designed to be used in Direct
> Server Return mode, in which the load balancer only sees forward
> traffic,
> and which forces it to keep the sessions for a long time after their end
> to
> avoid cutting sessions before they are closed.
>
>
> Regards,
> Joel
>
Hi Joel,
I guess that by layer4 mode, you mean tcp mode.
Unfortunately, when Willy speaks about layer 4 load-baancers in the
text above, he speaks about LVS like load-balancers
Exactly. The difference is between LBs that process a stream and which
are proxy-based, and the ones which process packets and are basically
routers. In order to parse and modify a stream, you need some memory,
while you don't need this to route packets (beyond the routing queue).
L4 load balancers often store a session table which is a few hundreds
of bytes per session, as opposed to a few tens of kB of buffers for
proxies. However, L4 LBs have to deal with TIME_WAIT, which proxies
don't since it's done in the system, so in practice, the ratio is not
really tens-of-thousands to millions but rather tens-of-thousands to
hundreds-of-thousands when the connection rate are high.
and why in HAProxy
you can have "only" thousends of connections while LVS like LBs can
annouce millions...
So in haproxy, whatever the mode, tcp or http, you'll always have
thousends of connexions.
In fact it depends a lot on the configured memory and on the kernel
tuning. With todays 64-bit systems and cheap RAM, there's plenty of
margin. We had one user who reported 1 million established connections
in a bench, and several ones reported more than 300k in production. In
Linux, by default, processes are limited to 1 million FDs so you need
to patch the kernel or to run in multi-process mode for this. I assume
it's not that crazy to run several processes when you have to deal with
1 million concurrent connections :-)
Cheers,
Willy
Thanks for the info!
I think we will setup both a HAProxy and LVS and run some tests.
We want to handle at least 100 000 connections without using a lot of RAM on
VMware ESX. There will be a very low connection rate/second and very
long-lived connections.
/joel
