Re: nbproc 1 vs >1 performance
On 15/04/2016 11:44 πμ, Willy Tarreau wrote: > Hi Christian, > > On Fri, Apr 15, 2016 at 11:26:18AM +0200, Christian Ruppert wrote: >> Just in case someone is interested in this setup: >> Don't put the two SSL binds into the frontend. Add a second listener for the >> two SSL binds and from there via send-proxy-v2 to the frontend. >> Why? Because having >1 processes in the frontend in combination with a >> backend with active checks is a bad idea. If you have for example 10 >> loadbalancer with 40 process each you'll basically DDoS yourself. > > Definitely. That's why the recommended setup is indeed to have two layers, > one for SSL and one for L7, possibly connected over unix or abstract sockets : > > listen ssl-offload > bind-process 2-40 > bind :443 ssl crt foo ... process 2 > ... > bind :443 ssl crt foo ... process 40 > server clear abns@clear send-proxy-v2 > > frontend clear > bind-process 1 > bind :80 ... # clear traffic > bind abns@clear accept-proxy > ... > default_backend application... > > etc. > But in case of HTTPs on the backend side, you can't use the above as you need CPU power on the backend. Cheers, Pavlos signature.asc Description: OpenPGP digital signature
Re: nbproc 1 vs >1 performance
On Fri, Apr 15, 2016 at 11:56:20AM +0200, Pavlos Parissis wrote: > nbproc > 1 introduces also 2 other __issues__ which has been discussed > several times in this ML: > > 1) the aggregation of statistics > 2) performing operations such as disable/enable/change weight. > In setups where the frontends/backends is unevenly distributed across > all process, this becomes more apparent as you have to track which > process manages which frontend. All of them are properly solved by having the backend in a single process. Even peers work fine this way :-) You just need to consider the front SSL processes as dumb off-loaders. Cheers, Willy
Re: nbproc 1 vs >1 performance
On 15/04/2016 11:26 πμ, Christian Ruppert wrote:
> On 2016-04-14 11:06, Christian Ruppert wrote:
>> Hi Willy,
>>
>> On 2016-04-14 10:17, Willy Tarreau wrote:
>>> On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
Le me put it this way:
frontend haproxy_test
bind-process 1-8
bind :12345 process 1
bind :12345 process 2
bind :12345 process 3
bind :12345 process 4
Leads to 8 processes, and the master process binds the socket 4
times (PID
16509):
>>> (...)
lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
lukas@ubuntuvm:~/haproxy-1.5$
>>>
>>> OK so it's netstat which gives a wrong report, I have the same here.
>>> I verified
>>> in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also
>>> shows all the
>>> process list :
>>>
>>> LISTEN 0 128 *:12345
>>> *:*
>>> users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
>>>
>>> LISTEN 0 128 *:12345
>>> *:*
>>> users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
>>>
>>> LISTEN 0 128 *:12345
>>> *:*
>>> users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
>>>
>>> LISTEN 0 128 *:12345
>>> *:*
>>> users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))
>>>
>>>
>>> A performance test also shows a fair distribution of the load :
>>>
>>> 25353 willy 20 0 21872 4216 1668 S 26 0.1 0:04.54 haproxy
>>> 25374 willy 20 0 7456 1080 S 25 0.0 0:02.26
>>> injectl464
>>> 25376 willy 20 0 7456 1080 S 25 0.0 0:02.27
>>> injectl464
>>> 25377 willy 20 0 7456 1080 S 25 0.0 0:02.26
>>> injectl464
>>> 25375 willy 20 0 7456 1080 S 24 0.0 0:02.26
>>> injectl464
>>> 25354 willy 20 0 21872 4168 1620 R 22 0.1 0:04.51 haproxy
>>> 25356 willy 20 0 21872 4216 1668 R 22 0.1 0:04.21 haproxy
>>> 25355 willy 20 0 21872 4168 1620 S 21 0.1 0:04.38 haproxy
>>>
>>> However, as you can see these sockets are still bound to all
>>> processes and
>>> that's not a good idea in the multi-queue mode.
>>>
>>> I have added a few debug lines in enable_listener() like this :
>>>
>>> $ git diff
>>> diff --git a/src/listener.c b/src/listener.c
>>> index 5abeb80..59c51a1 100644
>>> --- a/src/listener.c
>>> +++ b/src/listener.c
>>> @@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
>>> */
>>> void enable_listener(struct listener *listener)
>>> {
>>> + fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
>>> if (listener->state == LI_LISTEN) {
>>> if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
>>> listener->bind_conf->bind_proc &&
>>> @@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
>>> * want any fd event to reach it.
>>> */
>>> fd_stop_recv(listener->fd);
>>> + fddebug("%d: pausing fd %d\n", getpid(),
>>> listener->fd);
>>> listener->state = LI_PAUSED;
>>> }
>>> else if (listener->nbconn < listener->maxconn) {
>>>
>>> And we're seeing this upon startup for processes 25746..25755 :
>>>
>>> Thus as you can see that FDs are properly enabled and paused for the
>>> unavailable ones.
>>>
>>> willy@wtap:haproxy$ grep 4294967295 log | grep 25746
>>> 25746 write(4294967295, "25746: enabling fd 4\n", 21
>>> 25746 write(4294967295, "25746: enabling fd 5\n", 21
>>> 25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad
>>> file descriptor)
>>> 25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad
>>> file descriptor)
>>> 25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad
>>> file descriptor)
>>> 25746 write(4294967295, "25746: enabling fd 7\n", 21
>>> 25746 write(4294967295, "25746: pausing fd 7\n", 20
>>> willy@wtap:haproxy$ grep 4294967295 log | grep 25747
>>> 25747 write(4294967295, "25747: enabling fd 4\n", 21
>>> 25747 write(4294967295, "25747: pausing fd 4\n", 20
>>> 25747 write(
Re: nbproc 1 vs >1 performance
Hi Christian, On Fri, Apr 15, 2016 at 11:26:18AM +0200, Christian Ruppert wrote: > Just in case someone is interested in this setup: > Don't put the two SSL binds into the frontend. Add a second listener for the > two SSL binds and from there via send-proxy-v2 to the frontend. > Why? Because having >1 processes in the frontend in combination with a > backend with active checks is a bad idea. If you have for example 10 > loadbalancer with 40 process each you'll basically DDoS yourself. Definitely. That's why the recommended setup is indeed to have two layers, one for SSL and one for L7, possibly connected over unix or abstract sockets : listen ssl-offload bind-process 2-40 bind :443 ssl crt foo ... process 2 ... bind :443 ssl crt foo ... process 40 server clear abns@clear send-proxy-v2 frontend clear bind-process 1 bind :80 ... # clear traffic bind abns@clear accept-proxy ... default_backend application... etc. > We'll have > to wait for multi-threaded for the setup above or some kind of dedicated > check process that shares the information with all other processes. No, you really don't want to multiply your backends, it causes pain for checks, for monitoring, for maxconn etc. Prefer to have a single instance if that sustains the load. Willy
Re: nbproc 1 vs >1 performance
On 2016-04-14 11:06, Christian Ruppert wrote:
Hi Willy,
On 2016-04-14 10:17, Willy Tarreau wrote:
On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
Le me put it this way:
frontend haproxy_test
bind-process 1-8
bind :12345 process 1
bind :12345 process 2
bind :12345 process 3
bind :12345 process 4
Leads to 8 processes, and the master process binds the socket 4 times
(PID
16509):
(...)
lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
lukas@ubuntuvm:~/haproxy-1.5$
OK so it's netstat which gives a wrong report, I have the same here. I
verified
in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also
shows all the
process list :
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))
A performance test also shows a fair distribution of the load :
25353 willy 20 0 21872 4216 1668 S 26 0.1 0:04.54 haproxy
25374 willy 20 0 7456 1080 S 25 0.0 0:02.26
injectl464
25376 willy 20 0 7456 1080 S 25 0.0 0:02.27
injectl464
25377 willy 20 0 7456 1080 S 25 0.0 0:02.26
injectl464
25375 willy 20 0 7456 1080 S 24 0.0 0:02.26
injectl464
25354 willy 20 0 21872 4168 1620 R 22 0.1 0:04.51 haproxy
25356 willy 20 0 21872 4216 1668 R 22 0.1 0:04.21 haproxy
25355 willy 20 0 21872 4168 1620 S 21 0.1 0:04.38 haproxy
However, as you can see these sockets are still bound to all processes
and
that's not a good idea in the multi-queue mode.
I have added a few debug lines in enable_listener() like this :
$ git diff
diff --git a/src/listener.c b/src/listener.c
index 5abeb80..59c51a1 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
*/
void enable_listener(struct listener *listener)
{
+ fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
if (listener->state == LI_LISTEN) {
if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
listener->bind_conf->bind_proc &&
@@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
* want any fd event to reach it.
*/
fd_stop_recv(listener->fd);
+ fddebug("%d: pausing fd %d\n", getpid(),
listener->fd);
listener->state = LI_PAUSED;
}
else if (listener->nbconn < listener->maxconn) {
And we're seeing this upon startup for processes 25746..25755 :
Thus as you can see that FDs are properly enabled and paused for the
unavailable ones.
willy@wtap:haproxy$ grep 4294967295 log | grep 25746
25746 write(4294967295, "25746: enabling fd 4\n", 21
25746 write(4294967295, "25746: enabling fd 5\n", 21
25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 7\n", 21
25746 write(4294967295, "25746: pausing fd 7\n", 20
willy@wtap:haproxy$ grep 4294967295 log | grep 25747
25747 write(4294967295, "25747: enabling fd 4\n", 21
25747 write(4294967295, "25747: pausing fd 4\n", 20
25747 write(4294967295, "25747: enabling fd 5\n", 21
25747 write(4294967295, "25747: enabling fd 6\n", 21
25747 write(4294967295, "25747: pausing fd 6\n", 20
25747 write(4294967295, "25747: enabling fd 7\n", 21
25747 write(4294967295, "25747: pausing fd 7\n", 20
willy@wtap:haproxy$ grep 4294967295 log | grep 25748
25748 write(4294967295, "25748: enabling fd 4\n", 21
25748 write(4294967295, "25748: pausing fd 4\n", 20
25748 write(4294967295, "25748: enabling fd 5\n", 21
25748 write(4294967295, "25748:
Re: nbproc 1 vs >1 performance
On Thu, Apr 14, 2016 at 01:54:27PM +0200, Daniel Schneller wrote: > Trying not to hijack the thread here, but it seems to fit well in the context: > > Does this mean that in the following could happen due to the difference in > BSD/Linux SO_REUSEPORT: > > 1. haproxy process ???A??? binds say port 1234 > 2. client A connects to 1234 and keeps the connection open > 3. /etc/init.d/haproxy restart > 4. haproxy process ???B??? starts and _also_ binds 1234 > 5. haproxy ???A??? is still around, due to client A > 6. client B connects to 1234 > > Am I right to assume that client B can be handled by _either_ haproxy ???A??? > or ???B??? depending on the hash result underneath SO_REUSEPORT???s > implementation? If so, that would explain some issues I had in the past when > quickly iterating config changes and restarting haproxy each time, but > sometimes getting results that could only have come from an older config? Not exactly. "haproxy restart" is "haproxy stop ; haproxy start" so it will stop the old process. However "haproxy reload" will do that yes. But the old process (process A) releases the port on reload, so new connections will only go to the new process. There's a tiny fraction of second during which both processes are bound to the same socket which ensures there's no connection lost, but it lasts one millisecond or so. Willy
Re: nbproc 1 vs >1 performance
Trying not to hijack the thread here, but it seems to fit well in the context: Does this mean that in the following could happen due to the difference in BSD/Linux SO_REUSEPORT: 1. haproxy process “A” binds say port 1234 2. client A connects to 1234 and keeps the connection open 3. /etc/init.d/haproxy restart 4. haproxy process “B” starts and _also_ binds 1234 5. haproxy “A” is still around, due to client A 6. client B connects to 1234 Am I right to assume that client B can be handled by _either_ haproxy “A” or “B” depending on the hash result underneath SO_REUSEPORT’s implementation? If so, that would explain some issues I had in the past when quickly iterating config changes and restarting haproxy each time, but sometimes getting results that could only have come from an older config? Thanks, Daniel -- Daniel Schneller Principal Cloud Engineer CenterDevice GmbH > On 14.04.2016, at 12:01, Willy Tarreau wrote: > > On Thu, Apr 14, 2016 at 10:17:10AM +0200, Willy Tarreau wrote: >> So I guess that indeed, if not all the processes a frontend is bound to >> have a corresponding bind line, this can cause connection issues as some >> incoming connections will be distributed to queues that nobody listens to. > > I said rubish here. It's the same socket that is shared between all these > processes, so there's no issue with all of them picking from the same queue > even if in the end only one picks it. However I want to fix it to make things > cleaner and easier to debug and observe (and not fool netstat as the previous > example showed). > > Willy >
Re: nbproc 1 vs >1 performance
On Thu, Apr 14, 2016 at 10:17:10AM +0200, Willy Tarreau wrote: > So I guess that indeed, if not all the processes a frontend is bound to > have a corresponding bind line, this can cause connection issues as some > incoming connections will be distributed to queues that nobody listens to. I said rubish here. It's the same socket that is shared between all these processes, so there's no issue with all of them picking from the same queue even if in the end only one picks it. However I want to fix it to make things cleaner and easier to debug and observe (and not fool netstat as the previous example showed). Willy
Re: nbproc 1 vs >1 performance
On Thu, Apr 14, 2016 at 11:49:47AM +0200, Christian Ruppert wrote:
> Yep, that did it. With this setting there is no more performance decrease on
> the http bind. Thanks!
> I'm just not sure if that will (negatively) affect anything else.
It may, depending on your setup (eg: if some frontends can drain many
connections at once it can increase latency).
Better apply the attached patch instead which divides by the number of
processes the listener is bound to :-)
Willy
>From 7c0ffd23d2d13e464a401292dcf55f658f2d3e24 Mon Sep 17 00:00:00 2001
From: Willy Tarreau
Date: Thu, 14 Apr 2016 11:47:38 +0200
Subject: BUG/MEDIUM: fix maxaccept computation on per-process listeners
Christian Ruppert reported a performance degradation when binding a
single frontend to many processes while only one bind line was being
used, bound to a single process.
The reason comes from the fact that whenever a listener is bound to
multiple processes, the it is assigned a maxaccept value which equals
half the global maxaccept value divided by the number of processes the
frontend is bound to. The purpose is to ensure that no single process
will drain all the incoming requests at once and ensure a fair share
between all listeners. Usually this works pretty well, when a listener
is bound to all the processes of its frontend. But here we're in a
situation where the maxaccept of a listener which is bound to a single
process is still divided by a large value.
The fix consists in taking into account the number of processes the
listener is bound do and not only those of the frontend. This way it
is perfectly possible to benefit from nbproc and SO_REUSEPORT without
performance degradation.
1.6 and 1.5 normally suffer from the same issue.
---
src/cfgparse.c | 14 ++
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 2d0a020..c3b29d4 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -8694,9 +8694,6 @@ out_uri_auth_compat:
for (curproxy = proxy; curproxy; curproxy = curproxy->next) {
struct listener *listener;
unsigned int next_id;
- int nbproc;
-
- nbproc = my_popcountl(curproxy->bind_proc &
nbits(global.nbproc));
#ifdef USE_OPENSSL
/* Configure SSL for each bind line.
@@ -8741,6 +8738,15 @@ out_uri_auth_compat:
/* adjust this proxy's listeners */
next_id = 1;
list_for_each_entry(listener, &curproxy->conf.listeners, by_fe)
{
+ int nbproc;
+
+ nbproc = my_popcountl(curproxy->bind_proc &
+ listener->bind_conf->bind_proc &
+ nbits(global.nbproc));
+
+ if (!nbproc) /* no intersection between listener and
frontend */
+ nbproc = 1;
+
if (!listener->luid) {
/* listener ID not set, use automatic numbering
with first
* spare entry starting with next_luid.
@@ -8819,7 +8825,7 @@ out_uri_auth_compat:
#endif /* USE_OPENSSL */
}
- if (nbproc > 1) {
+ if (my_popcountl(curproxy->bind_proc & nbits(global.nbproc)) >
1) {
if (curproxy->uri_auth) {
int count, maxproc = 0;
--
1.7.12.1
Re: nbproc 1 vs >1 performance
On 2016-04-14 11:41, Willy Tarreau wrote:
Hi Christian,
On Thu, Apr 14, 2016 at 11:06:02AM +0200, Christian Ruppert wrote:
I've applied your patch and I just looked at the performance so far.
The
performance is still the same, so the lessperformant one is still less
performant than the moreperformant.cfg. So from the performance point
of
view there's no difference between with and without that patch.
We have a (IMHO) quite huge haproxy config with around 200 frontends,
~180
backends and ~90 listener. So the intention was to combine a http bind
with
a ssl bind in one frontend so that we keep it at a minimum that is
necessary
to get it working properly:
listen ssl-relay
mode tcp
bind-process 2
bind :443 process 2
tcp-request inspect-delay 7s
acl HAS_ECC req.ssl_ec_ext eq 1
tcp-request content accept if { req_ssl_hello_type 1 } # Client
Hello
use-server ecc if HAS_ECC
server ecc unix@/var/run/haproxy_ssl_ecc.sock send-proxy-v2
use-server rsa if !HAS_ECC
server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2
frontend http_https_combined
mode http
bind-process 1-40
bind :80 process 1
bind unix@/var/run/haproxy_ssl_ecc.sock accept-proxy ssl crt
/etc/haproxy/ssltest.pem-ECC user haproxy process 4-40
bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt
/etc/haproxy/ssltest.pem-RSA user haproxy process 3
...
default_backend somebackend
OK I know why. It's because there's a per-listener "maxaccept" setting
which divides the global.tune.maxaccept value by 2 then by the number
of processes a listener is bound to. But in fact it divides by the
number
of processes the frontend is bound to. In the past this used to be
efficient
but since the "process" directive on frontends it doesn't make sense
anymore.
So in practice above you end up with "maxaccept 1". You could try to
work
around it by setting "tune.maxaccept 4000" in your global section,
you'll
get the original performance back.
Yep, that did it. With this setting there is no more performance
decrease on the http bind. Thanks!
I'm just not sure if that will (negatively) affect anything else.
I have to change this obviously, so that we consider the intersection
between the listener and the frontend and not the frontend only. I'm
looking at this right now.
Willy
--
Regards,
Christian Ruppert
Re: nbproc 1 vs >1 performance
Hi Christian,
On Thu, Apr 14, 2016 at 11:06:02AM +0200, Christian Ruppert wrote:
> I've applied your patch and I just looked at the performance so far. The
> performance is still the same, so the lessperformant one is still less
> performant than the moreperformant.cfg. So from the performance point of
> view there's no difference between with and without that patch.
>
> We have a (IMHO) quite huge haproxy config with around 200 frontends, ~180
> backends and ~90 listener. So the intention was to combine a http bind with
> a ssl bind in one frontend so that we keep it at a minimum that is necessary
> to get it working properly:
> listen ssl-relay
> mode tcp
>
> bind-process 2
>
> bind :443 process 2
>
> tcp-request inspect-delay 7s
> acl HAS_ECC req.ssl_ec_ext eq 1
> tcp-request content accept if { req_ssl_hello_type 1 } # Client Hello
>
> use-server ecc if HAS_ECC
> server ecc unix@/var/run/haproxy_ssl_ecc.sock send-proxy-v2
>
> use-server rsa if !HAS_ECC
> server rsa unix@/var/run/haproxy_ssl_rsa.sock send-proxy-v2
>
>
> frontend http_https_combined
> mode http
>
> bind-process 1-40
>
> bind :80 process 1
> bind unix@/var/run/haproxy_ssl_ecc.sock accept-proxy ssl crt
> /etc/haproxy/ssltest.pem-ECC user haproxy process 4-40
> bind unix@/var/run/haproxy_ssl_rsa.sock accept-proxy ssl crt
> /etc/haproxy/ssltest.pem-RSA user haproxy process 3
>
> ...
>
> default_backend somebackend
OK I know why. It's because there's a per-listener "maxaccept" setting
which divides the global.tune.maxaccept value by 2 then by the number
of processes a listener is bound to. But in fact it divides by the number
of processes the frontend is bound to. In the past this used to be efficient
but since the "process" directive on frontends it doesn't make sense anymore.
So in practice above you end up with "maxaccept 1". You could try to work
around it by setting "tune.maxaccept 4000" in your global section, you'll
get the original performance back.
I have to change this obviously, so that we consider the intersection
between the listener and the frontend and not the frontend only. I'm
looking at this right now.
Willy
Re: nbproc 1 vs >1 performance
Hi Willy,
On 2016-04-14 10:17, Willy Tarreau wrote:
On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
Le me put it this way:
frontend haproxy_test
bind-process 1-8
bind :12345 process 1
bind :12345 process 2
bind :12345 process 3
bind :12345 process 4
Leads to 8 processes, and the master process binds the socket 4 times
(PID
16509):
(...)
lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
tcp0 0 *:12345 *:* LISTEN
16509/haproxy
lukas@ubuntuvm:~/haproxy-1.5$
OK so it's netstat which gives a wrong report, I have the same here. I
verified
in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also
shows all the
process list :
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
LISTEN 0 128 *:12345
*:*
users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))
A performance test also shows a fair distribution of the load :
25353 willy 20 0 21872 4216 1668 S 26 0.1 0:04.54 haproxy
25374 willy 20 0 7456 1080 S 25 0.0 0:02.26
injectl464
25376 willy 20 0 7456 1080 S 25 0.0 0:02.27
injectl464
25377 willy 20 0 7456 1080 S 25 0.0 0:02.26
injectl464
25375 willy 20 0 7456 1080 S 24 0.0 0:02.26
injectl464
25354 willy 20 0 21872 4168 1620 R 22 0.1 0:04.51 haproxy
25356 willy 20 0 21872 4216 1668 R 22 0.1 0:04.21 haproxy
25355 willy 20 0 21872 4168 1620 S 21 0.1 0:04.38 haproxy
However, as you can see these sockets are still bound to all processes
and
that's not a good idea in the multi-queue mode.
I have added a few debug lines in enable_listener() like this :
$ git diff
diff --git a/src/listener.c b/src/listener.c
index 5abeb80..59c51a1 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
*/
void enable_listener(struct listener *listener)
{
+ fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
if (listener->state == LI_LISTEN) {
if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
listener->bind_conf->bind_proc &&
@@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
* want any fd event to reach it.
*/
fd_stop_recv(listener->fd);
+ fddebug("%d: pausing fd %d\n", getpid(),
listener->fd);
listener->state = LI_PAUSED;
}
else if (listener->nbconn < listener->maxconn) {
And we're seeing this upon startup for processes 25746..25755 :
Thus as you can see that FDs are properly enabled and paused for the
unavailable ones.
willy@wtap:haproxy$ grep 4294967295 log | grep 25746
25746 write(4294967295, "25746: enabling fd 4\n", 21
25746 write(4294967295, "25746: enabling fd 5\n", 21
25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad
file descriptor)
25746 write(4294967295, "25746: enabling fd 7\n", 21
25746 write(4294967295, "25746: pausing fd 7\n", 20
willy@wtap:haproxy$ grep 4294967295 log | grep 25747
25747 write(4294967295, "25747: enabling fd 4\n", 21
25747 write(4294967295, "25747: pausing fd 4\n", 20
25747 write(4294967295, "25747: enabling fd 5\n", 21
25747 write(4294967295, "25747: enabling fd 6\n", 21
25747 write(4294967295, "25747: pausing fd 6\n", 20
25747 write(4294967295, "25747: enabling fd 7\n", 21
25747 write(4294967295, "25747: pausing fd 7\n", 20
willy@wtap:haproxy$ grep 4294967295 log | grep 25748
25748 write(4294967295, "25748: enabling fd 4\n", 21
25748 write(4294967295, "25748: pausing fd 4\n", 20
25748 write(4294967295, "25748: enabling fd 5\n", 21
25748 write(4294967295, "25748: pausing fd 5\n", 20
25748 write(4294967295, "
Re: nbproc 1 vs >1 performance
On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote:
> Le me put it this way:
>
> frontend haproxy_test
> bind-process 1-8
> bind :12345 process 1
> bind :12345 process 2
> bind :12345 process 3
> bind :12345 process 4
>
>
> Leads to 8 processes, and the master process binds the socket 4 times (PID
> 16509):
>
(...)
> lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap
> tcp0 0 *:12345 *:* LISTEN 16509/haproxy
> tcp0 0 *:12345 *:* LISTEN 16509/haproxy
> tcp0 0 *:12345 *:* LISTEN 16509/haproxy
> tcp0 0 *:12345 *:* LISTEN 16509/haproxy
> lukas@ubuntuvm:~/haproxy-1.5$
OK so it's netstat which gives a wrong report, I have the same here. I verified
in /proc/$PID/fd/ and I properly saw the FDs. Next, "ss -anp" also shows all the
process list :
LISTEN 0 128 *:12345*:*
users:(("haproxy",25360,7),("haproxy",25359,7),("haproxy",25358,7),("haproxy",25357,7),("haproxy",25356,7),("haproxy",25355,7),("haproxy",25354,7),("haproxy",25353,7))
LISTEN 0 128 *:12345*:*
users:(("haproxy",25360,6),("haproxy",25359,6),("haproxy",25358,6),("haproxy",25357,6),("haproxy",25356,6),("haproxy",25355,6),("haproxy",25354,6),("haproxy",25353,6))
LISTEN 0 128 *:12345*:*
users:(("haproxy",25360,5),("haproxy",25359,5),("haproxy",25358,5),("haproxy",25357,5),("haproxy",25356,5),("haproxy",25355,5),("haproxy",25354,5),("haproxy",25353,5))
LISTEN 0 128 *:12345*:*
users:(("haproxy",25360,4),("haproxy",25359,4),("haproxy",25358,4),("haproxy",25357,4),("haproxy",25356,4),("haproxy",25355,4),("haproxy",25354,4),("haproxy",25353,4))
A performance test also shows a fair distribution of the load :
25353 willy 20 0 21872 4216 1668 S 26 0.1 0:04.54 haproxy
25374 willy 20 0 7456 1080 S 25 0.0 0:02.26 injectl464
25376 willy 20 0 7456 1080 S 25 0.0 0:02.27 injectl464
25377 willy 20 0 7456 1080 S 25 0.0 0:02.26 injectl464
25375 willy 20 0 7456 1080 S 24 0.0 0:02.26 injectl464
25354 willy 20 0 21872 4168 1620 R 22 0.1 0:04.51 haproxy
25356 willy 20 0 21872 4216 1668 R 22 0.1 0:04.21 haproxy
25355 willy 20 0 21872 4168 1620 S 21 0.1 0:04.38 haproxy
However, as you can see these sockets are still bound to all processes and
that's not a good idea in the multi-queue mode.
I have added a few debug lines in enable_listener() like this :
$ git diff
diff --git a/src/listener.c b/src/listener.c
index 5abeb80..59c51a1 100644
--- a/src/listener.c
+++ b/src/listener.c
@@ -49,6 +49,7 @@ static struct bind_kw_list bind_keywords = {
*/
void enable_listener(struct listener *listener)
{
+ fddebug("%d: enabling fd %d\n", getpid(), listener->fd);
if (listener->state == LI_LISTEN) {
if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) &&
listener->bind_conf->bind_proc &&
@@ -57,6 +58,7 @@ void enable_listener(struct listener *listener)
* want any fd event to reach it.
*/
fd_stop_recv(listener->fd);
+ fddebug("%d: pausing fd %d\n", getpid(), listener->fd);
listener->state = LI_PAUSED;
}
else if (listener->nbconn < listener->maxconn) {
And we're seeing this upon startup for processes 25746..25755 :
Thus as you can see that FDs are properly enabled and paused for the
unavailable ones.
willy@wtap:haproxy$ grep 4294967295 log | grep 25746
25746 write(4294967295, "25746: enabling fd 4\n", 21
25746 write(4294967295, "25746: enabling fd 5\n", 21
25746 write(4294967295, "25746: pausing fd 5\n", 20) = -1 EBADF (Bad file
descriptor)
25746 write(4294967295, "25746: enabling fd 6\n", 21) = -1 EBADF (Bad file
descriptor)
25746 write(4294967295, "25746: pausing fd 6\n", 20) = -1 EBADF (Bad file
descriptor)
25746 write(4294967295, "25746: enabling fd 7\n", 21
25746 write(4294967295, "25746: pausing fd 7\n", 20
willy@wtap:haproxy$ grep 4294967295 log | grep 25747
25747 write(4294967295, "25747: enabling fd 4\n", 21
25747 write(4294967295, "25747: pausing fd 4\n", 20
25747 write(4294967295, "25747: enabling fd 5\n", 21
25747 write(4294967295, "25747: enabling fd 6\n", 21
25747 write(4294967295, "25747: pausing fd 6\n", 20
25747 write(4294967295, "25747: enabling fd 7\n", 21
25747 write(4294967295, "25747: pausing fd 7\n", 20
willy@wtap:haproxy$ grep 4294967295 log | grep 25748
25748 write(4294967295, "25748: enabling fd 4\n", 21
25748 write(4294967295, "25748: pausing fd 4\n", 20
25748 write(4294967295, "25748: enabling fd 5\n", 21
25748 write(4294967295,
Re: nbproc 1 vs >1 performance
On Thu, Apr 14, 2016 at 08:55:47AM +0200, Lukas Tribus wrote: > Hi Willy, > > > Am 14.04.2016 um 07:08 schrieb Willy Tarreau: > >Hi Lukas, > > > >On Thu, Apr 14, 2016 at 12:14:15AM +0200, Lukas Tribus wrote: > >>For example, the following configuration load balances the traffic across > >>all 40 processes, expected or not? > >> > >>frontend haproxy_test > >> bind-process 1-40 > >> bind :12345 process 1 > > > >It's not expected. What is indicated above is that the frontend will > >exist for the first 40 processes, and that port 12345 will be bound > >only in process 1. Processes 2..40 will thus have no listener at all. > > Well, we do only bind once here, not 40 times. But the traffic is then > handled by all 40 processes. No here it's handled by only one due to "bind :12345 process 1". If there was not this "process 1", then it would indeed be handled by all 40 processes in a random fashion. > Le me put it this way: > > frontend haproxy_test > bind-process 1-8 > bind :12345 process 1 > bind :12345 process 2 > bind :12345 process 3 > bind :12345 process 4 > > > Leads to 8 processes, and the master process binds the socket 4 times (PID > 16509): > > lukas@ubuntuvm:~/haproxy-1.5$ ps auxww | grep haproxy > haproxy 16509 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16510 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16511 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16512 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16513 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16514 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16515 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > haproxy 16516 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy > -f ../cert/ruppert-nbproc-stress.cfg -D > lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap > tcp0 0 *:12345 *:* LISTEN 16509/haproxy > tcp0 0 *:12345 *:* LISTEN 16509/haproxy > tcp0 0 *:12345 *:* LISTEN 16509/haproxy > tcp0 0 *:12345 *:* LISTEN 16509/haproxy > lukas@ubuntuvm:~/haproxy-1.5$ Then this is where we have a problem. Only processes 1..4 should be bound. > >But I'm rather surprized, maybe we recently broke something because > >we have a lot of people successfully running such configurations, > >which is why I'm a bit surprized. > > I can reproduce this in v1.5.0 too, I think it was always like that. I'm still very surprized because what this means is that each time we managed to make a config work with such a setup it was by pure luck. The most common use-case is with SSL and I'm aware of a significant number of deployments running this way with the expected results. I'll have to run some local tests to understand what's going on. Thanks, Willy
Re: nbproc 1 vs >1 performance
Hi Willy, Am 14.04.2016 um 07:08 schrieb Willy Tarreau: Hi Lukas, On Thu, Apr 14, 2016 at 12:14:15AM +0200, Lukas Tribus wrote: For example, the following configuration load balances the traffic across all 40 processes, expected or not? frontend haproxy_test bind-process 1-40 bind :12345 process 1 It's not expected. What is indicated above is that the frontend will exist for the first 40 processes, and that port 12345 will be bound only in process 1. Processes 2..40 will thus have no listener at all. Well, we do only bind once here, not 40 times. But the traffic is then handled by all 40 processes. Le me put it this way: frontend haproxy_test bind-process 1-8 bind :12345 process 1 bind :12345 process 2 bind :12345 process 3 bind :12345 process 4 Leads to 8 processes, and the master process binds the socket 4 times (PID 16509): lukas@ubuntuvm:~/haproxy-1.5$ ps auxww | grep haproxy haproxy 16509 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16510 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16511 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16512 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16513 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16514 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16515 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D haproxy 16516 0.0 0.0 18460 320 ?Ss 08:41 0:00 ./haproxy -f ../cert/ruppert-nbproc-stress.cfg -D lukas@ubuntuvm:~/haproxy-1.5$ sudo netstat -tlp | grep hap tcp0 0 *:12345 *:* LISTEN 16509/haproxy tcp0 0 *:12345 *:* LISTEN 16509/haproxy tcp0 0 *:12345 *:* LISTEN 16509/haproxy tcp0 0 *:12345 *:* LISTEN 16509/haproxy lukas@ubuntuvm:~/haproxy-1.5$ SO_REUSEPORT is almost irrelevant here. In fact it *happens* to perform some load balancing, but the first use was simply to permit new processes to rebind before unbinding older ones during reloads. I know, haproxy unconditionally enables SO_REUSEPORT since 2006, because of OpenBSD reload/restarts (OpenBSD does not load-balance). But I'm not aware we ever changed anything after Linux introduced SO_REUSEPORT with load-balancing support in 3.9, so the fact that we bind everything to the master process is in my opinion simply based on the fact that SO_REUSEPORT when introduced was just a fix for reload/restart problems. But I'm rather surprized, maybe we recently broke something because we have a lot of people successfully running such configurations, which is why I'm a bit surprized. I can reproduce this in v1.5.0 too, I think it was always like that. cheers, lukas
Re: nbproc 1 vs >1 performance
Hi Lukas, On Thu, Apr 14, 2016 at 12:14:15AM +0200, Lukas Tribus wrote: > For example, the following configuration load balances the traffic across > all 40 processes, expected or not? > > frontend haproxy_test > bind-process 1-40 > bind :12345 process 1 It's not expected. What is indicated above is that the frontend will exist for the first 40 processes, and that port 12345 will be bound only in process 1. Processes 2..40 will thus have no listener at all. > The docs [1] are not very clear and kind of contradict themselves: > >does not enforce any process but eliminates those which do not match > >[...] > >If the frontend uses a "bind-process" setting, the intersection between > >the > >two is applied > > > We mention in multiple places (docs, commits, ml) that we use the kernel's > SO_REUSEPORT to loadbalance incoming traffic if supported, however due to > the behavior mentioned above, when we do something like this: > > frontend haproxy_test > bind :12345 process 1 > bind :12345 process 2 > bind :12345 process 3 > bind :12345 process 4 > > > all those sockets are bound to the master process, not its forks: > > lukas@ubuntuvm:~/haproxy$ ps auxww | grep haproxy > haproxy 14457 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy > -D -f ../cert/ruppert-nbproc-stress.cfg > haproxy 14458 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy > -D -f ../cert/ruppert-nbproc-stress.cfg > haproxy 14459 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy > -D -f ../cert/ruppert-nbproc-stress.cfg > haproxy 14460 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy > -D -f ../cert/ruppert-nbproc-stress.cfg > lukas@ubuntuvm:~/haproxy$ sudo netstat -tlp | grep 12345 > tcp0 0 *:12345 *:* LISTEN 14457/haproxy > tcp0 0 *:12345 *:* LISTEN 14457/haproxy > tcp0 0 *:12345 *:* LISTEN 14457/haproxy > tcp0 0 *:12345 *:* LISTEN 14457/haproxy > lukas@ubuntuvm:~/haproxy$ Then there is a bug here. > So I'm not sure how SO_REUSEPORT is supposed to load-balance between > processes, if all sockets point to the master process (it will just pass the > request around, like without SO_REUSEPORT). The docs [1] really suggest this > configuration for SO_REUSEPORT based kernel side load-balancing. SO_REUSEPORT is almost irrelevant here. In fact it *happens* to perform some load balancing, but the first use was simply to permit new processes to rebind before unbinding older ones during reloads. The way SO_REUSEPORT works in the kernel is that it hashes ip and ports and picks a socket queue based on the result of the hash. So any process which is bound may be picked. That makes me realize that this behaviour will definitely kill the principle of a master socket server by the way since it will not be possible to keep a socket alive without taking one's share of the traffic. > It should look like this, imho: > > lukas@ubuntuvm:~/haproxy$ sudo netstat -tlp | grep 12345 > tcp0 0 *:12345 *:* LISTEN 14713/haproxy > tcp0 0 *:12345 *:* LISTEN 14712/haproxy > tcp0 0 *:12345 *:* LISTEN 14711/haproxy > tcp0 0 *:12345 *:* LISTEN 14710/haproxy > lukas@ubuntuvm:~/haproxy$ Yes I agree. > But that will only work across frontends: > > frontend haproxy_test1 > bind-process 1 > bind :12345 > default_backend backend_test > frontend haproxy_test2 > bind-process 2 > bind :12345 > default_backend backend_test > frontend haproxy_test3 > bind-process 3 > bind :12345 > default_backend backend_test > frontend haproxy_test4 > bind-process 4 > bind :12345 > default_backend backend_test That's not normal. The principle of bind-process is that it limits the processes on which a frontend stays enabled (in practice, it stops the frontend after the fork on all processes where it's not bound). Then the "process" directive on the "bind" line further refines this by stopping the listeners on processes where they're not bound. Hence the intersection mentionned in the doc. But I'm rather surprized, maybe we recently broke something because we have a lot of people successfully running such configurations, which is why I'm a bit surprized. Willy
Re: nbproc 1 vs >1 performance
Hi Christian, Willy, Am 13.04.2016 um 12:58 schrieb Christian Ruppert: With the first config I get around ~30-33k requests/s on my test system, with the second conf (only the bind-process in the frontend section has been changed!) I just get around 26-28k requests per second. I could get similar differences when playing with nbproc 1 and >1 as well as the default "bind-process" and/or the "process 1" on the actual bind. Is it really just the multi process overhead causing the performance drop here, even tough the bind uses the first / only one process anyway? With the "bind-process 1-40" in the frontend (haproxy-lessperformant.cfg), you are forcing haproxy to use all 40 processes for this frontend traffic. Therefor you get all the multi-process penalties (process 1 will load balance the requests to all 39 child processes). That's why performance sucks - single process configuration will benefit from your testcase because the cost of the error response is negligible. With the "bind-process 1" in the frontend (inherited from the default section or the actual haproxy default: bind to the union of all the listeners' processes - haproxy-moreperformant.cfg) just a single process is used for that frontend (and if no other processes are needed, no other processes will in fact be forked). Because your testcase favors single process mode, it outperforms multiple processes easily (no inter process overhead). Try adding a dummy frontend to your moreperformant config: frontend dummy_allproc bind-process 2-40 bind : You will see that haproxy_test frontend still performs fine, although 40 processes are running, because only one process handles your ab test traffic (again no inter process overhead). So that is what's happening here, the question is if this is really expected behavior. For example, the following configuration load balances the traffic across all 40 processes, expected or not? frontend haproxy_test bind-process 1-40 bind :12345 process 1 The docs [1] are not very clear and kind of contradict themselves: does not enforce any process but eliminates those which do not match [...] If the frontend uses a "bind-process" setting, the intersection between the two is applied We mention in multiple places (docs, commits, ml) that we use the kernel's SO_REUSEPORT to loadbalance incoming traffic if supported, however due to the behavior mentioned above, when we do something like this: frontend haproxy_test bind :12345 process 1 bind :12345 process 2 bind :12345 process 3 bind :12345 process 4 all those sockets are bound to the master process, not its forks: lukas@ubuntuvm:~/haproxy$ ps auxww | grep haproxy haproxy 14457 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy -D -f ../cert/ruppert-nbproc-stress.cfg haproxy 14458 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy -D -f ../cert/ruppert-nbproc-stress.cfg haproxy 14459 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy -D -f ../cert/ruppert-nbproc-stress.cfg haproxy 14460 0.0 0.1 16928 592 ?Ss 22:41 0:00 ./haproxy -D -f ../cert/ruppert-nbproc-stress.cfg lukas@ubuntuvm:~/haproxy$ sudo netstat -tlp | grep 12345 tcp0 0 *:12345 *:* LISTEN 14457/haproxy tcp0 0 *:12345 *:* LISTEN 14457/haproxy tcp0 0 *:12345 *:* LISTEN 14457/haproxy tcp0 0 *:12345 *:* LISTEN 14457/haproxy lukas@ubuntuvm:~/haproxy$ So I'm not sure how SO_REUSEPORT is supposed to load-balance between processes, if all sockets point to the master process (it will just pass the request around, like without SO_REUSEPORT). The docs [1] really suggest this configuration for SO_REUSEPORT based kernel side load-balancing. It should look like this, imho: lukas@ubuntuvm:~/haproxy$ sudo netstat -tlp | grep 12345 tcp0 0 *:12345 *:* LISTEN 14713/haproxy tcp0 0 *:12345 *:* LISTEN 14712/haproxy tcp0 0 *:12345 *:* LISTEN 14711/haproxy tcp0 0 *:12345 *:* LISTEN 14710/haproxy lukas@ubuntuvm:~/haproxy$ But that will only work across frontends: frontend haproxy_test1 bind-process 1 bind :12345 default_backend backend_test frontend haproxy_test2 bind-process 2 bind :12345 default_backend backend_test frontend haproxy_test3 bind-process 3 bind :12345 default_backend backend_test frontend haproxy_test4 bind-process 4 bind :12345 default_backend backend_test I think the process keyword on the bind line is either buggy or simply not designed to use with SO_REUSEPORT load-balancing, although the documentation suggest it. Any insides here, Willy? [1] http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#5.1-process
nbproc 1 vs >1 performance
Hi, I've prepared a simple testcase: haproxy-moreperformant.cfg: global nbproc 40 user haproxy group haproxy maxconn 175000 defaults timeout client 300s timeout server 300s timeout queue 60s timeout connect 7s timeout http-request 10s maxconn 175000 bind-process 1 frontend haproxy_test #bind-process 1-40 bind :12345 process 1 mode http default_backend backend_test backend backend_test mode http errorfile 503 /etc/haproxy/test.error # vim: set syntax=haproxy: haproxy-lessperformant.cfg: global nbproc 40 user haproxy group haproxy maxconn 175000 defaults timeout client 300s timeout server 300s timeout queue 60s timeout connect 7s timeout http-request 10s maxconn 175000 bind-process 1 frontend haproxy_test bind-process 1-40 bind :12345 process 1 mode http default_backend backend_test backend backend_test mode http errorfile 503 /etc/haproxy/test.error # vim: set syntax=haproxy: /etc/haproxy/test.error: HTTP/1.0 200 Cache-Control: no-cache Content-Type: text/plain Test123456 The test: ab -n 5000 -c 250 http://xx.xx.xx.xx:12345 With the first config I get around ~30-33k requests/s on my test system, with the second conf (only the bind-process in the frontend section has been changed!) I just get around 26-28k requests per second. I could get similar differences when playing with nbproc 1 and >1 as well as the default "bind-process" and/or the "process 1" on the actual bind. Is it really just the multi process overhead causing the performance drop here, even tough the bind uses the first / only one process anyway? -- Regards, Christian Ruppert
