HAProxy Response time performance

[Matt Christiansen]

Hello,

I am wanting to move to HAProxy for my load balancing solution. Over
all I have been greatly impressed with it. It has way more throughput
and can handle way more connections then our current LB Solution
(nginx). I have been noticing one issue in all of our tests though, it
seems like in the TP99.9 (and greater) of response times is much MUCH
higher then nginx and we have a lot of outliers.

Our test makes a call to the VIP and times the time it takes to
receive the data back then pauses for a sec or two and makes the next
response. In both of the sample results below I did 2000 requests.

HAProxy

Average: 39.71128451818
Median: 29.4217891182
tp90: 67.48199012481
tp99: 313.29083442688
tp99.9: 562.318801879883
Over 500ms: 10
Over 2000ms: 0

nginx

Average: 69.6072148084641
Median: 59.2541694641113
tp90: 87.6350402832031
tp99: 112.42142221222
tp99.9: 180.88918274272
Over 500ms: 0
Over 2000ms: 0

So as you can see a big difference in the TP99.9 and a big difference
in the outlier count but the average and median response time are
really low.

We are running a pretty stock centos 5.6 server install with HAProxy
1.4.15, HAProxy isn't using more then like 4% of the CPU and the
System CPU is closer to 12%.

I was wondering if you guys had any obvious response time related
performance tweaks I can try. If you need more info let me know too.

Thanks,
Matt C.

Re: start haproxy not as root?

[Willy Tarreau]

Hi Jacob,

On Wed, Jun 08, 2011 at 08:47:15PM -0400, Jacob Fenwick wrote:
> Willy,
> 
> Thanks for your quick response.
> 
> I did some experimenting and realized you were right.
> I think my issue why I couldn't run it as anything but root was because I
> was using the -p option to store the pid in /var/run/haproxy.pid, which
> could only accessed by root.

That's a common issue when switching a conf written for root to a non-root
user.

> However, I am still having some problems.
> 
> I'm trying to restart haproxy using a Python script that I call from Apache
> using this haproxy command:
> /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -st 4140

I'm assuming that you fixed the "-p /var/run/haproxy.pid" above.

> I'm not exactly sure what I'm doing wrong but I'm getting this error:
>  * Starting haproxy haproxy
> [ALERT] 158/203608 (4374) : error when trying to preserve previous UNIX
> socket (/tmp/haproxy). Aborting.

This looks like the global stats socket.

> [ALERT] 158/203608 (4374) : [/usr/sbin/haproxy.main()] Some protocols failed
> to start their listeners! Exiting.
>...fail!
> 
> Before this error I was having some issues that were clearly privilege
> related.
> I've verified that /var/run/haproxy.pid and /tmp/haproxy, which both seem to
> be accessed, both have chmod 777.

It makes no sense to set chmod 777 to such entries, because :
  - neither /var/run/haproxy.pid nor the socket needs to be executable
  - what is important is not that any user on the system can write to
those files (or sockets) but that you have write access to the
directory hosting them.

In practice, when haproxy tries to remove the pidfile, if it fails, it
silently ignores it provided that it can rewrite the file. But you should
definitely move that pidfile out of /var/run if you intend to run your
process without root permissions.

However, for the stats socket, it *needs* to be able to link/unlink the
old socket. If it fails, it means that you simply don't have the permissions
on /tmp/haproxy. I'm sure that you first started it as root and that you left
/tmp/haproxy owned by root. Whatever the chmod you'll do there will no way
for a non-root user to link/unlink this entry as long as it's owned by root
due to the sticky bit on the /tmp directory.

> Everyone at least has read/write access on /etc/haproxy/haproxy.cfg.

Here again, it makes no sense to give RW access to everyone. It's the worst
thing to do. Some services such as SSH carefully check permissions on their
config files and refuse to start or to accept keys if some files have too
much permission, because they expect those files could have been compromised.

> The user Apache runs the script as is www-data, so I made sure to start
> haproxy with the global parameter user set to www-data.
> Not sure if you have to be the same user as you started it as the first time
> but I assume you do.

You have to remove this global user, because haproxy will not have the
possibility to change it since it's not root. Same for the group.

> Do you know what I could be doing wrong here?

What I'm analysing is that you're trying to make it work with many locations
which are limited to root only, but without root permissions. What you should
*really* do is to make a specific directory which is owned by the www-data
user and put your sockets, pid file etc... there. This directory could very
well be /var/run/haproxy if it helps. The problems you're facing are just
standard unix permissions issues.

Hoping this helps,
Willy

Re: FW: Tech Site Seeks Parkers - Traffic Doubles Every Quarter

[Duncan Hall]

Wow,

$2300 in a whole year, do you think as an open source community we 
should do this? Imagine how much we could donate to the EFF in just 1 
month.


I always thought Eli Boggs was a pirate.

I'm not suggesting the haproxy list should be moderated but perhaps 
there should be a capture, challenge response or sign up before posting.


regards,

Duncan



On 09/06/11 14:27, Alexander Boggs wrote:


Leading Technology Site Seeks New Partners

The Techno Club 
 
Boasts...


$2,300+in revenue over the past 12 months
100%+ revenuegrowthover the past 12 months
130%+ traffic growth over the past 3 months(verified by Alexa) 


Average Visitor Annual Income - $42, 186

The above numbers speak for themselves. Revenue doubling annually 
andtraffic doubling quarterly, you will not find a site growing faster 
than that.
The Techno Club offers access to the most sought after demographic: 
technology professionals and early adopters. The costs to maintain the 
business is only in supplying content and the current owner is willing 
to stay on to ensure continuity. He is looking for new owners so that 
he can focus on writing.


*Reply if you would like to take The Techno Club to the next level.*
Best Regards,

Alexander Boggs
VP Technology
Eli Boggs Media









2885 Sanford Avenue, South West #15918 Grandville, Michigan  49418 if 
you would like to never hear from us again head to obsidianpunch. 
com/unsub

FW: Tech Site Seeks Parkers - Traffic Doubles Every Quarter

[Alexander Boggs]

Title: TTC


Leading
Technology Site Seeks New PartnersThe
Techno Club Boasts...
$2,300+ in revenue over the past 12 months100%+ revenue growth over the past 12 months130%+ traffic growth over the past 3 months (verified
by Alexa)Average Visitor Annual Income - $42, 186
The above numbers speak for themselves.
Revenue
doubling annually and traffic doubling quarterly, you will not
find a site growing faster than that. 
 
The Techno Club offers access to the most
sought after demographic: technology professionals and early adopters. The
costs to maintain the business is only in supplying content and the current
owner is willing to stay on to ensure continuity. He is looking for new
owners so that he can focus on writing. Reply if you would
like to take The Techno Club to the next level. 
 
Best Regards,Alexander BoggsVP
TechnologyEli Boggs Media




2885 Sanford
Avenue, South West #15918 Grandville, Michigan  49418 if you would like
to never hear from us again head to obsidianpunch.
com/unsub

Re: start haproxy not as root?

[Jacob Fenwick]

Willy,

Thanks for your quick response.

I did some experimenting and realized you were right.
I think my issue why I couldn't run it as anything but root was because I
was using the -p option to store the pid in /var/run/haproxy.pid, which
could only accessed by root.

However, I am still having some problems.

I'm trying to restart haproxy using a Python script that I call from Apache
using this haproxy command:
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -st
4140

I'm not exactly sure what I'm doing wrong but I'm getting this error:
 * Starting haproxy haproxy
[ALERT] 158/203608 (4374) : error when trying to preserve previous UNIX
socket (/tmp/haproxy). Aborting.
[ALERT] 158/203608 (4374) : [/usr/sbin/haproxy.main()] Some protocols failed
to start their listeners! Exiting.
   ...fail!

Before this error I was having some issues that were clearly privilege
related.
I've verified that /var/run/haproxy.pid and /tmp/haproxy, which both seem to
be accessed, both have chmod 777.
Everyone at least has read/write access on /etc/haproxy/haproxy.cfg.

The user Apache runs the script as is www-data, so I made sure to start
haproxy with the global parameter user set to www-data.
Not sure if you have to be the same user as you started it as the first time
but I assume you do.

Do you know what I could be doing wrong here?

Jacob

On Wed, Jun 8, 2011 at 6:20 PM, Willy Tarreau  wrote:

> On Thu, Jun 09, 2011 at 12:07:27AM +0200, Graeme Donaldson wrote:
> > On 9 June 2011 00:05, Jacob Fenwick  wrote:
> >
> > > It seems like I must be root to start haproxy.
> > >
> > > I know that I can add a user line in global so that the process will
> change
> > > to say it is running as a non-root user once it is running, but it
> seems
> > > like I still need to be root to actually start it, or restart it.
> > >
> > > Is there any way around this?
> > >
> > >
> > I don't think there is, and if there was, you would be unable to listen
> on
> > any ports <1024, as only root can do that.
>
> Well, many people start it as non-root on ports >= 1024. That's very common
> among developers who all like to have their own instance. For this you just
> have to remove the "user" and "group" lines and ensure that all your ports
> are >= 1024 and that your maxconn is low enough to accomodate the default
> 1024 file descriptors limit which is imposed to non-root users. Ah, and if
> you're not root, you can't chroot either.
>
> Those constraints are the precise reason why it's highly recommended to
> start it as root. But if you can't, there's no problem as long as you can
> live with the constraints.
>
> As an example, all my test config files use unprivileged ports so that I
> can start it without being root during development sessions.
>
> Cheers,
> Willy
>
>

Re: start haproxy not as root?

[Willy Tarreau]

On Thu, Jun 09, 2011 at 12:07:27AM +0200, Graeme Donaldson wrote:
> On 9 June 2011 00:05, Jacob Fenwick  wrote:
> 
> > It seems like I must be root to start haproxy.
> >
> > I know that I can add a user line in global so that the process will change
> > to say it is running as a non-root user once it is running, but it seems
> > like I still need to be root to actually start it, or restart it.
> >
> > Is there any way around this?
> >
> >
> I don't think there is, and if there was, you would be unable to listen on
> any ports <1024, as only root can do that.

Well, many people start it as non-root on ports >= 1024. That's very common
among developers who all like to have their own instance. For this you just
have to remove the "user" and "group" lines and ensure that all your ports
are >= 1024 and that your maxconn is low enough to accomodate the default
1024 file descriptors limit which is imposed to non-root users. Ah, and if
you're not root, you can't chroot either.

Those constraints are the precise reason why it's highly recommended to
start it as root. But if you can't, there's no problem as long as you can
live with the constraints.

As an example, all my test config files use unprivileged ports so that I
can start it without being root during development sessions.

Cheers,
Willy

Re: start haproxy not as root?

[Jacob Fenwick]

That's fine, I don't need any ports <1024.

If anyone knows of a solution let me know but I'm guessing it is unlikely.

On Wed, Jun 8, 2011 at 6:07 PM, Graeme Donaldson wrote:

> On 9 June 2011 00:05, Jacob Fenwick  wrote:
>
>> It seems like I must be root to start haproxy.
>>
>> I know that I can add a user line in global so that the process will
>> change to say it is running as a non-root user once it is running, but it
>> seems like I still need to be root to actually start it, or restart it.
>>
>> Is there any way around this?
>>
>>
> I don't think there is, and if there was, you would be unable to listen on
> any ports <1024, as only root can do that.
>
> Graeme.
>
>

Re: start haproxy not as root?

[Graeme Donaldson]

On 9 June 2011 00:05, Jacob Fenwick  wrote:

> It seems like I must be root to start haproxy.
>
> I know that I can add a user line in global so that the process will change
> to say it is running as a non-root user once it is running, but it seems
> like I still need to be root to actually start it, or restart it.
>
> Is there any way around this?
>
>
I don't think there is, and if there was, you would be unable to listen on
any ports <1024, as only root can do that.

Graeme.

start haproxy not as root?

[Jacob Fenwick]

It seems like I must be root to start haproxy.

I know that I can add a user line in global so that the process will change
to say it is running as a non-root user once it is running, but it seems
like I still need to be root to actually start it, or restart it.

Is there any way around this?

Jacob

Re: problem with ":" in name

[Cyril Bonté]

Hi Alexander,

Le mercredi 8 juin 2011 19:58:25, Alexander Hollerith a écrit :
> Hi,
> 
> after some frustration with the "enable/disable" feature on the stats web
> UI I stumbled across something that I thought might be intersting to know.
> (...)
> Not sure if I did something wrong in the rest of my config (which I can't
> disclose), but for now it looks like a problem with a ":" in the name.

This should have already been fixed with this commit :
http://haproxy.1wt.eu/git?p=haproxy-1.4.git;a=commit;h=25fa8b30f0bafa372dc1384b65bd5c5cafacf636

It will be available with haproxy 1.4.16 (or is already available in the last 
snapshot).

-- 
Cyril Bonté

problem with ":" in name

[Alexander Hollerith]

Hi,

after some frustration with the "enable/disable" feature on the stats web UI I 
stumbled across something that I thought might be intersting to know.

When trying to disable a server via stats web UI I found that it results in a 
"[X] Nothing has changed." message with a listen section configured like so:

listen .45:8443 192.168.240.45:8443
option ssl-hello-chk
server A 192.168.240.10:8443 check
server B 192.168.240.20:8443 check

However it works perfectly ("[X] Action processed successfully.") when 
configured like so:

listen some_other_name 192.168.240.45:8443
option ssl-hello-chk
server A 192.168.240.10:8443 check
server B 192.168.240.20:8443 check

First thing I checked was documentation regarding names, and there it says:

All proxy names must be formed from upper and lower case letters, digits,
'-' (dash), '_' (underscore) , '.' (dot) and ':' (colon).


Not sure if I did something wrong in the rest of my config (which I can't 
disclose), but for now it looks like a problem with a ":" in the name.

With kind regards
A.

--
Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis 
dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec 
preripiat quisquam non sibi parata.

Re: [PATCH] [MINOR] Allow shutdown of sessions when a server becomes unavailable

[Simon Horman]

On Wed, Jun 08, 2011 at 10:55:44AM +0200, Willy Tarreau wrote:
> On Wed, Jun 08, 2011 at 05:00:44PM +0900, Simon Horman wrote:
> > I'm happy with documenting things as you suggest.
> 
> Fine.
> 
> > Do you have any feeling of how difficult it would be
> > to add a per-server session list?
> 
> Not particularly difficult, we just need to add a struct list member
> into the session and use it to chain sessions behind their servers.
> We also need to ensure that each time we redispatch, we move this.
> Probably that we should have a list head in the backend so that sessions
> that are not yet on a server should be chained somewhere.
> 
> > > Another point is that a flapping server can definitely make a service
> > > totally unusable and that must be indicated too. You can check any stats
> (...)
> > I considered that too. My idea was that perhaps connections should only be
> > closed when they receive their next packet.  That is, when a read is made
> > from the client for a session whose server is down.
> 
> No, this will not cover most users' needs. Most of the times, the issues was
> to quickly notify the client that the server is dead so that the client does
> not have to wait for a timeout or for activity to be aware of it.
> 
> > But it it seems to me that there demand for the feature I have implemented
> > (modulo bugs and performance issues). And that alternative schemes such as
> > the one I describe above could be separate on-markeddown actions.
> 
> I really don't think it will be that much useful anyway. So yes, if there is
> demand that makes sense, a separate action can be added.
> 
> > > Well, maybe we should still consider the possibility that we only kill the
> > > connection on the server side. For current state of affairs, it will not
> > > change anything, but when we later support server-side keep-alive and 
> > > later
> > > connection pools, it will make it easier to retry the request over another
> > > connection.
> > 
> > Sure, perhaps that could be another on-markedown action?
> 
> Not necessarily : currently, if you only close on the server side, the
> effect will be propagated to the client side, so it already does the right
> thing and at the same time will allow us to improve on this later.

Ok, understood. I'll try your suggestion.

Re: haproxy participates to world IPv6 day

[Brane F. Gračnar]

On Wednesday 08 of June 2011 07:42:24 Willy Tarreau wrote:
> Hi all,
> 
> The haproxy website was registered among about 300 other ones which
> participates to the world IPv6 day event :

Willy, preparations for IPv6 day would be a really big challenge if HaProxy 
would not exist.

Thanks!

Best regards, Brane

Re: [PATCH] [MINOR] Allow shutdown of sessions when a server becomes unavailable

[Willy Tarreau]

On Wed, Jun 08, 2011 at 05:00:44PM +0900, Simon Horman wrote:
> I'm happy with documenting things as you suggest.

Fine.

> Do you have any feeling of how difficult it would be
> to add a per-server session list?

Not particularly difficult, we just need to add a struct list member
into the session and use it to chain sessions behind their servers.
We also need to ensure that each time we redispatch, we move this.
Probably that we should have a list head in the backend so that sessions
that are not yet on a server should be chained somewhere.

> > Another point is that a flapping server can definitely make a service
> > totally unusable and that must be indicated too. You can check any stats
(...)
> I considered that too. My idea was that perhaps connections should only be
> closed when they receive their next packet.  That is, when a read is made
> from the client for a session whose server is down.

No, this will not cover most users' needs. Most of the times, the issues was
to quickly notify the client that the server is dead so that the client does
not have to wait for a timeout or for activity to be aware of it.

> But it it seems to me that there demand for the feature I have implemented
> (modulo bugs and performance issues). And that alternative schemes such as
> the one I describe above could be separate on-markeddown actions.

I really don't think it will be that much useful anyway. So yes, if there is
demand that makes sense, a separate action can be added.

> > Well, maybe we should still consider the possibility that we only kill the
> > connection on the server side. For current state of affairs, it will not
> > change anything, but when we later support server-side keep-alive and later
> > connection pools, it will make it easier to retry the request over another
> > connection.
> 
> Sure, perhaps that could be another on-markedown action?

Not necessarily : currently, if you only close on the server side, the
effect will be propagated to the client side, so it already does the right
thing and at the same time will allow us to improve on this later.

Cheers,
Willy

Re: [PATCH] [MINOR] Allow shutdown of sessions when a server becomes unavailable

[Simon Horman]

On Wed, Jun 08, 2011 at 07:32:29AM +0200, Willy Tarreau wrote:
> Hi Simon,
> 
> On Wed, Jun 08, 2011 at 09:55:21AM +0900, Simon Horman wrote:
> (...)
> > I should note that I'm not entirely sure about the performance implications
> > of looping through all sessions like this. Perhaps a per-server list is 
> > needed.
> 
> That's exactly what I wanted to respond to your mail :-)
> I always refused to implement that until we have a per-server session
> list, but I also know there it increasing demand for this feature due to
> long-running sessions (eg: WebSockets).
> 
> There are sites that are running with more than 100k concurrent connections
> and the cost can be huge there. One could say that a site running at 100k
> conns will have a decent hardware which will not take too long to scan all
> the table though.
> 
> I think that what could be done is to document the feature as possibly
> very expensive when servers go down or when servers are flapping. If we
> explain all the risks in the doc, users will be able to check the impact
> for their usage.

I'm happy with documenting things as you suggest.
Do you have any feeling of how difficult it would be
to add a per-server session list?

> Another point is that a flapping server can definitely make a service
> totally unusable and that must be indicated too. You can check any stats
> page you have access to, and consider that for each occurrence of a Down
> event on a server, all those server's users will be kicked out. Right now,
> when checks have too short timeouts, it can happen that a server goes down
> for a short period without much impact. It does not get any new connections
> for a few seconds but still serve the current ones. With this feature, all
> connections will be killed for each such event. Flapping servers are a
> common thing in environments where servers are overloaded or support too
> few concurrent connections. I think that a warning about this in the doc
> is absolutely necessary too.

I considered that too. My idea was that perhaps connections should only be
closed when they receive their next packet.  That is, when a read is made
from the client for a session whose server is down.

But it it seems to me that there demand for the feature I have implemented
(modulo bugs and performance issues). And that alternative schemes such as
the one I describe above could be separate on-markeddown actions.


> > I'm also unsure of the performance implications of waking up a potentially
> > large number of sessions.
> 
> Yes it will cost a lot too. And it will flood the logs at extreme speed.
> Think that you're possibly sending 10-100k logs within a few hundreds of
> milliseconds, that can be up to one million log messages per second. Most
> likely, the time spent in sprintf() will sensibly extend the processing
> time (eg: about 100k messages per second, meaning a freeze for up to one
> second) and most of the logs will be lost. That can be documented too :-)
> 
> > Perhaps their priority should be reduced?
> 
> That's a good idea. At first read I was against this because there is no
> way to restore the priority after that, but since the tasks are dying,
> it's not an issue :-)

Haha. I was worried about not being able to restore their priority too.
But as you point out, its a moot point.

> Well, maybe we should still consider the possibility that we only kill the
> connection on the server side. For current state of affairs, it will not
> change anything, but when we later support server-side keep-alive and later
> connection pools, it will make it easier to retry the request over another
> connection.

Sure, perhaps that could be another on-markedown action?

> Oh, BTW, I think you'd better use TASK_WOKEN_OTHER than TASK_WOKEN_RES,
> because the later is to indicate that a previously unavailable resource
> has become available (reason why you see it in queue processing).

Thanks, I was unsure of which value to use.

> Another point, please check that it works with servers put into maintenance
> mode, I'm not sure this part calls set_server_down().

Thanks, will do.

Re: PythonPaste and HAProxy

[Christian Klinger]

Hi Baptiste,

unfortunatly this does not help.
Any other ideas?

Christian


Hi,

Maybe you should try this:
option httpchk HEAD /app/haproxycheck
http-check expect status 200

cheers

On Wed, Jun 1, 2011 at 10:59 AM, Christian Klinger  wrote:

Hi,

i try to loadbalance some PythonPaste Servers with the help of haproxy.

I have configured this healthcheck:

...
option httpchk GET /app/haproxycheck
http-check expect status 200
...

All works fine, so far. But i get this error on my PasteConsole...

Traceback (most recent call last):
  File
"/opt/extranet/lib/python2.6/site-packages/Paste-1.7.5.1-py2.6.egg/paste/httpserver.py",
line 1068, in process_request_in_thread
self.finish_request(request, client_address)
  File "/usr/local/lib/python2.6/SocketServer.py", line 322, in
finish_request
self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python2.6/SocketServer.py", line 618, in __init__
self.finish()
  File "/usr/local/lib/python2.6/SocketServer.py", line 661, in finish
self.wfile.flush()
  File "/usr/local/lib/python2.6/socket.py", line 297, in flush
self._sock.sendall(buffer(data, write_offset, buffer_size))
error: [Errno 32] Datenübergabe unterbrochen (broken pipe)


I guess it's because haproxy does not wait on the response of the
haproxycheck page from the webserver.

Any idea what i can do to fix it?

Christian