RE: reducing check requests with same servers in many backends

[Cassidy, Bryan]

I don't know of a way to only define the server list once, but in subsequent 
backends you can use 'track backend_name/sv_name' instead of 'check' to reduce 
the health check spam.

One gotcha I ran into is that if I have 'option httpchk HEAD /my/health/url' in 
the backend with 'check's I also needed to specify 'option httpchk HEAD 
/some/url' in the other backends with traces. The latter URL doesn't appear to 
matter; I use '/unused' in my config as a hint to my future self that it's the 
URLs in the backend with checks that matters, not the backends with traces. 
Unsure if this is a bug or if I'm doing something wrong (quite possibly the 
latter).

Bryan

From: Shawn Heisey [hapr...@elyograg.org]
Sent: Friday, October 05, 2012 17:10
To: haproxy@formilux.org
Subject: reducing check requests with same servers in many backends

I have a setup doing a lot of URL rewriting.  I set up an ACL to match
each string that I want to rewrite.  When each one matches, it sends the
request to a backend specific to that rewrite.  All of the backends
(over a dozen of them) have the same set of servers, each defined with
'check' in them.

This results in a lot of traffic just for checking -- pretty much
continuous.  It does not appear to be overwhelming the servers, but it
does make for an incredible amount of noise in packet captures taken at
the server.  I was able to reduce it a little bit by changing the check
interval to 20 seconds, but it's still pretty excessive, because it has
to check the same servers over a dozen times during that 20 second
interval.  I do have spread-checks enabled with a value of 5.  That
value is arbitrary, the documentation said values from 2 to 5 tend to be
good.

Is there any way to streamline the config so that I only need to define
the server list once, and have it only check the servers once for every
check interval?  I'm running 1.4.18 for this, but I could run a 1.5-dev
version if required.

Thanks,
Shawn

RE: Slowness in establishing connections from ruby to Haproxy after upgrading from 1.4.8 to 1.4.18

[Cassidy, Bryan]

Is there an ETA for this fix making its way to a 1.4 release? Also, is it known 
what version introduced undesirable behavior with ruby clients? 1.4.9 to 1.4.18 
is a fairly loose bound...

Thanks,
Bryan

-Original Message-
From: Willy Tarreau [mailto:w...@1wt.eu] 
Sent: Saturday, December 17, 2011 10:02 AM
To: Timothy Garnett
Cc: haproxy@formilux.org
Subject: Re: Slowness in establishing connections from ruby to Haproxy after 
upgrading from 1.4.8 to 1.4.18

Hi again Tim,

I could finally successfully reproduce the issue. Eventhough the client's 
behaviour is sub-optimal, haproxy's behaviour in this case is inappropriate, so 
I fixed it with the attached patch. The principle is that we re-enable TCP 
quick-ack if we know that we're waiting for more data from the client (headers 
or body).

It fixes all the test cases for me without removing the optimization for the 
common case. So I'm merging it into 1.5 and 1.4.

Regards,
Willy

RE: Apache translates 500 to 502 from haproxy

[Cassidy, Bryan]

Hello,

Check that Cherrypy is serving up valid HTTP. You could also try setting 
HAProxy to balance in TCP mode instead of HTTP mode, though if this helps it 
would just be masking any problem that might exist.

I once had a backend "500" response translated to 502 by HAProxy balancing in 
HTTP mode. I wrote 500 in quotes because the backend (Apache improperly 
configured in my case) served up an HTML document containing the words "500 
internal server error", but didn't actually serve up any HTTP headers prior to 
the document - just the document itself. HAProxy then changed the response to a 
502, as it should, because not including headers is obviously invalid HTTP. I 
was totally stumped until I ran tcpdump and saw what was happening.

Your setup is different than mine was, of course. But maybe this will give you 
a lead...

Hope this helps,
Bryan

-Original Message-
From: Sachin Shetty [mailto:sshe...@egnyte.com] 
Sent: Monday, June 13, 2011 11:12 AM
To: haproxy@formilux.org
Subject: Re: Apache translates 500 to 502 from haproxy

Willy Tarreau  1wt.eu> writes:

> 
> On Fri, Jun 10, 2011 at 04:41:08PM +0530, Manoj Kumar wrote:
> > Hi,
> > 
> > We are forwarding specific requests from apache to haproxy which 
> > interbally forwards it to a pool of cherry py servers. We have seen 
> > that certain requests end up in 500 in haproxy and cherry py logs 
> > which is ok and understood, but apache instead sends a 502 to the client.
> 
> Maybe for any reason apache is rejecting the response and aborting the 
> connection, which might explain that message in your logs :
> 
> > [Fri Jun 10 00:46:01 2011] [error] (103)Software caused connection
> > abort: proxy: pass request body failed to 192.168.2.100:9910 
> >   (192.168.2.15)
> 
> Willy
> 
> 


Hi Willy,

I spent some more time looking in to this, notice the error in apache log, it 
is parsing request body and not response

I think this is what is gong on:

1. Apache receives a POST request
2. Forwards to haproxy
3. haproxy forwards to Cherrypy
4. Cherrypy aborts the request due to some internal error, returns 401/500 5. 
haproxy sends the response back to Apache and terminates the connection 6. 
Apache however is still expecting somebody will read the posted response and 
barfs with pass request body failed error

Now this is definitely due to haproxy, since if I skip haproxy and make Apache 
hit cherrypy directly, I see a proper response code from Apache. I think 
haproxy is terminating the connection prematurely when the backend server 
returns and error status code


Any idea?

maxconn vs. option httpchk

[Cassidy, Bryan]

Hi all,

I've noticed an odd (lack of) interaction between "maxconn" and "option 
httpchk"... 

If a server's maxconn limit has been reached, it appears that HTTP health 
checks are still dispatched. If I've configured the maxconn limit to match the 
number of requests the backend server can concurrently dispatch, and all these 
connections are busy with slow requests, HAProxy will assume the server is 
down; once the server completes a request, HAProxy waits until "rise" health 
checks have succeeded (as expected if the server was really down, but it was 
only busy). This makes overly busy times even worse.

I'm not sure if this explanation is clear; perhaps a concrete configuration 
might help.

listen load_balancer
bind :80
mode http

balance leastconn
option httpchk HEAD /healthchk
http-check disable-on-404

default-server port 8080 inter 2s rise 2 fall 1 maxconn 3
server srv1 srv1.example.com:8080 check
server srv2 srv2.example.com:8080 check

With the above toy example, if each of srv1 and srv2 can only respond to 3 
requests concurrently, and 6 slow requests come in (each taking more than 2 
seconds), both backend servers will be considered down until up to 4 seconds in 
the worst case (inter 2s * rise 2) after one of the requests finishes.

I know I can work around this by setting maxconn to one less than a server's 
maximum capacity (perhaps this would be a good idea for other reasons). I 
suspect I could work around this by using TCP status checks instead of HTTP 
status checks, though I haven't tried this as I like the flexibility HTTP 
health checks offer (like "disable-on-404").

Is this behavior a bug or a feature? Intuitively I would have expected the HTTP 
health checks to respect maxconn limits, but perhaps there was a conscious 
decision to not do so (for instance, maybe it was considered unacceptable for a 
server's health to be unknown when it is fully loaded).

Thanks,
Bryan