Haproxy consulting

[Cory Forsyth]

Hi, my company would like to hire someone for a few hours' worth of
consulting time to help us gut-check our haproxy configuration and set
up.

In particular, this is what we are trying to do:

We are trying to limit connections to our server by IP address, but
over a given time window for each IP.  If it has connected in the last
5 minutes it is allowed to continue connecting, regardless of whether
the IP limit has been reached.
If it is a new IP, it is only allowed if the number of other IPs is
below the limit.  So if an IP gets "in", as long as it continues to
connect at least once every 5 minutes it will always be allowed to
continue connecting.

I have set something up to do this using a secondary process to check
the haproxy stick-table (via socat) for the number of entries (and the
entries are tracked by IP and expired after 5minutes), and if the
number of entries is greater than the limit this shuts down a Sinatra
ruby app that is configured as a backend in haproxy's config...and the
configured frontend has an ACL that checks whether that backend is
down when deciding if it can allow in a new IP.

We'd like some expert eyes to look over this setup and suggest
alternatives or improvements, and also suggestions for how to load
test this setup to make sure it will work well at scale.

thanks,
Cory

Re: unexpected server disconnect ("SH--") when using HAProxy 1.5x

[Cory Forsyth]

I don't think the number of seconds is significant. I tried it again now,
reloading several times and each time the number of milliseconds in that
SH-- line is different. I think that just indicates how long the
long-polling connection was open when the reload severed it.

Here's the config:

global
maxconn 3
stats socket /var/run/haproxy.stat mode 600 level admin

defaults
mode http
timeout connect 5000ms
timeout client 6ms
timeout server 6ms

backend stats_be
stats uri /stats
stats refresh 5s

backend the_backend
server the_backend1 x.x.x.x:80

frontend stats_fe
bind *:8080
default_backend stats_be

frontend http_proxy
log 127.0.0.1local0
log 127.0.0.1local1 err
bind *:80
mode http
option forwardfor
option httplog
option log-separate-errors

# log the name of the virtual server
capture request  header Host len 20

# log the amount of data uploaded during a POST
capture request  header Content-Length len 10

# log the beginning of the referrer
capture request  header Referer len 20
capture request  header User-Agent len 20

# server name (useful for outgoing proxies only)
capture response header Server len 20

# logging the content-length is useful with "option logasap"
capture response header Content-Length len 10

default_backend the_backend


Here are a few more examples of the SH-- lines:

Mar 25 12:22:35 localhost haproxy-1.5v4[28922]:
96.246.63.146:63288[25/Mar/2011:12:22:34.341] http_proxy
the_backend/the_backend1 6/0/0/-1/1166
502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|http://the-proxy|Mozilla/5.0
(Macinto} {|} "POST / HTTP/1.1"
Mar 25 12:22:40 localhost haproxy-1.5v4[28922]:
96.246.63.146:63296[25/Mar/2011:12:22:36.945] http_proxy
the_backend/the_backend1 6/0/0/-1/3199
502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|http://the-proxy|Mozilla/5.0
(Macinto} {|} "POST / HTTP/1.1"
Mar 25 12:22:52 localhost haproxy-1.5v4[28922]:
96.246.63.146:63302[25/Mar/2011:12:22:41.440] http_proxy
the_backend/the_backend1
6/0/0/-1/10988 502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|
http://the-proxy|Mozilla/5.0 (Macinto} {|} "POST / HTTP/1.1"
Mar 25 12:24:56 localhost haproxy-1.5v4[30407]:
96.246.63.146:63609[25/Mar/2011:12:24:49.936] http_proxy
the_backend/the_backend1 6/0/0/-1/6924
502 204 - - SH-- 0/0/0/0/0 0/0 {the-proxy.etc|63|http://the-proxy|Mozilla/5.0
(Macinto} {|} "POST / HTTP/1.1"

(I replaced some identifying info in those headers with "the-proxy.etc")

thanks,
-Cory

On Thu, Mar 24, 2011 at 4:06 PM, Willy Tarreau  wrote:

> Hi Cory,
>
> On Thu, Mar 24, 2011 at 01:10:49PM -0400, Cory Forsyth wrote:
> > I'm not sure if this is a bug in HAProxy, or if upgrading HAProxy to 1.4
> has
> > uncovered a bug in my application.
> >
> > I am developing a web app that uses long polling. I'm proxying its
> requests
> > through haproxy.  I've noticed that when reload the page in the browser
> > (which presumably severs the connection that the javascript is holding
> open
> > to the server through haproxy), I get an http status 502 error logged by
> > haproxy. It looks something like this:
> >
> > Mar 23 12:43:45 localhost haproxy-1.5v4[5969]:
> > 96.246.63.146:61899[23/Mar/2011:12:43:25.276] http_proxy
> > backend1/backendserver1 6/0/0/-1/20014
> > 502 204 - - SH-- 0/0/0/0/0 0/0
> > {proxy.myserver|65|http://proxy.myserver|Mozilla/5.0
> > (Macinto} {|} "POST / HTTP/1.1"
> >
> > This happens in Haproxy version 1.5-dev3 and 1.5-dev4, but it does not
> > happen in haproxy version 1.4.13.  Is there some additional investigation
> I
> > can do to determine if it's a bug in 1.5x or just something that my web
> app
> > shouldn't have been doing anyway that is now (correctly) being logged as
> an
> > error by 1.5x?
>
> What I understand from the logs is that the server suddenly closed
> after exactly 20 seconds. Maybe this is not what happened, but we
> have to find why it was logged like this. Could you please post your
> config ?
>
> Thanks,
> Willy
>
>

unexpected server disconnect ("SH--") when using HAProxy 1.5x

[Cory Forsyth]

I'm not sure if this is a bug in HAProxy, or if upgrading HAProxy to 1.4 has
uncovered a bug in my application.

I am developing a web app that uses long polling. I'm proxying its requests
through haproxy.  I've noticed that when reload the page in the browser
(which presumably severs the connection that the javascript is holding open
to the server through haproxy), I get an http status 502 error logged by
haproxy. It looks something like this:

Mar 23 12:43:45 localhost haproxy-1.5v4[5969]:
96.246.63.146:61899[23/Mar/2011:12:43:25.276] http_proxy
backend1/backendserver1 6/0/0/-1/20014
502 204 - - SH-- 0/0/0/0/0 0/0
{proxy.myserver|65|http://proxy.myserver|Mozilla/5.0
(Macinto} {|} "POST / HTTP/1.1"

This happens in Haproxy version 1.5-dev3 and 1.5-dev4, but it does not
happen in haproxy version 1.4.13.  Is there some additional investigation I
can do to determine if it's a bug in 1.5x or just something that my web app
shouldn't have been doing anyway that is now (correctly) being logged as an
error by 1.5x?

thanks,
Cory

Re: odd bug in 1.5-dev4

[Cory Forsyth]

Yes, that was the fix, thank you.

On Tue, Mar 22, 2011 at 6:55 PM, Cyril Bonté  wrote:

> Hi Cory,
>
> Le mardi 22 mars 2011 23:45:41, Cory Forsyth a écrit :
> > I'm running 1.5-dev4 on ubuntu (linux26 target) with the following config
> (...)
> > When I run it with "haproxy -f config_file -d" and then do a "curl
> > localhost" in another terminal, I get a screenful of this:
> >
> > 0010:http_proxy.clireq[0023:]: GET / HTTP/1.1
> > 0010:http_proxy.clihdr[0023:]: User-Agent: curl/7.19.7
> > (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3
> > libidn/1.15 0010:http_proxy.clihdr[0023:]: Host: localhost
> > 0010:http_proxy.clihdr[0023:]: Accept: */*
> > (...)
> >
> > I have the full output from haproxy; let me know if you need more
> > information for testing.
>
> This looks like the bug reported yesterday about a loop in haproxy 1.5-dev4
> and fixed by David de Colombier.
>
> Can you try to apply this patch and try again ?
>
> http://haproxy.1wt.eu/git?p=haproxy.git;a=commit;h=64e9c90e69cd8b0fe8dd60024ccbe528705fbd8f
>
> --
> Cyril Bonté
>

odd bug in 1.5-dev4

[Cory Forsyth]

I'm running 1.5-dev4 on ubuntu (linux26 target) with the following config
file:

defaults
mode http
timeout connect 5000ms
timeout client 6ms
timeout server 6ms

backend test_be
server goog google.com:80

frontend http_proxy
bind *:80
mode http
default_backend test_be

When I run it with "haproxy -f config_file -d" and then do a "curl
localhost" in another terminal, I get a screenful of this:

0010:http_proxy.clireq[0023:]: GET / HTTP/1.1
0010:http_proxy.clihdr[0023:]: User-Agent: curl/7.19.7
(x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
0010:http_proxy.clihdr[0023:]: Host: localhost
0010:http_proxy.clihdr[0023:]: Accept: */*

That snippet gets repeated exactly 16000 times, and eventually my curl
command errors out with this response:
504 Gateway Time-out
The server didn't respond in time.


If I add the line "option forwardfor" to the frontend config section, and
curl again, I get a similar result, along with many many lines that look
like:
0008:http_proxy.clihdr[0013:]: X-Forwarded-For: 127.0.0.1

I've tested the same config file with version 1.5-dev3 and 1.4.13, and they
both work fine.

I have the full output from haproxy; let me know if you need more
information for testing.

thanks,
Cory

Re: proper way to use an acl + stick-table to filter based on conn_cur

[Cory Forsyth]

Interesting...

I was able to get it to work using a stick-table on the front-end, as
bartavelle mentioned from this URL:
http://tehlose.wordpress.com/2010/12/15/fun-stuff-with-latest-haproxy-version/

I
don't know enough C to dig into the code to check on that, though.

On Tue, Mar 15, 2011 at 4:34 PM, Cyril Bonté  wrote:

> Hi Willy and Cory,
>
> Le mardi 15 mars 2011 22:17:50, Willy Tarreau a écrit :
> > > Whether I use "src_conn_cur" or sc1_conn_cur, with or without the table
> > > argument, this does not work. No matter how many concurrent connections
> > > per ip in the stick table, they never get denied.
> > >
> > > Any suggestions?
> >
> > At first glance, I cannot spot anything wrong.
>
> I think there's a bug in the function acl_fetch_src_conn_cur() :
> its code contains return acl_fetch_conn_cnt(...)
> where it probably should be return acl_fetch_conn_cur(...)
>
> Sorry, I can't test it tonight but maybe this can help you.
>
> --
> Cyril Bonté
>



-- 
[image: Follow me on twitter] get
your own Movable Ink 

proper way to use an acl + stick-table to filter based on conn_cur

[Cory Forsyth]

I have an haproxy.conf like so. I'm trying to limit based on the concurrent
connections.

backend thebackend
stick-table type ip size 8k expire 5m store gpc0,conn_cur
tcp-request content  track-sc1 src
acl mark_seen sc1_inc_gpc0
acl needs_increment src_get_gpc0(union) eq 0
tcp-response content accept if needs_increment mark_seen
server x.y.z:80

backend over_concurrent_per_ip
option httplog
log 127.0.0.1 local1
block if TRUE

frontend http_proxy
log 127.0.0.1local0
log 127.0.0.1local1 err
bind *:80
mode http
option forwardfor
option httplog
option log-separate-errors
default_backend thebackend
acl too_many_from_ip src_conn_cur(thebackend) gt 0
use_backend over_concurrent_per_ip if too_many_from_ip


Whether I use "src_conn_cur" or sc1_conn_cur, with or without the table
argument, this does not work. No matter how many concurrent connections per
ip in the stick table, they never get denied.

Any suggestions?

Re: question on incrementing and reading stick-table gpc counter

[Cory Forsyth]

I solved my problem.

The issue was I needed to specify the correct stick-table in the
src_get_gpc0 call. The corrected line is:

acl source_is_new src_get_gpc0*(thebackend)* eq 0


On Thu, Mar 10, 2011 at 7:52 PM, Cory Forsyth wrote:

> I am trying to make an haproxy config where if an IP visits the server then
> I'll track it in a stick-table and increment the gpc.
> If my capacity1 server gets marked down I want to only allow incoming
> requests from IP addresses that already appear in that stick-table.  The
> full config is shown below.
>
> I am using socat and haproxy's stats to view the contents of the
> stick-table, and I see the gpc0 counter variable getting incremented on
> every request.  But when the capacity server gets marked as down than every
> incoming request, even from IPs with positive gpc0 values, gets blocked
> immediately.
>
> Any suggestions? This is on 1.5-dev3.
>
>
> global
> maxconn 3
> ulimit-n 65536
> log 127.0.0.1 local0
> log 127.0.0.1 local1 debug
> stats socket /var/run/haproxy.stat mode 600 level operator
>
> defaults
> stats uri /stats
> mode http
> timeout connect 5000ms
> timeout client 5ms
> timeout server 5ms
>
> backend capacity
> option httpchk GET /
> server capacity1 127.0.0.1:81 check rise 1 fall 1
>
> backend thebackend
> stick-table type ip size 5 expire 30s store gpc0
> tcp-request content  track-sc1 src
> acl mark_seen sc1_inc_gpc0
> tcp-response content accept if TRUE mark_seen
> server union1 x.x.x.x:80
>
>
> frontend http_proxy
> bind *:80
> mode http
> option forwardfor
> default_backend thebackend
> acl have_capacity srv_is_up(capacity/capacity1)
> acl source_is_new src_get_gpc0 eq 0
> block if !have_capacity source_is_new
>



-- 
[image: Follow me on twitter] <http://movableink.com/twitter_pics/244/link>get
your own Movable Ink <http://movableink.com/?s=sig>

counting entries in a stick table

[Cory Forsyth]

Is it possible to count the number of entries in a stick table and take some
action based on that number?
Failing that is it possible to take some action when a stick-table gets
*full*?  Looking at the documentation it looks like the default would be to
refuse new clients when the stick-table gets full, but when I tried it what
happened was no new entries would get added to the stick-table but the
incoming request would still be allowed in.

thanks,
Cory

btw, I just have to say this is the first time I've really played around
seriously with haproxy and I really like it.  It's a cool tool.

question on incrementing and reading stick-table gpc counter

[Cory Forsyth]

I am trying to make an haproxy config where if an IP visits the server then
I'll track it in a stick-table and increment the gpc.
If my capacity1 server gets marked down I want to only allow incoming
requests from IP addresses that already appear in that stick-table.  The
full config is shown below.

I am using socat and haproxy's stats to view the contents of the
stick-table, and I see the gpc0 counter variable getting incremented on
every request.  But when the capacity server gets marked as down than every
incoming request, even from IPs with positive gpc0 values, gets blocked
immediately.

Any suggestions? This is on 1.5-dev3.


global
maxconn 3
ulimit-n 65536
log 127.0.0.1 local0
log 127.0.0.1 local1 debug
stats socket /var/run/haproxy.stat mode 600 level operator

defaults
stats uri /stats
mode http
timeout connect 5000ms
timeout client 5ms
timeout server 5ms

backend capacity
option httpchk GET /
server capacity1 127.0.0.1:81 check rise 1 fall 1

backend thebackend
stick-table type ip size 5 expire 30s store gpc0
tcp-request content  track-sc1 src
acl mark_seen sc1_inc_gpc0
tcp-response content accept if TRUE mark_seen
server union1 x.x.x.x:80


frontend http_proxy
bind *:80
mode http
option forwardfor
default_backend thebackend
acl have_capacity srv_is_up(capacity/capacity1)
acl source_is_new src_get_gpc0 eq 0
block if !have_capacity source_is_new