Thanks for response and information.
And I came across this link :
https://www.haproxy.com/blog/introduction-to-haproxy-maps/#rate-limiting-by-url-path/
for url based rate-limit.
With this exact config (copied exact config to haproxy.cfg and also created
maps/rates.map file) I made one observation that the when I start a load
towards haproxy, and if the rate for a url is say 100 requests per second.
haproxy does send only 100 requests to backend server and no more requests
are sent to backend and rest of requests are getting rejected with 429 !
And from admin socket :
root@VM-Ubuntu:/etc/haproxy# echo "show table api_gateway" | socat
unix:/var/lib/haproxy/stats stdio
# table: api_gateway, type: binary, size:1048576, used:1
0x1bbb600: key=42EE33090A25091E use=0 exp=10000 http_req_rate(10000)=17351
Any info on whats going on with this config ?
Thanks
badari
On Fri, Feb 22, 2019 at 1:03 PM Jarno Huuskonen <jarno.huusko...@uef.fi>
wrote:
> Hi,
>
> On Thu, Feb 21, Badari Prasad wrote:
> > But by replacing 'src' with 'path', rate-limiting did not work. My
> current
> > config after the change is :
> >
> > backend st_src_as2_monte
> > stick-table type string len 64 size 1m expire 1s store
> http_req_rate(1s)
>
> (for testing it helps to use longer expire eg. 60s and longer rate
> (60s). Then it's easier to use admin socket to view stick table values
> to see if the stick table is updated etc).
>
> > frontend scef
> > bind 0.0.0.0:80
> > bind 0.0.0.0:443 ssl crt /etc/ssl/private/as1.pem
> > mode http
> > option forwardfor
> >
> > http-request track-sc1 path table st_src_as2_monte
>
> You're using sc1 here.
>
> > acl monte_as2_api_url path_beg /api/v1/monitoring-event/A000002/
> > #500 requests per second.
> > acl monte_as1_exceeds_limit sc0_http_req_rate(st_src_as1_monte) gt
> 500
>
> And sc0 here, change this to sc1 (or use track-sc1).
>
> -Jarno
>
> > http-request deny deny_status 429 if monte_as2_api_url
> > monte_as2_exceeds_limit
> > use_backend nodes
> > Appreciate the response on this, and going further I will have to extend
> > the rate limiting to multiple url's .
> >
> >
> > Thanks
> > badari
> >
> >
> >
> > On Wed, Feb 20, 2019 at 11:13 PM Jarno Huuskonen <jarno.huusko...@uef.fi
> >
> > wrote:
> >
> > > Hi,
> > >
> > > On Wed, Feb 20, Badari Prasad wrote:
> > > > Thank you for responding. Came up with based on the inputs:
> > > >
> > > > #printf "as2monte" | mkpasswd --stdin --method=md5
> > > > userlist AuthUsers_MONTE_AS2
> > > > user appuser_as2 password $1$t25fZ7Oe$bjthsMcXgbCt2EJvQo8r0/
> > > >
> > > > backend st_src_as2_monte
> > > > stick-table type string len 64 size 1000 expire 1s store
> > > > http_req_rate(1s)
> > > >
> > > > frontend scef
> > > > bind 0.0.0.0:80
> > > > bind 0.0.0.0:443 ssl crt /etc/ssl/private/as1.pem
> > > > mode http
> > > > #option httpclose
> > > > option forwardfor
> > > >
> > > > acl monte_as2_api_url url_beg /api/v1/monitoring-event/A000002/
> > > > #500 requests per second.
> > > > acl monte_as2_exceeds_limit src_http_req_rate(st_src_as2_monte)
> gt
> > > 500
> > > > http-request track-sc1 src table st_src_as2_monte unless
> > > > monte_as2_exceeds_limit
> > > > http-request deny deny_status 429 if monte_as2_api_url
> > > > monte_as2_exceeds_limit
> > >
> > > I'm confused :) what your requirements are but I think with
> > > this configuration each src address can have rate 500 to
> > > /api/v1/monitoring-event/A000002/. (so with 10 different src addresses
> > > you can have 5000 rate to /api/v1/monitoring-event/A000002/).
> > >
> > > (And you're using type string stick table, type ip or ipv6 is better
> > > fit for tracking src).
> > >
> > > But if it fits your requirements then I'm glad you found a working
> > > solution.
> > >
> > > -Jarno
> > >
> > > > http-request auth realm basicauth if monte_as2_api_url
> > > > !authorized_monte_as2
> > > >
> > > > use_backend nodes
> > > >
> > > > With this config I was able to rate limit per url basis.
> > > >
> > > > Thanks
> > > > badari
> > > >
> > > >
> > > >
> > > > On Tue, Feb 19, 2019 at 10:01 PM Jarno Huuskonen <
> jarno.huusko...@uef.fi
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > On Mon, Feb 11, Badari Prasad wrote:
> > > > > > I want to rate limit based on url
> > > > > > [/api/v1/monitoring-event/A000001, /api/v1/client1/transfer_data,
> > > > > > /api/v1/client2/transfer_data ] no matter what the source ip
> > > address
> > > > > is.
> > > > >
> > > > > Something like this might help you. Unfortunately at the moment
> > > > > I don't have time to create a better example.
> > > > >
> > > > > acl api_a1 path_beg /a1
> > > > > acl api_b1 path_beg /b1
> > > > > acl rate_5 sc0_http_req_rate(test_be) gt 5
> > > > > acl rate_15 sc0_http_req_rate(test_be) gt 15
> > > > >
> > > > > # You might want to add acl so you'll only track paths
> you're
> > > > > # interested in.
> > > > > http-request track-sc0 path table test_be
> > > > > # if you want to track only /a1 /b1 part of path
> > > > > # you can use for example field converter:
> > > > > #http-request track-sc0 path,field(1,/,2) table test_be
> > > > > #http-request set-header X-Rate
> %[sc0_http_req_rate(test_be)]
> > > > >
> > > > > http-request deny deny_status 429 if api_a1 rate_5
> > > > > http-request deny deny_status 403 if api_b1 rate_15
> > > > >
> > > > > # adjust len and size etc. to your needs
> > > > > backend test_be
> > > > > stick-table type string len 40 size 20 expire 180s store
> > > > > http_req_rate(60s)
> > > > >
> > > > > -Jarno
> > > > >
> > > > > > On Mon, Feb 11, 2019 at 7:34 PM Jarno Huuskonen <
> > > jarno.huusko...@uef.fi>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > On Mon, Feb 11, Badari Prasad wrote:
> > > > > > > > Thank you for the response. I came up with my own haproxy
> > > cfg,
> > > > > where
> > > > > > > i
> > > > > > > > would want to rate limit based on event name and client id in
> > > url.
> > > > > > > > URL ex : /api/v1/<event_name>/<clientid>
> > > > > > > >
> > > > > > > > Have attached a file for my haproxy cfg. But it does not
> seems
> > > to be
> > > > > > > rate
> > > > > > > > limiting the incoming requests.
> > > > > > >
> > > > > > > > backend st_src_monte
> > > > > > > > stick-table type string size 1m expire 10s store
> > > > > http_req_rate(10s)
> > > > > > > > ...
> > > > > > > >
> > > > > > > > acl monte_as1_exceeds_limit
> > > src_http_req_rate(st_src_as1_monte)
> > > > > gt 990
> > > > > > > > acl monte_in_limit src_http_req_rate(st_src_as1_monte) lt
> 1000
> > > > > > > > http-request track-sc0 src table st_src_as1_monte
> > > > > > >
> > > > > > > There's no st_src_as1_monte table in your example config,
> there's
> > > > > > > st_src_monte table.
> > > > > > >
> > > > > > > > http-request deny deny_status 429 if { path_beg
> > > > > > > /api/v1/monitoring-event/A000001 AND monte_as1_exceeds_limit }
> > > > > > >
> > > > > > > You're tracking connections with src, but the stick table is
> type
> > > > > string,
> > > > > > > have you checked from admin socket that the stick table has
> > > entries,
> > > > > > > something like:
> > > > > > > echo 'show table st_src_monte' | nc -U /var/lib/haproxy/stats
> > > > > > > (insted of nc -U, socat stdio /var/lib/haproxy/stats should
> also
> > > work).
> > > > > > >
> > > > > > > If you want to track src ip, then stick-table type ip or ipv6
> is
> > > > > > > probably better.
> > > > > > >
> > > > > > > >> I would want to configure 1000 tps for url
> > > > > > > /api/v1/client1/transfer_data or
> > > > > > > >> 500 tps for /api/v1/client2/user_data and so on....
> > > > > > >
> > > > > > > Do you mean that only 1000 tps goes to
> > > > > > > /api/v1/client1/transfer_data (no matter what the source ip
> > > addresses
> > > > > > > are) or each source ip can send 1000 tps to
> > > > > /api/v1/client1/transfer_data ?
> > > > >
> > > > > --
> > > > > Jarno Huuskonen
> > > > >
> > >
> > > --
> > > Jarno Huuskonen
> > >
>
> --
> Jarno Huuskonen
>
