load shedding in haproxy

[Jerry Scharf]

Hi,

I am wondering if there is a way in haproxy to do this without 
generating/loading new configs (an option but ugly.)


We have multiple data centers that serve the same apps. We would like to 
be able to do the following:


Normally, traffic that lands at a data center gets processed there and 
returned. If a data center goes above some level and there is spare 
capacity at another data center, we want to ship some of the traffic to 
the other data center to balance the load. This can be seen as adding 
servers or backends of servers in the other data center to the server set.


The information about the relative load needs to be calculated by an 
external program, the set of apps is large and complex. Simple timing 
can't make the right decision.


I realize this is an odd case, but we are looking for a more dynamic way 
to balance traffic than what we have now.


thanks in advance,
jerry

--
Soundhound Devops
"What could possibly go wrong?"

Re: Queries Rearding to the Redirections According to the ports

[Bryan Talbot]

> On Jan 31, 2017, at Jan 31, 11:26 PM, parag bharne 
>  wrote:
> 
> HI,
> Here our scenario where we wnat to work using haproxy
> 
> (client) -> http://www.example.com  -> (redirect) -> 
> https://www.example.com 
> (client) -> http://www.example.com:8080  -> 
> (redirect) ->
> https://www.example.com :8080
> 
> This is Possible in haproxy or not, plz try to reply as fast as possible
> 

Yes.



> Parag Bharne

Queries Rearding to the Redirections According to the ports

[parag bharne]

HI,
Here our scenario where we wnat to work using haproxy

(client) -> http://www.example.com -> (redirect) -> https://www.example.com
(client) -> http://www.example.com:8080 -> (redirect) ->
https://www.example.com:8080

This is Possible in haproxy or not, plz try to reply as fast as possible

Parag Bharne

Start From Zero concept

[Thilina Manamgoda]

Hi,

I am Thilina Manamgoda, an undergraduate of Faculty of Engineering,
University of Peradeniya, Sri Lanka.  What I meant by "Start from Zero" is
start a server from stop state to running state when the first request
comes. This functionality  is needed in the Server-less architecture
concept where request is served in that way.

Currently I am working with a *Kubernetes Cluster* where the servers are
deployed as pods. What I am trying to do is when the first request comes
for a server which is in stop state, rest call should be made to a service
which will start the server.
 May be this is not a
functionality that is relevant at the moment for the project  but I am
trying to Implement it and all suggestions are welcome.


regards,
Thilina Manamgoda

Re: [PATCH] MEDIUM/RFC: Implement time-based server latency metrics

[Krishna Kumar (Engineering)]

Hi Willy,

Thanks for your detailed mail. I will get back to you very soon on this.

Regards,
- Krishna


On Tue, Jan 31, 2017 at 12:54 AM, Willy Tarreau  wrote:

> Hi Krishna,
>
> back on earth ;-)
>
> On Tue, Jan 03, 2017 at 03:07:26PM +0530, Krishna Kumar (Engineering)
> wrote:
> > I explored your suggestion of "hard-coded periods", and have some
> > problems: code complexity seems to be very high at updates (as well
> > as retrievals possibly); and I may not be able to get accurate results.
> > E.g. I have data for 1, 4, 16 seconds; and at 18 seconds, a request is
> > made for retrieval of the last 16 seconds (or 1,4,16). At this time I
> have
> > values for last 18 seconds not 16 seconds. I explored using timers to
> > cascade (will not work as it may run into races with the setters, and
> > also adds too much overhead) vs doing this synchronously when the
> > event happens. Both are complicated and have the above issue of not
> > able to get accurate information depending on when the request is
> > made.
>
> The principle is quite simple but requires a small explanation of how our
> frequency counters manage to report such accurate values first.
>
> We're counting events that happen over a period, represented by a cross
> over the time line below :
>
> X X  XX  XX  X X X X   X
>   |> t
>
> In order to be able to measure an approximately accurate frequency without
> storing too much data, we'll report an average over a period. The problem
> is, if we only report a past average, we'll get slowly changing data and
> in particular it would not be usable to detect quick changes such as a high
> rate happening over the last second. Thus we always keep two measures, the
> one of the previous period, and the one of the current period.
>
> X X  XX  XX  X X X X   X
>   |-|--> t
>   <--- previous --->|<--- current --->
>count=6count=5
>
> And using this we implement a pseudo sliding window. With a true sliding
> window, you would count events between two points that are always exactly
> the same distance apart, which requires to memorize all of them. With
> this pseudo sliding window, instead, we make use of the fact that events
> follow an approximately homogenous distribution over each period and have
> approximately the same frequency over both periods. Thus the mechanism
> becomes more of a "fading" window in fact : we consider one part of the
> previous window that we sum with the current one. The closer the current
> date is from the end of the current window, the least we count on the
> previous window. See below, I've represented 5 different sliding windows
> with the ratio of the previous window that we consider as still valid
> marked with stars and the ratio of the old period at the end of the line :
>
> X X  XX  XX  X X X X   X
>   |-|> t
> |***--|  80%
>   |*|  60%
> |***--|  40%
>   |*|  20%
> |-|  0%
>
> Thus the measure exactly is current + (1-relative_position) * previous.
> Once the period is over, the "current" value overwrites the "previous"
> one, and is cleared again, waiting for new events. This is done during
> insertion and possibly during value retrieval if it's noticed that the
> period has switched since last time.
>
>
> In fact this mechanism can be repeated over multiple periods if needed,
> the principle must always be the same :
>
> X X  XX  XX  X X X X   X
>   |||||--> t
>   old  N-3 N-2   N-1current
>
>
> The "old" period is the one on which we apply the fading effect. The
> current period is the one being entirely accounted (since it's being
> filled as we're looking at it) so here you could always keep a few
> past values (rotating copies of "current") and keep the last one for
> the fade out. In the simplified case above we just don't have the
> N-1..N-M, we only have "current" and "old".
>
> Thus as you can see here, the "current" period is the only dynamic one,
> which is the exact sum of events over the last period. But now what if
> we wanted to implement this with multiple levels ? It's very easy, the
> "current" period being the sum of events, it simply is the sum of the
> lower layers when you cascade multiple counters, thus it's the lower
> layer's sum of [current] + [old*ratio] + [n-1] + [n-2] +...+ [n-m].
>
> Thus let's have an example with periods of 1s, 2s, 4s and 8s :
>
>   previous

Re: [PATCH] BUG/MAJOR: dns: create one client UDP socket per process

[Willy Tarreau]

On Tue, Jan 31, 2017 at 01:09:15PM +0100, Baptiste wrote:
> Understood. Let's use this Sunday as a dead line.

Perfect, that works for me.

> I think I can make dns_init_resolvers()  to take an argument 'int
> close_socket' whose value could be 0 or 1.
> If 1, then HAProxy will try to close the socket before re-opening it.
> if 0, then it simply tries to open the socket.

I think that might be the best way to do it, doesn't require many
changes and leaves the code reasonably understandable.

Thanks!
Willy

Re: How to password protect a subdirectory

[Jarno Huuskonen]

Hi,

On Tue, Jan 31, Artur wrote:
> Hello,
> 
> I'm currently serving public static content from a webserver behind haproxy.
> What could be the right way to password protect only a single
> subdirectory (and all its content) with haproxy ?
> / -> Public
> /directory/private -> Password protected

You could try with something like this:

userlist testlist
user test insecure-password test

your listen/frontend:
acl need_auth path_beg /directory/private
acl auth_ok http_auth(testlist)

http-request auth if need_auth !auth_ok

For more information check userlists / auth docs:
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#3.4
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#7.3.6-http_auth
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-http-request

-Jarno

-- 
Jarno Huuskonen

Re: Gzip compression and transfer: chunked

[Christopher Faulet]

Hi Vladimir,

Sorry for my late reply, I was pretty busy these last days. I 
investigated a little on your problem. I've done some tests and 
carefully read the code. Everything seems to work as expected. I was not 
able to reproduce what you experienced with HAProxy 1.7.2.


First, in HAProxy, except with a specific configuration, we never remove 
any "Transfer-Encoding" headers. And we never add any "Content-Length" 
headers. During the HTTP headers parsing, if the "Transfer-Encoding" 
header is found with the value "chunked", we remove all "Content-Length" 
headers, if any. This is always done, with or without the compression 
filter. Then, when the compression is enabled, if the response payload 
must be compressed by HAProxy, we remove all "Content-Length" headers 
and add "Transfer-Encoding: chunked" if needed.


Then, when the response is truncated, there is no "Content-Encoding" 
header. So I'm tempted to think that the GZIP compression is not used on 
the response.


So, if there is a bug, it is well hidden (as always with chunked HTTP 
payload ...*sigh*...). And it will be hard for me to hunt it without 
more information about exchanges between HAProxy and your backend. The 
best would be a full-packet network capture. But, if the bug is 
reproducible, it can be a good start to have the output of the following 
command:


 curl -H header_1 -H header_2 ... -v --raw --compress -o raw_response 
your_backend_url


Be sure to set same headers than for a request on HAProxy.

If the response contains sensitive information, you can remove them. I 
only need to have the chunk sizes.


--
Christopher

How to password protect a subdirectory

[Artur]

Hello,

I'm currently serving public static content from a webserver behind haproxy.
What could be the right way to password protect only a single
subdirectory (and all its content) with haproxy ?
/ -> Public
/directory/private -> Password protected

-- 
Best regards,
Artur

Re: [PATCH] BUG/MAJOR: dns: create one client UDP socket per process

[Baptiste]

On Tue, Jan 31, 2017 at 2:14 AM, Willy Tarreau  wrote:

> Hi Baptiste,
>
> On Mon, Jan 30, 2017 at 11:07:53PM +0100, Baptiste wrote:
> > Hi all,
> >
> > Please find attached a patch to fix the issue reported by Joshua on the
> ML
> > and sjiveson on discourse.
> > I moved the initialisation of the dns_resolvers() after the fork. I can
> > confirm now than each process has its own UDP socket to send DNS
> requests.
>
> Thanks for doing this. I think that for stable versions it's perfect.
> However calling dns_init_resolvers() after the fork also means we've
> closed the stderr and detached the processes so we'll never get the
> possible errors. I've checked and the function above deals with two
> types of errors :
>
>   - memory allocation issues (OK, rare enough)
>
>   - network issues (eg: missing route causing connect() to fail)
> This last one is more problematic because it will cause all the
> processes to silently quit just after forking when the init
> scripts says "OK", which is pretty bad.
>
> I'd really like that for the mid-term we can have a different method,
> consisting in this :
>
>   - running dns_init_resolvers() as it is today, before the fork(). This
> way we're certain that the config is OK.
>
>   - just after the fork(), simply walk over all resolvers, close and
> reopen the sockets (thus perform a small part of dns_init_resolvers()
> again) so that each process has its own socket. We're doing this
> already for epoll.
>
> Alternatively, we can decide that dns_init_resolvers() creates enough
> sockets for all processes and keeps them available, then closes the
> unneeded ones after the fork, but that's in fact trickier in my opinion.
>
> If you see how to implement something like what is described above, I
> can wait a bit more for a better fix. If you're facing difficulties,
> I can merge it now as a temporary fix. Just let me know.
>
> Thanks!
> Willy
>


Understood. Let's use this Sunday as a dead line.

I think I can make dns_init_resolvers()  to take an argument 'int
close_socket' whose value could be 0 or 1.
If 1, then HAProxy will try to close the socket before re-opening it.
if 0, then it simply tries to open the socket.

Until this new patch is available, people needing DNS and multiprocess can
use the first patch.

Baptiste