Re: resolvers hold valid clarification

[Michał Pasierb]

I dug deeper in the code and found out that server health checks also call
dns resolution. The hold.valid was used before the change at
https://github.com/haproxy/haproxy/commit/f50e1ac4442be41ed8b9b7372310d1d068b85b33
to trigger the resolution. After the change it is triggered no more. So the
resolution if driven only by timeout resolve now. In other words hold.valid
has not much meaning.

When a server is in layer 4 connection error there are tasks inserted into
scheduler queue which when executed never do any job after the change. This
is suboptimal but I guess are not many installations with servers in this
state for a long time.

Can anyone confirm my understating of this setting ?

śr., 22 wrz 2021 o 19:52 Michał Pasierb  napisał(a):

> I would like to clarify how *hold valid* is used by resolvers. I have
> this configuration:
>
> resolvers mydns
>   nameserver dns1 192.168.122.202:53
>   accepted_payload_size 8192
>
>   timeout resolve 5s
>   timeout retry   2s
>   resolve_retries 3
>
>   hold other  30s
>   hold refused120s
>   hold nx 30s
>   hold timeout10s
>   hold valid  1h
>   hold obsolete   0s
>
> The *valid* setting is a bit confusing. I can not find good explanation
> of it in the documentation. From what I see in the code of version 2.0.25,
> it is used only when doing DNS queries in tcp/http path:
>
> http-request do-resolve(txn.myip,mydns,ipv4) str(services.example.com)
>
> only when requests arrive in parallel, I see less queries to DNS servers
> than http requests. When requests are done in sequence, I see the same
> count of DNS requests as http requests. For example when I send 3000
> requests to HAProxy with 3 clients in parallel, there are about 2600
> requests to DNS servers.
>
> So it doesn't look like a proper cache to me. Whole HAProxy becomes
> unusable 10 seconds (hold timeout) after DNS servers stop responding
> because every server which is using DNS SRV record is put to maintenance
> state due to resolution error.
>
> Is this proper assessment of current state and is this what was intended ?
>
> Regards,
> Michal
>

resolvers hold valid clarification

[Michał Pasierb]

I would like to clarify how *hold valid* is used by resolvers. I have this
configuration:

resolvers mydns
  nameserver dns1 192.168.122.202:53
  accepted_payload_size 8192

  timeout resolve 5s
  timeout retry   2s
  resolve_retries 3

  hold other  30s
  hold refused120s
  hold nx 30s
  hold timeout10s
  hold valid  1h
  hold obsolete   0s

The *valid* setting is a bit confusing. I can not find good explanation of
it in the documentation. From what I see in the code of version 2.0.25, it
is used only when doing DNS queries in tcp/http path:

http-request do-resolve(txn.myip,mydns,ipv4) str(services.example.com)

only when requests arrive in parallel, I see less queries to DNS servers
than http requests. When requests are done in sequence, I see the same
count of DNS requests as http requests. For example when I send 3000
requests to HAProxy with 3 clients in parallel, there are about 2600
requests to DNS servers.

So it doesn't look like a proper cache to me. Whole HAProxy becomes
unusable 10 seconds (hold timeout) after DNS servers stop responding
because every server which is using DNS SRV record is put to maintenance
state due to resolution error.

Is this proper assessment of current state and is this what was intended ?

Regards,
Michal

no log on http-keep-alive race

[Michał Pasierb]

Hello,

I have situation where HAProxy receives a request but doesn't log it. I
would like to know if this is desired behaviour or this is an actual bug.

To replicate this issue:
1) create HAProxy config which doesn't disable http-keep-alive
2) on backend server set http-keep-alive timeout to a low value of 2 seconds
3) run a script which sends requests in a loop and waits say 1.996 seconds
between each iteration

>From time to time, there will be no response from HAProxy. This is
understandable. Connecting directly to backend server, without using
HAProxy, gives same result. This is all desired.

However, there is no log indicating that HAProxy was processing a request
and that it failed on server side.

This issue has already been discussed at
https://discourse.haproxy.org/t/haproxy-1-7-11-intermittently-closes-connection-sends-empty-response-on-post-requests/3052/5

I understand this configuration is clearly wrong because the timeout on
server should be higher than on HAProxy. In this scenario, I'm responsible
for HAProxy configuration and somebody else is responsible for
configuration of servers in backends. I can not know if every server is
correctly configured therefore I would like to enable logging of such
requests.

I have tried version 1.7, 1.8 and 2.0 with different options like logasap.
All behave the same.

Regards,
Michael

Re: confused by HAProxy log line

[Michał Pasierb]

Hello,

I did not mention it but all servers in c_backend have a httpchk
configured. There is nothing in the HAProxy logs indicating the servers or
the backend was not available during the time of the request. Indeed the
stats page shows only 4 health checks did not pass on each server since
HAProxy started and I know they were triggered by application deployment to
the servers. This is rock solid. The servers in the backend handle
thousands of requests during a day and log lines like these occur about 50
times a day.

I tried to replicate the termination state in my lab but I failed. I got
only CC--, CR-- and CD-- and http status is always set correctly (not to
-1). The origin log line has CH-- which indicates HAProxy was waiting for
reponse headers from a server so (this implies?) it connected somewhere. So
why server is marked as  ?

Regards,
Michal

On Thu, Oct 11, 2018 at 1:00 PM Aleksandar Lazic  wrote:

> Am 11.10.2018 um 10:33 schrieb Michał Pasierb:
> > Hello,
> >
> > I have a problem understanding a particular log line from HAProxy 1.7.11
> in
> > production system. My clients report problems from time to time. They
> make
> > another request and all is OK. This is the log format used:
> >
> > log-format %tr\ %ci:%cp\ %ft\ %b/%s\ %TR/%Tw/%Tc/%Tr/%Ta\ %ST\ %B\ %CC\
> %CS\
> > %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r\ %ID\ %U\
> %[res.comp]
> >
> > and the problematic log line:
> >
> > 02/Oct/2018:09:55:14.997 :46007 http-in c_backend/
> > 44/-1/-1/-1/44 -1 25 - - CHNN 143/36/2/0/3 0/0 {} {} "PUT /api/xyz/status
> > HTTP/1.1"  764 0
> >
> > I can see that c_backend was properly selected based on acls. I can see
> that
> > client sent 764 bytes to HAProxy. The termination code is CH - client
> aborted
> > connection while HAProxy was waiting for headers from server. What
> server ?
> > There is  and connection time of -1. There were 3 connection
> retries.
> > HAProxy got 25 bytes from a server. The config contains:
> >
> > defaults
> >   retries   3
> >   optionredispatch
> >
> > Yet the retries is not +3 so it seems the redispatch did not take place.
> >
> > This is all confusing evidence. Can you explain what really happened ?
>
> None of the c_backend servers are reachable.
> I assume that the -1 value is the status_code.
>
> Cite from
> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#8.2.3
>
> ###
>
>  - "status_code" is the HTTP status code returned to the client. This
> status
> is generally set by the server, but it might also be set by haproxy
> when
> the server cannot be reached or when its response is blocked by
> haproxy.
>
> ...
>
>   - "server_name" is the name of the last server to which the connection
> was
> sent, which might differ from the first one if there were connection
> errors
> and a redispatch occurred. Note that this server belongs to the backend
> which processed the request. If the request was aborted before
> reaching a
> server, "" is indicated instead of a server name. If the
> request was
> intercepted by the stats subsystem, "" is indicated instead.
> ###
>
> I suggest to use some checks for the c_backend for example
>
>
> https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-option%20httpchk
>
> > Thanks,
> > Michal
>
> Best regards
> Aleks
>

confused by HAProxy log line

[Michał Pasierb]

Hello,

I have a problem understanding a particular log line from HAProxy 1.7.11 in
production system. My clients report problems from time to time. They make
another request and all is OK. This is the log format used:

log-format %tr\ %ci:%cp\ %ft\ %b/%s\ %TR/%Tw/%Tc/%Tr/%Ta\ %ST\ %B\ %CC\
%CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r\ %ID\ %U\
%[res.comp]

and the problematic log line:

02/Oct/2018:09:55:14.997 :46007 http-in c_backend/
44/-1/-1/-1/44 -1 25 - - CHNN 143/36/2/0/3 0/0 {} {} "PUT /api/xyz/status
HTTP/1.1"  764 0

I can see that c_backend was properly selected based on acls. I can see
that client sent 764 bytes to HAProxy. The termination code is CH - client
aborted connection while HAProxy was waiting for headers from server. What
server ? There is  and connection time of -1. There were 3
connection retries. HAProxy got 25 bytes from a server. The config contains:

defaults
  retries   3
  optionredispatch

Yet the retries is not +3 so it seems the redispatch did not take place.

This is all confusing evidence. Can you explain what really happened ?

Thanks,
Michal

gzip compression and logging response size

[Michał Pasierb]

Hi,

I would like to log size of response before and after compression. Is it
possible ?
I see this question was asked before at
http://thread.gmane.org/gmane.comp.web.haproxy/10008 without any resolution.

Regards,
Michal

regression in 1.6.13 - wrong http status in logs and stats

[Michał Pasierb]

Hi,

I would like to report a regression in HAProxy 1.6.13 after upgrading from
1.6.9 in production :(

Reproduce with config:

---
global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  maxconn 4096
  chroot /usr/share/haproxy
  uid 99
  gid 99
  daemon

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option redispatch
  stats enable
  stats refresh 5s
  timeout connect 5s
  timeout client  30s
  timeout server  30s
  balance roundrobin
  option forwardfor

listen Statistics
  bind 192.168.122.202:
  mode http
  stats refresh 5s
  stats uri /

frontend http-in
  bind 192.168.122.202:9000

  acl is_root_path  path /
  redirect location /my_custom_page if is_root_path

backend nomatch
  redirect location http://www.google.com
---

Send a request with curl -v 192.168.122.202:9000/ - response is OK and has
status code 302 but logs and statistics have 503 instead:

192.168.122.202:37880 [12/Jul/2017:15:54:49.573] http-in http-in/
0/-1/-1/-1/2 503 305 - - LR-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"

git bisect shows this commit to be the culprit:

commit b12d699543adb84fa543297d12b64fce7ec94803
Author: Christopher Faulet 
Date:   Tue Mar 28 11:51:33 2017 +0200

BUG/MINOR: http: Fix conditions to clean up a txn and to handle the
next request

I also tested 1.7.8 and 1.8-dev2 - they are OK. So it seems it is a
backport issue.

Regards,
Michal

Re: Propagating agent-check weight change to tracking servers

[Michał]

Hi,
I didn't want to continue this discussion, but there is one thing that's
totally not true.

2017-04-13 13:50 GMT+02:00 Frederic Lecaille <flecai...@haproxy.com>:

> Hello Michal,
>
> On 04/11/2017 04:41 PM, Michał wrote:
>
>> Hello Willy,
>>
>> So I'm fighting with dba97077 made by Frédéric Lécaille - it broke many
>> things.
>>
>
> This patch broke haproxy non-transparent builds. Thanks to Steven
> Davidovitz, Pavlos Parissis and David Carlier for having promptly helped in
> fixing this.
>
> Excepted this building issue, AFAIK this patch may break only server  or
> default server 'source' setting parsing. How much other things did it break?
>
> Do not forgive that as far as your configuration files do not use any
> 'default-server' line with new supported settings (I agree they are
> numerous), they should be parsed as before.
>
>
That's the case. My config don't use this functionality and the parsing
broke.
That's why I got so angry, because breaking existing configs it's not best
practice IMHO.


> So, if you want to have more chances not to be impacted by my
> "default-server patches", please do not use any 'default-server' new
> supported setting as a first workaround, or no 'default-server' line at all
> as a second workaround. This should help you to continue and develop your
> features.
>
> E.g. you can't use "source", because that patch broke it. I'm curious how
>> many other stuff got broken with those patches around default-server.
>>
>>
> [snipped rude/useless/unproductive published anonymous comments,
> especially towards my colleagues]
>
>
>> In attachment there is patch fixed after your (Willy) review. Sorry for
>> loop,
>> I was focused on fixing all those problems with Frédérics patch I just
>> didn't
>> think how to replace do..while (which obviously works) with this cleaner
>> version - thanks for this. The patch got even simpler.
>>
>
> As Willy wrote before, please feel free to contact me to fix asap any
> issue I alone am responsible. I am keeping an eye on HAproxy ML, and I
> would still be glad to help you and consequently any other haproxy users.
>
> Regards,
> Fred.
>
>

Re: Propagating agent-check weight change to tracking servers

[Michał]

gt; server s1 127.0.0.1:8001 check inter 1s
>
>   backend customer1
> hash-type consistent
> server s1 127.0.0.1:8001 track servers/s1
>
>   backend customer2
> hash-type consistent
> server s1 127.0.0.1:8001 track servers/s1
>
>   backend customer3
> hash-type consistent
> server s1 127.0.0.1:8001 track servers/s1
>
> This one will always fail to use non-integral weigths because of
> the farm receiving the order which itself doesn't use a dynamic
> algorithm, and will not be propagated. We could decide that we can
> ignore the problem, but then it will render this one inconsistent :
>
>   defaults
> balance uri
>
>   backend servers
> server s1 127.0.0.1:8001 check inter 1s
>
>   backend customer1
> server s1 127.0.0.1:8001 track servers/s1
>
>   backend customer2
> hash-type consistent
> server s1 127.0.0.1:8001 track customer2/s1
>
> where one could expect that a dynamic backend tracking a static one
> would always stay in sync with it, but that would not be true anymore.
>
> So for now I think this requires more thinking, probable a specific server
> option like "track-weight" or something like this to let the user decide
> whether or not the weights should be inherited from tracked servers. This
> would limit the risk of very bad surprizes.
>

Probably yes, we need another option for tracking weights and that will fix
issues with those unpredictable variants.

Regards,
> Willy
>



Please don't mess allegrogroup here, I'm posting those patches from my
private
e-mail.

Michał

Re: Propagating agent-check weight change to tracking servers

[Michał]

Hello Willy,

So I'm fighting with dba97077 made by Frédéric Lécaille - it broke many
things.
E.g. you can't use "source", because that patch broke it. I'm curious how
many other stuff got broken with those patches around default-server.

We need some CI (even if they will only build haproxy) and IMHO people with
@haproxy.com mails should test their code before posting and merging :(

In attachment there is patch fixed after your (Willy) review. Sorry for
loop,
I was focused on fixing all those problems with Frédérics patch I just
didn't
think how to replace do..while (which obviously works) with this cleaner
version - thanks for this. The patch got even simpler.

Thanks again,
Michał

2017-03-27 15:34 GMT+02:00 Willy Tarreau <w...@1wt.eu>:

> Hi Michal,
>
> On Mon, Mar 27, 2017 at 03:18:10PM +0200, Michal wrote:
> > Hello,
> > Thank you for your response. I agree with part about backward
> compatibility
> > and of course I don't want to break working things
> >
> > I prepared patch with described functionality and with two notes in doc
> to
> > let users know about this behaviour.
>
> Thanks. Just a few points :
>
> 1) cosmetic cleanups below :
>
> > diff --git a/doc/configuration.txt b/doc/configuration.txt
> > index fb3e691..7f22782 100644
> > --- a/doc/configuration.txt
> > +++ b/doc/configuration.txt
> > @@ -11370,6 +11370,10 @@ track [/]
> >enabled. If  is omitted the current one is used. If
> disable-on-404 is
> >used, it has to be enabled on both proxies.
> >
> > +  Note:
> > +Relative weight changes are propagated to all tracking servers. Each
> > +tracking server will have it's weight recalculated separately.
>
> s/it's/its/
>
> > +
> >Supported in default-server: No
> >
> >  verify [none|required]
> > diff --git a/doc/management.txt b/doc/management.txt
> > index 1d34f84..3f3730a 100644
> > --- a/doc/management.txt
> > +++ b/doc/management.txt
> > @@ -1694,6 +1694,10 @@ set weight / [%]
> >"admin". Both the backend and the server may be specified either by
> their
> >name or by their numeric ID, prefixed with a sharp ('#').
> >
> > +  Note:
> > +Relative weight changes are propagated to all tracking servers. Each
> > +tracking server will have it's weight recalculated separately.
>
> s/it's/its/
>
> > +
> >  show cli sockets
> >List CLI sockets. The output format is composed of 3 fields separated
> by
> >spaces. The first field is the socket address, it can be a unix
> socket, a
> > diff --git a/src/server.c b/src/server.c
> > index 5589723..9462a16 100644
> > --- a/src/server.c
> > +++ b/src/server.c
> > @@ -45,6 +45,9 @@
> >  static void srv_update_state(struct server *srv, int version, char
> **params);
> >  static int srv_apply_lastaddr(struct server *srv, int *err_code);
> >
> > +const char *server_propagate_weight_change_request(struct server *sv,
> > + const char *weight_str);
> > +
> >  /* List head of all known server keywords */
> >  static struct srv_kw_list srv_keywords = {
> >   .list = LIST_HEAD_INIT(srv_keywords.list)
> > @@ -912,6 +915,8 @@ const char *server_parse_weight_change_request(struct
> server *sv,
> >   struct proxy *px;
> >   long int w;
> >   char *end;
> > + const char *msg;
> > + int relative = 0;
> >
> >   px = sv->proxy;
> >
> > @@ -933,6 +938,8 @@ const char *server_parse_weight_change_request(struct
> server *sv,
> >   w = sv->iweight * w / 100;
> >   if (w > 256)
> >   w = 256;
> > +
> > + relative = 1;
> >   }
> >   else if (w < 0 || w > 256)
> >   return "Absolute weight can only be between 0 and 256
> inclusive.\n";
> > @@ -945,6 +952,34 @@ const char *server_parse_weight_change_request(struct
> server *sv,
> >   sv->uweight = w;
> >   server_recalc_eweight(sv);
> >
> > + if (relative) {
> > + msg = server_propagate_weight_change_request(sv,
> weight_str);
> > + if (msg != NULL) {
> > + return msg;
> > + }
> > + }
> > +
> > + return NULL;
> > +}
> > +
> > +const char *server_propagate_weight_change_request(struct server *sv,
> > + const char *weight_str)
> > +{
> > + if (sv->trackers == NULL)
> > + return NULL;
> > +
> > + struct serv

Re: Propagating agent-check weight change to tracking servers

[Michał]

Hello!
Any news in this topic? Is there anything wrong with my patch?

Michał

2017-02-04 9:38 GMT+01:00 Michał <creating@gmail.com>:

> Hi,
> I checked it and during synthetic tests it worked. I use same
> mechanism as origin agent-check, so it's ready to merge.
>
> It doesn't need to be backported.
>
> 2017-01-27 15:38 GMT+01:00 Michał <creating@gmail.com>:
>
>> Hello,
>>
>> So here's patch, which includes all functionalities I think about.
>> It propagates the response for every tracking server without changing it
>> and without intercepting it. In my opinion we should propagate relative
>> and absolute weights, because if you use weight=0 server's to offload
>> checks then you can send relative weight change and 0 will stay where it
>> is.
>>
>> Regards,
>> Michał
>>
>>
>> 2017-01-20 10:54 GMT+01:00 Willy Tarreau <w...@1wt.eu>:
>>
>>> Hi Michal,
>>>
>>> On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
>>> > Hello,
>>> >
>>> > We use track's in haproxy to minimize check traffic in some situations
>>> and
>>> > after my last patch we are probably going to switch to agent-checks for
>>> > live management of weights and statuses. One problem I see now - track
>>> > don't propagate weight setting to trackers, so if we set agent-check on
>>> > track we can manage status only.
>>> >
>>> > My first PoC solution works good, so I thought about introducing
>>> something
>>> > like agent-track or track-agent directive set on backends (or maybe it
>>> > should be default, non-configurable behaviour) to propagate agent-check
>>> > responses from main check to all tracking backends. Both default
>>> behaviour
>>> > and directive approach are small changes in code, but a little bigger
>>> in
>>> > functionality.
>>> >
>>> > In my opinion if we set agent-check on backend which is tracked by
>>> others -
>>> > it should propagate agent-check weight response to those tracking
>>> backend
>>> > and set weight on them too. Configurable or not - it will be good
>>> feature.
>>>
>>> I think we at least propagate the DRAIN state which is equivalent to
>>> weight == 0. If so I too think we should propagate *relative* weights.
>>> Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
>>> absolute weight (eg: 10, 20). If you have two farms configured like this
>>> :
>>>
>>>backend farm1
>>>  server new1 1.1.1.1:8000 weight 10 agent-check
>>>  server new2 1.1.1.2:8000 weight 10 agent-check
>>>
>>>backend farm2
>>>  server new1 1.1.1.1:8000 weight 20 track farm1/new1
>>>  server new2 1.1.1.2:8000 weight 20 track farm1/new2
>>>  server old1 1.1.1.3:8000 weight 10 check
>>>  server old2 1.1.1.4:8000 weight 10 check
>>>
>>> Then you want the weight changes on farm1 to be applied proportionally
>>> to farm2 (ie: a ratio of the configured absolute weight, which is iweight
>>> IIRC).
>>>
>>> Otherwise that sounds quite reasonable to me given that the agent-check's
>>> purpose is to provide a more accurate vision of the server's health, and
>>> that tracking is made to share the same vision across multiple farms.
>>>
>>> Regards,
>>> Willy
>>>
>>
>>
>>
>

Re: Propagating agent-check weight change to tracking servers

[Michał]

Hi,
I checked it and during synthetic tests it worked. I use same
mechanism as origin agent-check, so it's ready to merge.

It doesn't need to be backported.

2017-01-27 15:38 GMT+01:00 Michał <creating@gmail.com>:

> Hello,
>
> So here's patch, which includes all functionalities I think about.
> It propagates the response for every tracking server without changing it
> and without intercepting it. In my opinion we should propagate relative
> and absolute weights, because if you use weight=0 server's to offload
> checks then you can send relative weight change and 0 will stay where it
> is.
>
> Regards,
> Michał
>
>
> 2017-01-20 10:54 GMT+01:00 Willy Tarreau <w...@1wt.eu>:
>
>> Hi Michal,
>>
>> On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
>> > Hello,
>> >
>> > We use track's in haproxy to minimize check traffic in some situations
>> and
>> > after my last patch we are probably going to switch to agent-checks for
>> > live management of weights and statuses. One problem I see now - track
>> > don't propagate weight setting to trackers, so if we set agent-check on
>> > track we can manage status only.
>> >
>> > My first PoC solution works good, so I thought about introducing
>> something
>> > like agent-track or track-agent directive set on backends (or maybe it
>> > should be default, non-configurable behaviour) to propagate agent-check
>> > responses from main check to all tracking backends. Both default
>> behaviour
>> > and directive approach are small changes in code, but a little bigger in
>> > functionality.
>> >
>> > In my opinion if we set agent-check on backend which is tracked by
>> others -
>> > it should propagate agent-check weight response to those tracking
>> backend
>> > and set weight on them too. Configurable or not - it will be good
>> feature.
>>
>> I think we at least propagate the DRAIN state which is equivalent to
>> weight == 0. If so I too think we should propagate *relative* weights.
>> Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
>> absolute weight (eg: 10, 20). If you have two farms configured like this :
>>
>>backend farm1
>>  server new1 1.1.1.1:8000 weight 10 agent-check
>>  server new2 1.1.1.2:8000 weight 10 agent-check
>>
>>backend farm2
>>  server new1 1.1.1.1:8000 weight 20 track farm1/new1
>>  server new2 1.1.1.2:8000 weight 20 track farm1/new2
>>  server old1 1.1.1.3:8000 weight 10 check
>>  server old2 1.1.1.4:8000 weight 10 check
>>
>> Then you want the weight changes on farm1 to be applied proportionally
>> to farm2 (ie: a ratio of the configured absolute weight, which is iweight
>> IIRC).
>>
>> Otherwise that sounds quite reasonable to me given that the agent-check's
>> purpose is to provide a more accurate vision of the server's health, and
>> that tracking is made to share the same vision across multiple farms.
>>
>> Regards,
>> Willy
>>
>
>
>

Re: Propagating agent-check weight change to tracking servers

[Michał]

Hello,

So here's patch, which includes all functionalities I think about.
It propagates the response for every tracking server without changing it
and without intercepting it. In my opinion we should propagate relative
and absolute weights, because if you use weight=0 server's to offload
checks then you can send relative weight change and 0 will stay where it is.

Regards,
Michał


2017-01-20 10:54 GMT+01:00 Willy Tarreau <w...@1wt.eu>:

> Hi Michal,
>
> On Thu, Jan 19, 2017 at 11:45:57PM +0100, Micha?? wrote:
> > Hello,
> >
> > We use track's in haproxy to minimize check traffic in some situations
> and
> > after my last patch we are probably going to switch to agent-checks for
> > live management of weights and statuses. One problem I see now - track
> > don't propagate weight setting to trackers, so if we set agent-check on
> > track we can manage status only.
> >
> > My first PoC solution works good, so I thought about introducing
> something
> > like agent-track or track-agent directive set on backends (or maybe it
> > should be default, non-configurable behaviour) to propagate agent-check
> > responses from main check to all tracking backends. Both default
> behaviour
> > and directive approach are small changes in code, but a little bigger in
> > functionality.
> >
> > In my opinion if we set agent-check on backend which is tracked by
> others -
> > it should propagate agent-check weight response to those tracking backend
> > and set weight on them too. Configurable or not - it will be good
> feature.
>
> I think we at least propagate the DRAIN state which is equivalent to
> weight == 0. If so I too think we should propagate *relative* weights.
> Agent-checks can return a relative weight (eg: 50%, 100%, 150%) or an
> absolute weight (eg: 10, 20). If you have two farms configured like this :
>
>backend farm1
>  server new1 1.1.1.1:8000 weight 10 agent-check
>  server new2 1.1.1.2:8000 weight 10 agent-check
>
>backend farm2
>  server new1 1.1.1.1:8000 weight 20 track farm1/new1
>  server new2 1.1.1.2:8000 weight 20 track farm1/new2
>  server old1 1.1.1.3:8000 weight 10 check
>  server old2 1.1.1.4:8000 weight 10 check
>
> Then you want the weight changes on farm1 to be applied proportionally
> to farm2 (ie: a ratio of the configured absolute weight, which is iweight
> IIRC).
>
> Otherwise that sounds quite reasonable to me given that the agent-check's
> purpose is to provide a more accurate vision of the server's health, and
> that tracking is made to share the same vision across multiple farms.
>
> Regards,
> Willy
>


0001-MINOR-checks-propagate-agent-check-weight-to-tracker.patch
Description: Binary data

Re: Dynamically adding server's to backend

[Michał]

For the first iteration I just dynamically added server to proxy and then
called
check_config_validity() which resolves all track's and checks things.
IMHO check_config_validity() need some refactoring, because in current state
it not only check config, but also resolves some dependencies and state, so
it
can't be called repeatedly. It can be done in two ways - move resolving of
all
those thing to separate function(s) or place additional checks to prevent
overwrtiting data that is already in memory.

Now after you wrote about possible problems I'll look into them and try to
find
ways to fix them. I'll let you know with some patches if I will find some
good
way to do it. The more people looking into it the more ways to find better
solution :)

Michał

2017-01-20 11:02 GMT+01:00 Willy Tarreau <w...@1wt.eu>:

> On Thu, Jan 19, 2017 at 11:54:32PM +0100, Micha?? wrote:
> > Hello,
> >
> > What do you think about making possible to add servers to backend at
> > runtime via CLI/socket?
>
> This is exactly what is being worked on, but it's not as trivial as you
> imagine :-)
>
> > Changing addresses and wieghts of servers is ok,
> > because we can use "placeholders", but it's not the best solution and
> > requires to configure many placeholders when it comes to autoscaling.
>
> We'll still use placeholders, we'll just ensure they're more discrete.
> Eg with Baptiste we discussed the idea of a "server template" which
> would declare a number of server slots at once that will be usable to
> be enabled on the CLI or via the DNS.
>
> > This problem solves reloading haproxy with safe loading certificates to
> > encrypted volume which takes some time and in dynamically changing world
> of
> > microservices - it just kills whole load balancing solutions with endless
> > reloads.
>
> Sure it solves a lot of things but it creates a vry long trail of
> new problems at the same time. Just one example : assigning automatic
> cookie values for servers created on the fly, that do not conflict with
> existing ones and which are invariant between multiple LB nodes and along
> DNS updates. Another one ? Consistent server IDs across all LBs for shared
> stickiness and consistent stats. There will also be quite a number of
> functional limitations, such as the impossibility to use such servers
> in "use-server" directives, the impossibility to reference them in ACLs
> (eg: srv_is_up()), impossibility to track them from other servers, etc.
> All this because these references are resolved upon startup and are
> mandatory to validate the consistency of the configuration.
>
> But some progress is being made in this area (and in 1.7 you can at least
> start a server with no address and fill it later, which is a huge progress
> compared to 1.6).
>
> Best regards,
> Willy
>

Dynamically adding server's to backend

[Michał]

Hello,

What do you think about making possible to add servers to backend at
runtime via CLI/socket? Changing addresses and wieghts of servers is ok,
because we can use "placeholders", but it's not the best solution and
requires to configure many placeholders when it comes to autoscaling.

This problem solves reloading haproxy with safe loading certificates to
encrypted volume which takes some time and in dynamically changing world of
microservices - it just kills whole load balancing solutions with endless
reloads.

I'm not sure how it will work, so the only question is if this feature is
welcome in haproxy or you don't want to add it now. I will try to do it
easiest way possible and check if any algorithms can break after adding
server at runtime, so if you are interested in such patch - I will report
how it works and if there is such need - what we should do to support it if
there's no such possibility to merge this feature in current state.

Michał

Propagating agent-check weight change to tracking servers

[Michał]

Hello,

We use track's in haproxy to minimize check traffic in some situations and
after my last patch we are probably going to switch to agent-checks for
live management of weights and statuses. One problem I see now - track
don't propagate weight setting to trackers, so if we set agent-check on
track we can manage status only.

My first PoC solution works good, so I thought about introducing something
like agent-track or track-agent directive set on backends (or maybe it
should be default, non-configurable behaviour) to propagate agent-check
responses from main check to all tracking backends. Both default behaviour
and directive approach are small changes in code, but a little bigger in
functionality.

In my opinion if we set agent-check on backend which is tracked by others -
it should propagate agent-check weight response to those tracking backend
and set weight on them too. Configurable or not - it will be good feature.

Michał

Re: Add agent-host configuration directive and allow changing it and agent-send via socket/CLI

[Michał]

Hello!
Thank you for reviewing. For me agent-addr looks better too, I hope it
won't
be confused with "addr" directive.

So here are patches with "agent-addr" changes and I added extebded commit
messages to code commits.

I wrote "Can be backported", because those are not bugfix'es, because they
don't
have to be backported, but on the other hand upgrading to 1.8 won't be
near-future for many projects and backport to 1.7 will spread this feature
to
more people.


Patches files md5:

MD5 (./0001-PATCH-MINOR-checks-Add-agent-addr-config-directive.patch) =
555a04df5dc56fa44ab94cb321df469b
MD5 (./0002-PATCH-MINOR-cli-Add-possiblity-to-change-agent-confi.patch) =
4b1d732dff72a3349766d74a7f28dcd6
MD5 (./0003-PATCH-MINOR-doc-Add-docs-for-agent-addr-configuratio.patch) =
2421ff01f26936e59461955b4ea5684f
MD5 (./0004-PATCH-MINOR-doc-Add-docs-for-agent-addr-and-agent-se.patch) =
971d6c1d23657130db4045d5b9e8cb73

Michał


0001-PATCH-MINOR-checks-Add-agent-addr-config-directive.patch
Description: Binary data


0002-PATCH-MINOR-cli-Add-possiblity-to-change-agent-confi.patch
Description: Binary data


0003-PATCH-MINOR-doc-Add-docs-for-agent-addr-configuratio.patch
Description: Binary data


0004-PATCH-MINOR-doc-Add-docs-for-agent-addr-and-agent-se.patch
Description: Binary data

Add agent-host configuration directive and allow changing it and agent-send via socket/CLI

[Michał]

Hello!
It's my first PR to haproxy, so please tell me if anything still wrong.
I've read CONTRIBUTING.

This patches implements possiblity to define different host (agent-host) for
agent checks in config and they also allow changing agent-host and
agent-send
variables via CLI/socket. We wonna use this options to dynamically swap
backends
without reloading haproxy. agent-host will allow us to control weight and
status
of servers from single place, so we can enable server when it's ready
(warmed)
instead just after healthcheck passing. I think this change isn't touching
any
hot paths and will find adopters, because reloading haproxy with all SSL
stuff
and losing statstics for less dynamic backends is pain.

In my opinion changes in code don't require any comments. I documented those
features of course, so everyone can use them.

In attachments there are same patches as below, just in .patch format in
case
mail get corrupted.

MD5 (0001-Add-agent-host-config-directive.patch) =
37a3c156b7a7d213f25b1ea52e37df10
MD5 (0002-Add-possiblity-to-change-agent-config-via-CLI-socket.patch) =
f9919fa73a21bd0a178aa301e1585b73
MD5 (0003-Add-docs-for-agent-host-configuration-variable.patch) =
f7d54b0aaef0ea5034ed80193d6f40b9
MD5 (0004-Add-docs-for-agent-host-and-agent-send-CLI-commands.patch) =
8cb920bbab07d964ae5f6d7c6e1c830b

---
 src/server.c | 8 
 1 file changed, 8 insertions(+)

diff --git a/src/server.c b/src/server.c
index 2f539c9..0ca372a 100644
--- a/src/server.c
+++ b/src/server.c
@@ -1153,6 +1153,14 @@ int parse_server(const char *file, int linenum, char
**args, struct proxy *curpr
newsrv->agent.inter = val;
cur_arg += 2;
}
+   else if (!strcmp(args[cur_arg], "agent-host")) {
+   if(str2ip(args[cur_arg + 1], >agent.addr) == NULL) {
+   Alert("parsing agent-host failed. Check if %s is
correct address.\n", args[cur_arg + 1]);
+   goto out;
+   }
+
+   cur_arg += 2;
+   }
else if (!strcmp(args[cur_arg], "agent-port")) {
global.maxsock++;
newsrv->agent.port = atol(args[cur_arg + 1]);
-- 

---
 doc/configuration.txt | 8 
 1 file changed, 8 insertions(+)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index ae76ef3..088bba6 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -10684,6 +10684,14 @@ agent-inter 

   Supported in default-server: Yes

+agent-host 
+  The "agent-host" parameter sets address for agent check.
+
+  You can offload agent-check to another target, so you can make single
place
+  managing status and weights of servers defined in haproxy in case you
can't
+  make self-aware and self-managing services. You can specify both IP or
+  hostname, it will be resolved.
+
 agent-port 
   The "agent-port" parameter sets the TCP port used for agent checks.

-- 

---
 src/server.c | 27 +++
 1 file changed, 27 insertions(+)

diff --git a/src/server.c b/src/server.c
index 0ca372a..1e6810c 100644
--- a/src/server.c
+++ b/src/server.c
@@ -3467,6 +3467,33 @@ static int cli_parse_set_server(char **args, struct
appctx *appctx, void *privat
appctx->st0 = CLI_ST_PRINT;
}
}
+   else if (strcmp(args[3], "agent-host") == 0) {
+   if(!(sv->agent.state & CHK_ST_ENABLED)) {
+   appctx->ctx.cli.msg = "agent checks are not enabled on this
server.\n";
+   appctx->st0 = CLI_ST_PRINT;
+   } else {
+   if(str2ip(args[4], >agent.addr) == NULL) {
+   appctx->ctx.cli.msg = "incorrect host address given for
agent.\n";
+   appctx->st0 = CLI_ST_PRINT;
+   }
+   }
+   }
+   else if(strcmp(args[3], "agent-send") == 0) {
+   if(!(sv->agent.state & CHK_ST_ENABLED)) {
+   appctx->ctx.cli.msg = "agent checks are not enabled on this
server.\n";
+   appctx->st0 = CLI_ST_PRINT;
+   } else {
+   char *nss = strdup(args[4]);
+   if(!nss) {
+   appctx->ctx.cli.msg = "cannot allocate memory for new
string.\n";
+   appctx->st0 = CLI_ST_PRINT;
+   } else {
+   free(sv->agent.send_string);
+   sv->agent.send_string = nss;
+   sv->agent.send_string_len = strlen(args[4]);
+   }
+   }
+   }
else if (strcmp(args[3], "check-port") == 0) {
int i = 0;
if (strl2irc(args[4], strlen(args[4]), ) != 0) {
-- 

---
 doc/management.txt | 9 +
 1 file changed, 9 insertions(+)

diff --git a/doc/management.txt b/doc/management.txt
index dd886de..ec4490f 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -1603,6 +1603,15 @@ set server / agent [ up | down ]
   switch a server's state regardless of some slow agent checks for example.
   Note that the change is propagated to tracking servers if any.

+set server / agent-host 
+  Change addr for servers agent checks. Allows to migrate agent-checks 

Segmentation fault in 1.6.6

[Michał Łowicki]

Hi,

I'm using:

> HA-Proxy version 1.6.6 2016/06/26
> Copyright 2000-2016 Willy Tarreau <wi...@haproxy.org>


on:

> cat /proc/version
> Linux version 4.5.0-0.bpo.1-amd64 (debian-ker...@lists.debian.org) (gcc
> version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Debian 4.5.1-1~bpo8+1 (2016-04-20)

and found core dump (using HAProxy through Marathon-lb
<https://github.com/mesosphere/marathon-lb>):

> [New LWP 20445]
> Core was generated by `/bin/bash ./run'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x004b8f97 in action_lookup (keywords= Cannot access memory at address 0xffe8>,
> kw= 0xffe0>) at include/proto/action.h:38
> 38 include/proto/action.h: No such file or directory.

Is this a known issue or any clue how to debug it futher?

-- 
BR,
Michał Łowicki

HTTP keep-alive reuse count & max-age

[Michał Pasierb]

Hi,

Is it possible to influence how HAProxy handles HTTP keep-alives to backend
servers ? I want it to close TCP connection after x many HTTP requests.
Also a max-age time for single TCP connection cloud be useful.

These features are available in F5 LTM OneConnect (
https://support.f5.com/kb/en-us/solutions/public/7000/200/sol7208.html)
as *Maximum
Reuse* and *Maximum Age*.

Goal is to move all TIME_WAITs to HAProxy host instead of creating them on
backend servers.

Thanks,
Michal

cookie prefix & option httpclose

[Michał Pasierb]

Hi,

I have a question about cookie prefix mode. I use configuration like this:

backend b1
balance roundrobin
cookie JSESSIONID prefix
server s1 192.168.122.1:8080 cookie c1

Now the docs
https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4-cookie for
prefix say:

Since all requests and responses are subject to being modified, this
mode requires the HTTP close mode.

Behavior for version 1.4 is consistent with docs. The cookie was not
prefixed for second request within one http keepalive session so I needed
to add "option httpclose" to modify all requests.

Now I did the same setup with version 1.5 and found out that even without
"option httpclose" both requests have the cookie prefixed.

My dillema - are the docs inconsistent and this section should be removed
or am I missing something ?

Thanks,
Michal

Backend connection resets in http-heep-alive mode

[Michał Ślizak]

Hi,

I'm experiencing a problem with backend TCP connections being reset
[RST+ACK] by HAProxy after serving one HTTP request and receiving the [ACK]
for the HTTP response. Delay between backend's [ACK] and haproxy's
[RST+ACK] seems random, ranging from single seconds to several minutes.

What I wanted to achieve is: for each backend server keep a limited pool of
backend TCP connections open, and when a HTTP request comes in through the
frontend, reuse one of the existing connections (in HTTP keep-alive mode),
creating one if necessary.

It was my understanding that 'timeout server 5m' should keep backend
connections opened for 5 minutes before closing them. Was I mistaken? Would
timeout tunnel allow me to specify such timeout, despite HAProxy working
in http-keep-alive mode?

I'm attaching the haproxy config.

Thank you,



-- 

*Michał Ślizak*
DaftCode Sp. z o.o.
ul. Domaniewska 34a, 02-672 Warszawa
tel. +48 506 175 074


haproxy_redacted.cfg
Description: Binary data

Re: HAProxy 1.4.18 performance issue

[Michał Ślizak]

On Mon, Apr 13, 2015 at 6:09 PM, Lukas Tribus luky...@hotmail.com wrote:

  On Mon, Apr 13, 2015 at 4:58 PM, Lukas Tribus
  luky...@hotmail.commailto:luky...@hotmail.com wrote:
  Hi,
 
  I'm experiencing latency problems while running HAProxy 1.4.18.
 
  Our backend servers reply to HAProxy almost instantly (~4ms), but some
  of those replies are sent to the clients more than 100ms later.
 
  We have approx. 50k sessions opened at any time, with a HTTP request
  coming in approximately every minute over each session.
 
  I suggest you try option http-no-delay but really try to understand
  the implications:
 
 http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#option%20http-no-delay

 
 
  Thanks Lucas, option http-no-delay seems to have solved the problem.

 Good, but this will just hide the real problem and may cause others (as
 per the documentation).
 Both 1.4.23 and 1.4.20 fix latency (MSG_MORE/DONTWAIT) related problems,
 also, if you always
 expect zero latency from the proxy then you are misusing HTTP.

 I strongly suggest you consider upgrading to latest stable (either 1.4 or
 better yet 1.5) and retry
 without this command.

 You didn't provide your configuration so its not possible to tell if your
 are running into those
 already fixed bugs, or if you simply need zero latency in all cases per
 application design.


Hi Lukas,

We've upgraded to 1.5.11 and are working fine without option
http-no-delay.

Thank you for your help,

-- 

*Michał Ślizak*
DaftCode Sp. z o.o.
ul. Domaniewska 34a, 02-672 Warszawa
tel. +48 506 175 074

HAProxy 1.4.18 performance issue

[Michał Ślizak]

Hi,

I'm experiencing latency problems while running HAProxy 1.4.18.

Our backend servers reply to HAProxy almost instantly (~4ms), but some of
those replies are sent to the clients more than 100ms later.

We have approx. 50k sessions opened at any time, with a HTTP request coming
in approximately every minute over each session.

I realize the version we're running is very old and should be updated.
I'd like to know if there was a bug that could explain what we're seeing?
If so, which version of HAProxy fixed it?

Best regards,

-- 

*Michał Ślizak*
DaftCode Sp. z o.o.
ul. Domaniewska 34a, 02-672 Warszawa
tel. +48 506 175 074

Re: HAProxy 1.4.18 performance issue

[Michał Ślizak]

On Mon, Apr 13, 2015 at 6:09 PM, Lukas Tribus luky...@hotmail.com wrote:

  On Mon, Apr 13, 2015 at 4:58 PM, Lukas Tribus
  luky...@hotmail.commailto:luky...@hotmail.com wrote:
  Hi,
 
  I'm experiencing latency problems while running HAProxy 1.4.18.
 
  Our backend servers reply to HAProxy almost instantly (~4ms), but some
  of those replies are sent to the clients more than 100ms later.
 
  We have approx. 50k sessions opened at any time, with a HTTP request
  coming in approximately every minute over each session.
 
  I suggest you try option http-no-delay but really try to understand
  the implications:
 
 http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#option%20http-no-delay

 
 
  Thanks Lucas, option http-no-delay seems to have solved the problem.

 Good, but this will just hide the real problem and may cause others (as
 per the documentation).
 Both 1.4.23 and 1.4.20 fix latency (MSG_MORE/DONTWAIT) related problems,
 also, if you always
 expect zero latency from the proxy then you are misusing HTTP.

 I strongly suggest you consider upgrading to latest stable (either 1.4 or
 better yet 1.5) and retry
 without this command.

 You didn't provide your configuration so its not possible to tell if your
 are running into those
 already fixed bugs, or if you simply need zero latency in all cases per
 application design.



Thank you for your suggestions, I'll discuss upgrading with our sysops team.

Unfortunately misusing HTTP keep-alive is what we do ;-)
We're running a real-time bidding system which has to reply to auctions
(HTTP GET or POST requests) over thousands of connections within 75ms.
Each connection doesn't carry much traffic, maybe one request per minute,
and both the requests and our replies are very small.
Waiting for a full reply packet will never work in our case.

It would be much more efficient to decrease the number of concurrent
connections and send requests more often, but unfortunately we have no
control over this.

Best regards,

-- 

*Michał Ślizak*
DaftCode Sp. z o.o.
ul. Domaniewska 34a, 02-672 Warszawa
tel. +48 506 175 074

Re: HAProxy 1.4.18 performance issue

[Michał Ślizak]

On Mon, Apr 13, 2015 at 4:58 PM, Lukas Tribus luky...@hotmail.com wrote:

  Hi,
 
  I'm experiencing latency problems while running HAProxy 1.4.18.
 
  Our backend servers reply to HAProxy almost instantly (~4ms), but some
  of those replies are sent to the clients more than 100ms later.
 
  We have approx. 50k sessions opened at any time, with a HTTP request
  coming in approximately every minute over each session.

 I suggest you try option http-no-delay but really try to understand
 the implications:

 http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#option%20http-no-delay


Thanks Lucas, option http-no-delay seems to have solved the problem.

-- 

*Michał Ślizak*
DaftCode Sp. z o.o.
ul. Domaniewska 34a, 02-672 Warszawa
tel. +48 506 175 074