Hello!

I am currently on vacation for two weeks, but I'll see to it when I get back.
There is no particular reason for the specific check address here, as you 
correctly figured. It is just an artefact of the template used to create the 
configuration. I can remove that, but there might be cases were it matters 
(though I don't think we have any ATM AFAIR). Would not have guessed there 
would be different resolution paths; if this is intentional, a note in the 
documentation would be helpful. I can provide that when I am back and when 
there is clarity on why it would be like this.

Thank you very much for your help!

Cheers,
Daniel


> On 23. Mar 2019, at 14:53, PiBa-NL <piba.nl....@gmail.com> wrote:
> 
> Hi Daniel, Baptiste,
> 
> @Daniel, can you remove the 'addr loadbalancer-internal.xxx.yyy' from the 
> server check? It seems to me that that name is not being resolved by the 
> 'resolvers'. And even if it would it would be kinda redundant as it is in the 
> example as it is the same as the servername.?. Not sure how far below 
> scenarios are all explained by this though..
> 
> @Baptiste, is it intentional that a wrong 'addr' dns name makes haproxy fail 
> to start despite having the supposedly never failing 'default-server 
> init-addr last,libc,none' ? Is it possibly a good feature request to support 
> re-resolving a dns name for the addr setting as well ?
> 
> Regards,
> PiBa-NL (Pieter)
> 
> Op 21-3-2019 om 20:37 schreef Daniel Schneller:
>> Hi!
>> 
>> Thanks for the response. I had looked at the "hold" directives, but since 
>> they all seem to have reasonable defaults, I did not touch them.
>> I specified 10s explictly, but it did not make a difference.
>> 
>> I did some more tests, however, and it seems to have more to do with the 
>> number of responses for the initial(?) DNS queries.
>> Hopefully these three tables make sense and don't get mangled in the mail. 
>> The "templated"
>> proxy is defined via "server-template" with 3 "slots". The "regular" one 
>> just as "server".
>> 
>> 
>> Test 1: Start out  with both "valid" and "broken" DNS entries. Then comment 
>> out/add back
>> one at a time as described in (1)-(5).
>> Each time after changing /etc/hosts, restart dnsmasq and check haproxy via 
>> hatop.
>> Haproxy started fresh once dnsmasq was set up to (1).
>> 
>>                        |  state           state
>>             /etc/hosts |  regular         templated
>>            ------------|-----------------------------
>> (1)         BRK        |  UP/L7OK         DOWN/L4TOUT
>>             VALID      |                  MAINT/resolution
>>                        |                  UP/L7OK
>>            ------------|--------------------------------
>> 
>> (2)         BRK        |  DOWN/L4TOUT     DOWN/L4TOUT
>>             #VALID     |                  MAINT/resolution
>>                        |                  MAINT/resolution
>>                        |
>> (3)         #BRK       |  UP/L7OK         UP/L7OK
>>             VALID      |                  MAINT/resolution
>>                        |                  MAINT/resolution
>>            ------------|------------------------------------
>> (4)         BRK        |  UP/L7OK         UP/L7OK
>>             VALID      |                  DOWN/L4TOUT
>>                        |                  MAINT/resolution
>>            ------------|------------------------------------
>> (5)         BRK        |  DOWN/L4TOUT     DOWN/L4TOUT
>>             #VALID     |                  MAINT/resolution
>>                        |                  MAINT/resolution
>>                       This all looks normal and as expected. As soon as the 
>> "VALID" DNS entry is present, the
>> UP state follows within a few seconds.
>>                               
>> 
>> Test 2: Start out "valid only" (1) and proceed as described in (2)-(5), 
>> again restarting
>> dnsmasq each time, and haproxy reloaded after dnsmasq was set up to (1).
>> 
>>                        |  state           state
>>             /etc/hosts |  regular         templated
>>            ------------|------------------------------------
>> (1)         #BRK       |  UP/L7OK         MAINT/resolution
>>             VALID      |                  MAINT/resolution
>>                        |                  UP/L7OK
>>            ------------|------------------------------------
>> (2)         BRK        |  UP/L7OK         DOWN/L4TOUT
>>             VALID      |                  MAINT/resolution
>>                        |                  UP/L7OK
>>            ------------|------------------------------------
>> (3)         #BRK       |  UP/L7OK         MAINT/resolution
>>             VALID      |                  MAINT/resolution
>>                        |                  UP/L7OK
>>            ------------|------------------------------------
>> (4)         BRK        |  UP/L7OK         DOWN/L4TOUT
>>             VALID      |                  MAINT/resolution
>>                        |                  UP/L7OK
>>            ------------|------------------------------------
>> (5)         BRK        |  DOWN/L4TOUT     DOWN/L4TOUT
>>             #VALID     |                  MAINT/resolution
>>                        |                  MAINT/resolution
>>                               Everything good here, too. Adding the broken 
>> DNS entry does not bring the proxies down
>> until only the broken one is left.
>> 
>> 
>> 
>> Test 3: Start out "broken only" (1).
>> Again, same as before, haproxy restarted once dnsmasq was initialized to (1).
>> 
>>                        |  state           state
>>             /etc/hosts |  regular         templated
>>            ------------|------------------------------------
>> (1)         BRK        |  DOWN/L4TOUT     DOWN/L4TOUT
>>             #VALID     |                  MAINT/resolution
>>                        |                  MAINT/resolution
>>            ------------|------------------------------------
>> (2)         BRK        |  DOWN/L4TOUT     UP/L7OK
>>             VALID      |                  MAINT/resolution
>>                        |                  MAINT/resolution
>>            ------------|------------------------------------
>> (3)         #BRK       |  UP/L7OK         MAINT/resolution
>>             VALID      |                  UP/L7OK
>>                        |                  MAINT/resolution
>>            ------------|------------------------------------
>> (4)         BRK        |  UP/L7OK         DOWN/L4TOUT
>>             VALID      |                  UP/L7OK
>>                        |                  MAINT/resolution
>>            ------------|------------------------------------
>> (5)         BRK        |  DOWN/L4TOUT     DOWN/L4TOUT
>>             #VALID     |                  MAINT/resolution
>>                        |                  MAINT/resolution
>> 
>> 
>> Here it becomes interesting. In (1) both regular and templated proxies are 
>> DOWN, of course.
>> However, adding in a second DNS response in (2) brings the templated proxy 
>> UP, but the regular
>> one stays DOWN. Only when in (3) the valid response is the only one 
>> presented, does it go
>> UP as well. Adding the broken one back (4) is of no consequence then. And 
>> again, after
>> leaving just the broken response (5), both correctly go DOWN.
>> 
>> So it would appear that if haproxy starts with just a single "broken" DNS 
>> response, adding
>> a healthy one later one is not recognized. Instead, it stays DOWN. 
>> "Replacing" the single
>> broken response with a single "valid" response, however, brings it to life, 
>> and it won't be
>> discouraged by bringing the broken one back in.
>> 
>> Tests 1 and 2 make sense to me, but test 3 I don't understand. For now, I 
>> have worked
>> around the issue by defining all my relevant backends with server-template 
>> and at least
>> 2 slots, but I would still like to understand it. And maybe it is a bug, 
>> after all ;)
>> 
>> Kind regards, and thanks for a great piece of software!
>> 
>> Daniel
>> 
>> 
>> 
>> 
>> 
>>> On 21. Mar 2019, at 14:28, Bruno Henc <brh...@nua-avenir.net> wrote:
>>> 
>>> Hello Daniel,
>>> 
>>> 
>>> You might be missing the hold-valid directive in your resolvers section: 
>>> https://www.haproxy.com/documentation/hapee/1-9r1/onepage/#5.3.2-timeout
>>> 
>>> This should force HAProxy to fetch the DNS record values from the resolver.
>>> 
>>> A reload of the HAProxy instance also forces the instances to query all 
>>> records from the resolver.
>>> 
>>> Can you please retest with the updated configuration and report back the 
>>> results?
>>> 
>>> 
>>> Best regards,
>>> 
>>> Bruno Henc
>>> 
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On Thursday, March 21, 2019 12:09 PM, Daniel Schneller 
>>> <daniel.schnel...@centerdevice.com> wrote:
>>> 
>>>> Hello!
>>>> 
>>>> Friendly bump :)
>>>> I'd be willing to amend the documentation once I understand what's going 
>>>> on :D
>>>> 
>>>> Cheers,
>>>> Daniel
>>>> 
>>>>> On 18. Mar 2019, at 20:28, Daniel Schneller 
>>>>> daniel.schnel...@centerdevice.com wrote:
>>>>> Hi everyone!
>>>>> I assume I am misunderstanding something, but I cannot figure out what it 
>>>>> is.
>>>>> We are using haproxy in AWS, in this case as sidecars to applications so 
>>>>> they need not
>>>>> know about changing backend addresses at all, but can always talk to 
>>>>> localhost.
>>>>> Haproxy listens on localhost and then forwards traffic to an ELB instance.
>>>>> This works great, but there have been two occasions now, where due to a 
>>>>> change in the
>>>>> ELB's IP addresses, our services went down, because the backends could 
>>>>> not be reached
>>>>> anymore. I don't understand why haproxy sticks to the old IP address 
>>>>> instead of going
>>>>> to one of the updated ones.
>>>>> There is a resolvers section which points to the local dnsmasq instance 
>>>>> (there to send
>>>>> some requests to consul, but that's not used here). All other traffic is 
>>>>> forwarded on
>>>>> to the AWS DNS server set via DHCP.
>>>>> I managed to get timely updates and updated backend servers when using 
>>>>> server-template,
>>>>> but form what I understand this should not really be necessary for this.
>>>>> This is the trimmed down sidecar config. I have not made any changes to 
>>>>> dns timeouts etc.
>>>>> resolvers default
>>>>> 
>>>>> dnsmasq
>>>>> 
>>>>> ========
>>>>> 
>>>>> nameserver local 127.0.0.1:53
>>>>> listen regular
>>>>> bind 127.0.0.1:9300
>>>>> option dontlog-normal
>>>>> server lb-internal loadbalancer-internal.xxx.yyy:9300 resolvers default 
>>>>> check addr loadbalancer-internal.xxx.yyy port 9300
>>>>> listen templated
>>>>> bind 127.0.0.1:9200
>>>>> option dontlog-normal
>>>>> option httpchk /haproxy-simple-healthcheck
>>>>> server-template lb-internal 2 loadbalancer-internal.xxx.yyy:9200 
>>>>> resolvers default check port 9299
>>>>> To simulate changing ELB adresses, I added entries for 
>>>>> loadbalancer-internal.xxx.yyy in /etc/hosts
>>>>> and to be able to control them via dnsmasq.
>>>>> I tried different scenarios, but could not reliably predict what would 
>>>>> happen in all cases.
>>>>> The address ending in 52 (marked as "valid" below) is a currently (as of 
>>>>> the time of testing)
>>>>> valid IP for the ELB. The one ending in 199 (marked "invalid") is an 
>>>>> unused private IP address
>>>>> in my VPC.
>>>>> Starting with /etc/hosts:
>>>>> 10.205.100.52 loadbalancer-internal.xxx.yyy # valid
>>>>> 10.205.100.199 loadbalancer-internal.xxx.yyy # invalid
>>>>> haproxy starts and reports:
>>>>> regular: lb-internal UP/L7OK
>>>>> templated: lb-internal1 DOWN/L4TOUT
>>>>> lb-internal2 UP/L7OK
>>>>> That's expected. Now when I edit /etc/hosts to only contain the invalid 
>>>>> address
>>>>> and restart dnsmasq, I would expect both proxies to go fully down. But 
>>>>> only the templated
>>>>> proxy behaves like that:
>>>>> regular: lb-internal UP/L7OK
>>>>> templated: lb-internal1 DOWN/L4TOUT
>>>>> lb-internal2 MAINT (resolution)
>>>>> Reloading haproxy in this state leads to:
>>>>> regular: lb-internal DOWN/L4TOUT
>>>>> templated: lb-internal1 MAINT (resolution)
>>>>> lb-internal2 DOWN/L4TOUT
>>>>> After fixing /etc/hosts to include the valid server again and restarting 
>>>>> dnsmasq:
>>>>> regular: lb-internal DOWN/L4TOUT
>>>>> templated: lb-internal1 UP/L7OK
>>>>> lb-internal2 DOWN/L4TOUT
>>>>> Shouldn't the regular proxy also recognize the change and bring the 
>>>>> backend up or down
>>>>> depending on the DNS change? I have waited for several health check 
>>>>> rounds (seeing
>>>>> "* L4TOUT" and "L4TOUT") toggle, but it still never updates.
>>>>> I also tried to have only the invalid address in /etc/hosts, then 
>>>>> restarting haproxy.
>>>>> The regular backends will never recognize it when I add the valid one 
>>>>> back in.
>>>>> The templated one does, unless I set it up to have only 1 instead of 2 
>>>>> server slots.
>>>>> In that case it behaves will also only pick up the valid server when 
>>>>> reloaded.
>>>>> On the other hand, it will recognize when I remove the valid server 
>>>>> without a reload
>>>>> on the next health check, but not bring them back in and make the proxy 
>>>>> UP when it
>>>>> comes back.
>>>>> I assume my understanding of something here is broken, and I would gladly 
>>>>> be told
>>>>> about it :)
>>>>> Thanks a lot!
>>>>> Daniel
>>>>> 
>>>>> Version Info:
>>>>> 
>>>>> --------------
>>>>> 
>>>>> $ haproxy -vv
>>>>> HA-Proxy version 1.8.19-1ppa1~trusty 2019/02/12
>>>>> Copyright 2000-2019 Willy Tarreau wi...@haproxy.org
>>>>> Build options :
>>>>> TARGET = linux2628
>>>>> CPU = generic
>>>>> CC = gcc
>>>>> CFLAGS = -O2 -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 
>>>>> -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-strict-aliasing 
>>>>> -Wdeclaration-after-statement -fwrapv -Wno-unused-label
>>>>> OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 
>>>>> USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1
>>>>> Default settings :
>>>>> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>>>>> Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
>>>>> Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
>>>>> OpenSSL library supports TLS extensions : yes
>>>>> OpenSSL library supports SNI : yes
>>>>> OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
>>>>> Built with Lua version : Lua 5.3.1
>>>>> Built with transparent proxy support using: IP_TRANSPARENT 
>>>>> IPV6_TRANSPARENT IP_FREEBIND
>>>>> Encrypted password support via crypt(3): yes
>>>>> Built with multi-threading support.
>>>>> Built with PCRE version : 8.31 2012-07-06
>>>>> Running on PCRE version : 8.31 2012-07-06
>>>>> PCRE library supports JIT : no (libpcre build without JIT?)
>>>>> Built with zlib version : 1.2.8
>>>>> Running on zlib version : 1.2.8
>>>>> Compression algorithms supported : identity("identity"), 
>>>>> deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
>>>>> Built with network namespace support.
>>>>> Available polling systems :
>>>>> epoll : pref=300, test result OK
>>>>> poll : pref=200, test result OK
>>>>> select : pref=150, test result OK
>>>>> Total: 3 (3 usable), will use epoll.
>>>>> Available filters :
>>>>> [SPOE] spoe
>>>>> [COMP] compression
>>>>> [TRACE] trace
>>>>> --
>>>>> Daniel Schneller
>>>>> Principal Cloud Engineer
>>>>> CenterDevice GmbH
>>>>> Rheinwerkallee 3
>>>>> 53227 Bonn
>>>>> www.centerdevice.com
>>>>> 
>>>>> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina, Michael 
>>>>> Rosbach, Handelsregister-Nr.: HRB 18655, HR-Gericht: Bonn, USt-IdNr.: 
>>>>> DE-815299431
>>>>> Diese E-Mail einschließlich evtl. beigefügter Dateien enthält 
>>>>> vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht 
>>>>> der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, 
>>>>> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail 
>>>>> und evtl. beigefügter Dateien umgehend. Das unerlaubte Kopieren, Nutzen 
>>>>> oder Öffnen evtl. beigefügter Dateien sowie die unbefugte Weitergabe 
>>>>> dieser E-Mail ist nicht gestattet.
>>> 
>> 
> 


Reply via email to