Re: Using server-template for DNS resolution
Hi Baptise,
On Fri, Feb 8, 2019 at 6:10 PM Baptiste wrote:
>
>
> On Fri, Feb 8, 2019 at 6:09 AM Igor Cicimov <
> ig...@encompasscorporation.com> wrote:
>
>> On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov <
>> ig...@encompasscorporation.com> wrote:
>>
>>> Hi,
>>>
>>> I have a Jetty frontend exposed for couple of ActiveMQ servers behind
>>> SSL terminating Haproxy-1.8.18. They share same storage and state via lock
>>> file and there is only one active AMQ at any given time. I'm testing this
>>> now with dynamic backend using Consul DNS resolution:
>>>
>>> # dig +short @127.0.0.1 -p 8600 activemq.service.consul
>>> 10.140.4.122
>>> 10.140.3.171
>>>
>>> # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV
>>> 1 1 61616 ip-10-140-4-122.node.dc1.consul.
>>> 1 1 61616 ip-10-140-3-171.node.dc1.consul.
>>>
>>> The backends status, the current "master":
>>>
>>> root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep
>>> java
>>> tcp0 0 0.0.0.0:81610.0.0.0:*
>>> LISTEN 5031374919617256/java
>>> tcp0 0 0.0.0.0:6161 0.0.0.0:*
>>> LISTEN 5031374919317256/java
>>>
>>> and the "slave":
>>>
>>> root@ip-10-140-4-122:~# netstat -tuplen | grep java
>>>
>>> So the service ports are not available on the second one.
>>>
>>> This is the relevant part of the HAP config that I think might be of
>>> interest:
>>>
>>> global
>>> server-state-base /var/lib/haproxy
>>> server-state-file hap_state
>>>
>>> defaults
>>> load-server-state-from-file global
>>> default-server init-addrlast,libc,none
>>>
>>> listen amq
>>> bind ... ssl crt ...
>>> mode http
>>>
>>> option prefer-last-server
>>>
>>> # when this is on the backend is down
>>> #option tcp-check
>>>
>>> default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s
>>> maxconn 25 maxqueue 256 weight 100
>>>
>>> # working but both show as up
>>> server-template amqs 2 activemq.service.consul:8161 check
>>>
>>> # working old static setup
>>> #server ip-10-140-3-171 10.140.3.171:8161 check
>>> #server ip-10-140-4-122 10.140.4.122:8161 check
>>>
>>> This is working but the thing is I see both servers as UP in the HAP
>>> console:
>>> [image: amqs.png]
>>> Is this normal for this kind of setup or I'm doing something wrong?
>>>
>>> Another observation, when I have tcp check enabled like:
>>>
>>> option tcp-check
>>>
>>> the way I had it with the static lines like:
>>>
>>> server ip-10-140-3-171 10.140.3.171:8161 check
>>> server ip-10-140-4-122 10.140.4.122:8161 check
>>>
>>> then both servers show as down.
>>> Thanks in advance for any kind of input.
>>> Igor
>>>
>>> Ok, the state has changed now, I have correct state on one haproxy:
>>
>> [image: amqs_hap1.png]
>> but on the second the whole backend is down:
>>
>> [image: amqs_hap2.png]
>> I confirmed via telnet that I can connect to port 8161 to the running amq
>> server from both haproxy servers.
>>
>>
>
>
> Hi Igor,
>
> You're using the libc resolver function at startup time to resolve your
> backend, this is not recommended integration with Consul.
> You will find some good explanations in this blog article:
>
> https://www.haproxy.com/fr/blog/haproxy-and-consul-with-dns-for-service-discovery/
>
> Basically, you should first create a "resolvers" section, in order to
> allow HAProxy to perform DNS resolution at runtime too.
>
> resolvers consul
> nameserver consul 127.0.0.1:8600
> accepted_payload_size 8192
>
> Then, you need to adjust your server-template line, like this:
> server-template amqs 10 _activemq._tcp.service.consul resolvers consul
> resolve-prefer ipv4 check
>
> In the example above, I am using on purpose the SRV records, because
> HAProxy supports it and it will use all information available in the
> response to update server's IP, weight and port.
>
> I hope this will help you.
>
> Baptiste
>
All sorted now. For the record and those interested here is my setup:
Haproxy:
global
server-state-base /var/lib/haproxy
server-state-file hap_state
defaults
load-server-state-from-file global
default-server init-addrlast,libc,none
resolvers consul
nameserver consul 127.0.0.1:8600
accepted_payload_size 8192
resolve_retries 30
timeout resolve 1s
timeout retry 2s
hold valid30s
hold other30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold obsolete 30s
listen jetty
bind _port_ ssl crt ...
mode http
option forwardfor except 127.0.0.1 header X-Forwarded-For
option http-ignore-probes
option prefer-last-server
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s
maxconn 25 maxqueue 256 weight 100
server-template jettys 2 _jetty._tcp.service.consul resolvers consul
resolve-prefer ipv4 check
Consul:
---
{
"services": [
{
Re: Using server-template for DNS resolution
On Fri, Feb 8, 2019 at 6:09 AM Igor Cicimov wrote: > On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov < > ig...@encompasscorporation.com> wrote: > >> Hi, >> >> I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL >> terminating Haproxy-1.8.18. They share same storage and state via lock file >> and there is only one active AMQ at any given time. I'm testing this now >> with dynamic backend using Consul DNS resolution: >> >> # dig +short @127.0.0.1 -p 8600 activemq.service.consul >> 10.140.4.122 >> 10.140.3.171 >> >> # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV >> 1 1 61616 ip-10-140-4-122.node.dc1.consul. >> 1 1 61616 ip-10-140-3-171.node.dc1.consul. >> >> The backends status, the current "master": >> >> root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep >> java >> tcp0 0 0.0.0.0:81610.0.0.0:* >> LISTEN 5031374919617256/java >> tcp0 0 0.0.0.0:6161 0.0.0.0:* >> LISTEN 5031374919317256/java >> >> and the "slave": >> >> root@ip-10-140-4-122:~# netstat -tuplen | grep java >> >> So the service ports are not available on the second one. >> >> This is the relevant part of the HAP config that I think might be of >> interest: >> >> global >> server-state-base /var/lib/haproxy >> server-state-file hap_state >> >> defaults >> load-server-state-from-file global >> default-server init-addrlast,libc,none >> >> listen amq >> bind ... ssl crt ... >> mode http >> >> option prefer-last-server >> >> # when this is on the backend is down >> #option tcp-check >> >> default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s >> maxconn 25 maxqueue 256 weight 100 >> >> # working but both show as up >> server-template amqs 2 activemq.service.consul:8161 check >> >> # working old static setup >> #server ip-10-140-3-171 10.140.3.171:8161 check >> #server ip-10-140-4-122 10.140.4.122:8161 check >> >> This is working but the thing is I see both servers as UP in the HAP >> console: >> [image: amqs.png] >> Is this normal for this kind of setup or I'm doing something wrong? >> >> Another observation, when I have tcp check enabled like: >> >> option tcp-check >> >> the way I had it with the static lines like: >> >> server ip-10-140-3-171 10.140.3.171:8161 check >> server ip-10-140-4-122 10.140.4.122:8161 check >> >> then both servers show as down. >> Thanks in advance for any kind of input. >> Igor >> >> Ok, the state has changed now, I have correct state on one haproxy: > > [image: amqs_hap1.png] > but on the second the whole backend is down: > > [image: amqs_hap2.png] > I confirmed via telnet that I can connect to port 8161 to the running amq > server from both haproxy servers. > > Hi Igor, You're using the libc resolver function at startup time to resolve your backend, this is not recommended integration with Consul. You will find some good explanations in this blog article: https://www.haproxy.com/fr/blog/haproxy-and-consul-with-dns-for-service-discovery/ Basically, you should first create a "resolvers" section, in order to allow HAProxy to perform DNS resolution at runtime too. resolvers consul nameserver consul 127.0.0.1:8600 accepted_payload_size 8192 Then, you need to adjust your server-template line, like this: server-template amqs 10 _activemq._tcp.service.consul resolvers consul resolve-prefer ipv4 check In the example above, I am using on purpose the SRV records, because HAProxy supports it and it will use all information available in the response to update server's IP, weight and port. I hope this will help you. Baptiste
Re: Using server-template for DNS resolution
On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov wrote: > Hi, > > I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL > terminating Haproxy-1.8.18. They share same storage and state via lock file > and there is only one active AMQ at any given time. I'm testing this now > with dynamic backend using Consul DNS resolution: > > # dig +short @127.0.0.1 -p 8600 activemq.service.consul > 10.140.4.122 > 10.140.3.171 > > # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV > 1 1 61616 ip-10-140-4-122.node.dc1.consul. > 1 1 61616 ip-10-140-3-171.node.dc1.consul. > > The backends status, the current "master": > > root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep > java > tcp0 0 0.0.0.0:81610.0.0.0:* > LISTEN 5031374919617256/java > tcp0 0 0.0.0.0:6161 0.0.0.0:* > LISTEN 5031374919317256/java > > and the "slave": > > root@ip-10-140-4-122:~# netstat -tuplen | grep java > > So the service ports are not available on the second one. > > This is the relevant part of the HAP config that I think might be of > interest: > > global > server-state-base /var/lib/haproxy > server-state-file hap_state > > defaults > load-server-state-from-file global > default-server init-addrlast,libc,none > > listen amq > bind ... ssl crt ... > mode http > > option prefer-last-server > > # when this is on the backend is down > #option tcp-check > > default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s > maxconn 25 maxqueue 256 weight 100 > > # working but both show as up > server-template amqs 2 activemq.service.consul:8161 check > > # working old static setup > #server ip-10-140-3-171 10.140.3.171:8161 check > #server ip-10-140-4-122 10.140.4.122:8161 check > > This is working but the thing is I see both servers as UP in the HAP > console: > [image: amqs.png] > Is this normal for this kind of setup or I'm doing something wrong? > > Another observation, when I have tcp check enabled like: > > option tcp-check > > the way I had it with the static lines like: > > server ip-10-140-3-171 10.140.3.171:8161 check > server ip-10-140-4-122 10.140.4.122:8161 check > > then both servers show as down. > Thanks in advance for any kind of input. > Igor > > Ok, the state has changed now, I have correct state on one haproxy: [image: amqs_hap1.png] but on the second the whole backend is down: [image: amqs_hap2.png] I confirmed via telnet that I can connect to port 8161 to the running amq server from both haproxy servers.
