Re: Using server-template for DNS resolution
Hi Baptise,
On Fri, Feb 8, 2019 at 6:10 PM Baptiste wrote:
>
>
> On Fri, Feb 8, 2019 at 6:09 AM Igor Cicimov <
> ig...@encompasscorporation.com> wrote:
>
>> On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov <
>> ig...@encompasscorporation.com> wrote:
>>
>>> Hi,
>>>
>>> I have a Jetty frontend exposed for couple of ActiveMQ servers behind
>>> SSL terminating Haproxy-1.8.18. They share same storage and state via lock
>>> file and there is only one active AMQ at any given time. I'm testing this
>>> now with dynamic backend using Consul DNS resolution:
>>>
>>> # dig +short @127.0.0.1 -p 8600 activemq.service.consul
>>> 10.140.4.122
>>> 10.140.3.171
>>>
>>> # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV
>>> 1 1 61616 ip-10-140-4-122.node.dc1.consul.
>>> 1 1 61616 ip-10-140-3-171.node.dc1.consul.
>>>
>>> The backends status, the current "master":
>>>
>>> root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep
>>> java
>>> tcp0 0 0.0.0.0:81610.0.0.0:*
>>> LISTEN 5031374919617256/java
>>> tcp0 0 0.0.0.0:6161 0.0.0.0:*
>>> LISTEN 5031374919317256/java
>>>
>>> and the "slave":
>>>
>>> root@ip-10-140-4-122:~# netstat -tuplen | grep java
>>>
>>> So the service ports are not available on the second one.
>>>
>>> This is the relevant part of the HAP config that I think might be of
>>> interest:
>>>
>>> global
>>> server-state-base /var/lib/haproxy
>>> server-state-file hap_state
>>>
>>> defaults
>>> load-server-state-from-file global
>>> default-server init-addrlast,libc,none
>>>
>>> listen amq
>>> bind ... ssl crt ...
>>> mode http
>>>
>>> option prefer-last-server
>>>
>>> # when this is on the backend is down
>>> #option tcp-check
>>>
>>> default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s
>>> maxconn 25 maxqueue 256 weight 100
>>>
>>> # working but both show as up
>>> server-template amqs 2 activemq.service.consul:8161 check
>>>
>>> # working old static setup
>>> #server ip-10-140-3-171 10.140.3.171:8161 check
>>> #server ip-10-140-4-122 10.140.4.122:8161 check
>>>
>>> This is working but the thing is I see both servers as UP in the HAP
>>> console:
>>> [image: amqs.png]
>>> Is this normal for this kind of setup or I'm doing something wrong?
>>>
>>> Another observation, when I have tcp check enabled like:
>>>
>>> option tcp-check
>>>
>>> the way I had it with the static lines like:
>>>
>>> server ip-10-140-3-171 10.140.3.171:8161 check
>>> server ip-10-140-4-122 10.140.4.122:8161 check
>>>
>>> then both servers show as down.
>>> Thanks in advance for any kind of input.
>>> Igor
>>>
>>> Ok, the state has changed now, I have correct state on one haproxy:
>>
>> [image: amqs_hap1.png]
>> but on the second the whole backend is down:
>>
>> [image: amqs_hap2.png]
>> I confirmed via telnet that I can connect to port 8161 to the running amq
>> server from both haproxy servers.
>>
>>
>
>
> Hi Igor,
>
> You're using the libc resolver function at startup time to resolve your
> backend, this is not recommended integration with Consul.
> You will find some good explanations in this blog article:
>
> https://www.haproxy.com/fr/blog/haproxy-and-consul-with-dns-for-service-discovery/
>
> Basically, you should first create a "resolvers" section, in order to
> allow HAProxy to perform DNS resolution at runtime too.
>
> resolvers consul
> nameserver consul 127.0.0.1:8600
> accepted_payload_size 8192
>
> Then, you need to adjust your server-template line, like this:
> server-template amqs 10 _activemq._tcp.service.consul resolvers consul
> resolve-prefer ipv4 check
>
> In the example above, I am using on purpose the SRV records, because
> HAProxy supports it and it will use all information available in the
> response to update server's IP, weight and port.
>
> I hope this will help you.
>
> Baptiste
>
All sorted now. For the record and those interested here is my setup:
Haproxy:
global
server-state-base /var/lib/haproxy
server-state-file hap_state
defaults
load-server-state-from-file global
default-server init-addrlast,libc,none
resolvers consul
nameserver consul 127.0.0.1:8600
accepted_payload_size 8192
resolve_retries 30
timeout resolve 1s
timeout retry 2s
hold valid30s
hold other30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold obsolete 30s
listen jetty
bind _port_ ssl crt ...
mode http
option forwardfor except 127.0.0.1 header X-Forwarded-For
option http-ignore-probes
option prefer-last-server
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s
maxconn 25 maxqueue 256 weight 100
server-template jettys 2 _jetty._tcp.service.consul resolvers consul
resolve-prefer ipv4 check
Consul:
---
{
"services": [
{
Re: Using server-template for DNS resolution
On Fri, Feb 8, 2019 at 6:09 AM Igor Cicimov wrote: > On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov < > ig...@encompasscorporation.com> wrote: > >> Hi, >> >> I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL >> terminating Haproxy-1.8.18. They share same storage and state via lock file >> and there is only one active AMQ at any given time. I'm testing this now >> with dynamic backend using Consul DNS resolution: >> >> # dig +short @127.0.0.1 -p 8600 activemq.service.consul >> 10.140.4.122 >> 10.140.3.171 >> >> # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV >> 1 1 61616 ip-10-140-4-122.node.dc1.consul. >> 1 1 61616 ip-10-140-3-171.node.dc1.consul. >> >> The backends status, the current "master": >> >> root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep >> java >> tcp0 0 0.0.0.0:81610.0.0.0:* >> LISTEN 5031374919617256/java >> tcp0 0 0.0.0.0:6161 0.0.0.0:* >> LISTEN 5031374919317256/java >> >> and the "slave": >> >> root@ip-10-140-4-122:~# netstat -tuplen | grep java >> >> So the service ports are not available on the second one. >> >> This is the relevant part of the HAP config that I think might be of >> interest: >> >> global >> server-state-base /var/lib/haproxy >> server-state-file hap_state >> >> defaults >> load-server-state-from-file global >> default-server init-addrlast,libc,none >> >> listen amq >> bind ... ssl crt ... >> mode http >> >> option prefer-last-server >> >> # when this is on the backend is down >> #option tcp-check >> >> default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s >> maxconn 25 maxqueue 256 weight 100 >> >> # working but both show as up >> server-template amqs 2 activemq.service.consul:8161 check >> >> # working old static setup >> #server ip-10-140-3-171 10.140.3.171:8161 check >> #server ip-10-140-4-122 10.140.4.122:8161 check >> >> This is working but the thing is I see both servers as UP in the HAP >> console: >> [image: amqs.png] >> Is this normal for this kind of setup or I'm doing something wrong? >> >> Another observation, when I have tcp check enabled like: >> >> option tcp-check >> >> the way I had it with the static lines like: >> >> server ip-10-140-3-171 10.140.3.171:8161 check >> server ip-10-140-4-122 10.140.4.122:8161 check >> >> then both servers show as down. >> Thanks in advance for any kind of input. >> Igor >> >> Ok, the state has changed now, I have correct state on one haproxy: > > [image: amqs_hap1.png] > but on the second the whole backend is down: > > [image: amqs_hap2.png] > I confirmed via telnet that I can connect to port 8161 to the running amq > server from both haproxy servers. > > Hi Igor, You're using the libc resolver function at startup time to resolve your backend, this is not recommended integration with Consul. You will find some good explanations in this blog article: https://www.haproxy.com/fr/blog/haproxy-and-consul-with-dns-for-service-discovery/ Basically, you should first create a "resolvers" section, in order to allow HAProxy to perform DNS resolution at runtime too. resolvers consul nameserver consul 127.0.0.1:8600 accepted_payload_size 8192 Then, you need to adjust your server-template line, like this: server-template amqs 10 _activemq._tcp.service.consul resolvers consul resolve-prefer ipv4 check In the example above, I am using on purpose the SRV records, because HAProxy supports it and it will use all information available in the response to update server's IP, weight and port. I hope this will help you. Baptiste
Re: Using server-template for DNS resolution
On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov wrote: > Hi, > > I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL > terminating Haproxy-1.8.18. They share same storage and state via lock file > and there is only one active AMQ at any given time. I'm testing this now > with dynamic backend using Consul DNS resolution: > > # dig +short @127.0.0.1 -p 8600 activemq.service.consul > 10.140.4.122 > 10.140.3.171 > > # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV > 1 1 61616 ip-10-140-4-122.node.dc1.consul. > 1 1 61616 ip-10-140-3-171.node.dc1.consul. > > The backends status, the current "master": > > root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep > java > tcp0 0 0.0.0.0:81610.0.0.0:* > LISTEN 5031374919617256/java > tcp0 0 0.0.0.0:6161 0.0.0.0:* > LISTEN 5031374919317256/java > > and the "slave": > > root@ip-10-140-4-122:~# netstat -tuplen | grep java > > So the service ports are not available on the second one. > > This is the relevant part of the HAP config that I think might be of > interest: > > global > server-state-base /var/lib/haproxy > server-state-file hap_state > > defaults > load-server-state-from-file global > default-server init-addrlast,libc,none > > listen amq > bind ... ssl crt ... > mode http > > option prefer-last-server > > # when this is on the backend is down > #option tcp-check > > default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s > maxconn 25 maxqueue 256 weight 100 > > # working but both show as up > server-template amqs 2 activemq.service.consul:8161 check > > # working old static setup > #server ip-10-140-3-171 10.140.3.171:8161 check > #server ip-10-140-4-122 10.140.4.122:8161 check > > This is working but the thing is I see both servers as UP in the HAP > console: > [image: amqs.png] > Is this normal for this kind of setup or I'm doing something wrong? > > Another observation, when I have tcp check enabled like: > > option tcp-check > > the way I had it with the static lines like: > > server ip-10-140-3-171 10.140.3.171:8161 check > server ip-10-140-4-122 10.140.4.122:8161 check > > then both servers show as down. > Thanks in advance for any kind of input. > Igor > > Ok, the state has changed now, I have correct state on one haproxy: [image: amqs_hap1.png] but on the second the whole backend is down: [image: amqs_hap2.png] I confirmed via telnet that I can connect to port 8161 to the running amq server from both haproxy servers.
Using server-template for DNS resolution
Hi, I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL terminating Haproxy-1.8.18. They share same storage and state via lock file and there is only one active AMQ at any given time. I'm testing this now with dynamic backend using Consul DNS resolution: # dig +short @127.0.0.1 -p 8600 activemq.service.consul 10.140.4.122 10.140.3.171 # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV 1 1 61616 ip-10-140-4-122.node.dc1.consul. 1 1 61616 ip-10-140-3-171.node.dc1.consul. The backends status, the current "master": root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep java tcp0 0 0.0.0.0:81610.0.0.0:* LISTEN 5031374919617256/java tcp0 0 0.0.0.0:6161 0.0.0.0:* LISTEN 5031374919317256/java and the "slave": root@ip-10-140-4-122:~# netstat -tuplen | grep java So the service ports are not available on the second one. This is the relevant part of the HAP config that I think might be of interest: global server-state-base /var/lib/haproxy server-state-file hap_state defaults load-server-state-from-file global default-server init-addrlast,libc,none listen amq bind ... ssl crt ... mode http option prefer-last-server # when this is on the backend is down #option tcp-check default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 25 maxqueue 256 weight 100 # working but both show as up server-template amqs 2 activemq.service.consul:8161 check # working old static setup #server ip-10-140-3-171 10.140.3.171:8161 check #server ip-10-140-4-122 10.140.4.122:8161 check This is working but the thing is I see both servers as UP in the HAP console: [image: amqs.png] Is this normal for this kind of setup or I'm doing something wrong? Another observation, when I have tcp check enabled like: option tcp-check the way I had it with the static lines like: server ip-10-140-3-171 10.140.3.171:8161 check server ip-10-140-4-122 10.140.4.122:8161 check then both servers show as down. Thanks in advance for any kind of input. Igor
