Hi!
I have a Consul as service discovery tool and HAProxy as load balancer.
In Consul registered a service running on a number of servers, and this
service can be scaled by adding and removing nodes and by moving nodes from
one server to another.
Consul has DNS service which randomizes responses for services like that:
[bux] michep@bux:~$ dig +short mfm-monitor-opentsdb.service.mfmconsul
10.182.161.239
10.182.161.152
10.182.161.240
10.182.161.92
[bux] michep@bux:~$ dig +short mfm-monitor-opentsdb.service.mfmconsul
10.182.161.92
10.182.161.152
10.182.161.240
10.182.161.239
In HAProxy 1.8.3 im using server-template configuration, like that:
resolvers dns
nameserver dns1 ${HAPROXY_NAMESERVER}
hold valid 2s
backend tsdb_backend_query
server-template tsdb_query 5
mfm-monitor-opentsdb.service.mfmconsul:4242 check resolvers dns inter
1000
And in that case I get alot of warinings in haproxy log:
time="2018-02-02T15:44:32+03:00" level=info msg="[WARNING] 032/154432
(32983) : tsdb_backend_query/tsdb_query1 changed its IP from
10.182.161.240 to 10.182.161.239 by DNS cache."
job=mfm-monitor-haproxy pid=32983
time="2018-02-02T15:44:42+03:00" level=info msg="[WARNING] 032/154442
(32983) : tsdb_backend_query/tsdb_query1 changed its IP from
10.182.161.239 to 10.182.161.240 by DNS cache."
job=mfm-monitor-haproxy pid=32983
time="2018-02-02T15:44:46+03:00" level=info msg="[WARNING] 032/154446
(32983) : tsdb_backend_query/tsdb_query3 changed its IP from
10.182.161.152 to 10.182.161.239 by DNS cache."
job=mfm-monitor-haproxy pid=32983
time="2018-02-02T15:44:50+03:00" level=info msg="[WARNING] 032/154450
(32983) : tsdb_backend_query/tsdb_query2 changed its IP from
10.182.161.92 to 10.182.161.152 by DNS cache." job=mfm-monitor-haproxy
pid=32983
time="2018-02-02T15:44:52+03:00" level=info msg="[WARNING] 032/154452
(32983) : tsdb_backend_query/tsdb_query3 changed its IP from
10.182.161.239 to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy
pid=32983
time="2018-02-02T15:44:56+03:00" level=info msg="[WARNING] 032/154456
(32983) : tsdb_backend_query/tsdb_query1 changed its IP from
10.182.161.240 to 10.182.161.239 by DNS cache."
job=mfm-monitor-haproxy pid=32983
time="2018-02-02T15:45:00+03:00" level=info msg="[WARNING] 032/154500
(32983) : tsdb_backend_query/tsdb_query3 changed its IP from
10.182.161.92 to 10.182.161.240 by DNS cache." job=mfm-monitor-haproxy
pid=32983
time="2018-02-02T15:45:02+03:00" level=info msg="[WARNING] 032/154502
(32983) : tsdb_backend_query/tsdb_query3 changed its IP from
10.182.161.240 to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy
pid=32983
time="2018-02-02T15:45:04+03:00" level=info msg="[WARNING] 032/154504
(32983) : tsdb_backend_query/tsdb_query2 changed its IP from
10.182.161.152 to 10.182.161.240 by DNS cache."
job=mfm-monitor-haproxy pid=32983
time="2018-02-02T15:45:06+03:00" level=info msg="[WARNING] 032/154506
(32983) : tsdb_backend_query/tsdb_query1 changed its IP from
10.182.161.239 to 10.182.161.152 by DNS cache."
job=mfm-monitor-haproxy pid=32983
time="2018-02-02T15:45:10+03:00" level=info msg="[WARNING] 032/154510
(32983) : tsdb_backend_query/tsdb_query3 changed its IP from
10.182.161.92 to 10.182.161.239 by DNS cache." job=mfm-monitor-haproxy
pid=32983
time="2018-02-02T15:45:18+03:00" level=info msg="[WARNING] 032/154518
(32983) : tsdb_backend_query/tsdb_query3 changed its IP from
10.182.161.239 to 10.182.161.92 by DNS cache." job=mfm-monitor-haproxy
pid=32983
time="2018-02-02T15:45:20+03:00" level=info msg="[WARNING] 032/154520
(32983) : tsdb_backend_query/tsdb_query2 changed its IP from
10.182.161.240 to 10.182.161.239 by DNS cache."
job=mfm-monitor-haproxy pid=32983
This isn’t really break the service, but I think this is not quite normal.
Any advise on how to resolve this issue?
--
Mike Chepaykin
