Hi,
I believe I’ve found a bug in HAProxy 1.6.3. If both the resolvers feature and
the seamless reload/state saving feature are both used, it seems that the
backends are considered permanently unreachable.
Here’s my config;
global
stats socket u...@haproxy.sock level admin
defaults
load-server-state-from-file local
timeout client 15s
timeout connect 5s
timeout queue 5s
timeout server 1h
resolvers dns
nameserver ns0 8.8.8.8:53
frontend fk
bind :8000
default_backend bk
backend bk
option httpchk GET /
server rnewson rnewson.cloudant.com:5984 check inter 7s resolvers dns
—
here's the build details;
HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = generic
CPU = generic
CC = clang
CFLAGS =
OPTIONS = USE_ZLIB=1 USE_POLL=1 USE_KQUEUE=1 USE_OPENSSL=1 USE_LUA=1
USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): no
Built with zlib version : 1.2.5
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2e 3 Dec 2015
Running on OpenSSL version : OpenSSL 1.0.2e 3 Dec 2015
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
-
On first use, this works fine, you can hit port 8000 and get the expected
result;
➜ ~ curl 127.0.0.1:8000
{"couchdb":"Welcome","version":"1.0.2","cloudant_build":"2530"}
If you then save the state of the 'bk' backend to a file called 'bk';
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state
srv_uweight srv_iweight srv_time_since_last_change srv_check_status
srv_check_result srv_check_health srv_check_state srv_agent_state
bk_f_forced_id srv_f_forced_id
3 bk 1 rnewson 159.253.138.60 2 0 1 1 38 15 3 4 6 0 0 0
and reload HAProxy, you will instead get an empty reply and HAProxy warns;
Connect() failed for backend bk: no free ports.
If you comment out either the load-server-state-from-file line or remove
'resolvers dns' from the end of the server line, then the backend works
correctly.
The same behaviour with 1.7-dev1, and I encounter the issue on Linux and OSX.
It’s worth noting that "show servers state" returns nothing if there are a lot
of backends, hence I’m using the 'local' option.
B.
