I found the problem this weekend. It's not HAproxy that's at fault, rather syslog-ng was configured to never close it's tcp connection. So syslog-ng would establish a single tcp connection to haproxy and then never re-connect. You can't load balance if there are no new connections to load balance.
I told you it was something simple and I was an idiot. :-) I've configured syslog-ng to close it's connection and flush every 100 lines (still playing with that) and it now successfully moves between all three graylog hosts. It's not quite what I had in mind, but at least it's not the fault of HAproxy. Thanks!! Michael -- --------------------------------o--------------------------------- Michael H. Martel | Director of Data Center Administration michael.mar...@vsc.edu | Systems and Security Administrator Vermont State Colleges | PH:802-224-3010 FX:802-224-3035 -----Original Message----- From: Martel, Michael H [mailto:michael.mar...@vsc.edu] Sent: Saturday, October 1, 2016 6:59 AM To: haproxy@formilux.org Subject: Using HAProxy to front end Graylog servers Greetings! I'm trying to use HAproxy to front end my three graylog servers. It works, in the sense that it connects to one of them and routes traffic. But I was trying to get it to load balance between all three. I have haproxy doing this for some websites, so I thought it would be similar, but I'm missing something. I'm running CentOS 7, selinux disabled. HAproxy v1.5 from epel or version 1.6.9 from source, both have the same results. I've verified that the firewalls are correct, and if I fail one of the graylog servers, HAproxy sees it fail and will move on to another server. I am positive it's something that I'm not doing right, but I can't for the life of me find it. Any thoughts ? [root@graylog haproxy]# haproxy -vv HA-Proxy version 1.6.9 2016/08/30 Copyright 2000-2016 Willy Tarreau <wi...@haproxy.org> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.7 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.32 2012-11-30 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built without Lua support Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. And my config file looks like this : global log 192.168.1.62 local2 debug chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close #option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend graylog bind 192.168.1.62:10514 mode tcp option tcplog # option logasap log global default_backend graylog_backend timeout client 1s backend graylog_backend mode tcp balance roundrobin # balance leastconn option httpchk GET /system/lbstatus server graylog1 192.168.1.63:10515 check port 12900 server graylog2 192.168.1.64:10515 check port 12900 server graylog3 192.168.1.65:10515 check port 12900 timeout connect 10s timeout server 1s frontend stats bind 192.168.1.62:80 mode http option httplog option logasap log global maxconn 10 timeout client 100s default_backend stats_backend backend stats_backend mode http timeout server 100s timeout connect 100s timeout queue 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:********* stats uri /haproxy?stats Thanks!! Michael -- --------------------------------o--------------------------------- Michael H. Martel | Director of Data Center Administration michael.mar...@vsc.edu | Systems and Security Administrator Vermont State Colleges | PH:802-224-3010 FX:802-224-3035
