hi all,
I'm working on standing up a new haproxy instance to manage redis directly
on our redis hosts since our main load-balancer does periodic reloads and
restarts for things like OCSP stapling that good ol' amnesiac HTTP handles
just fine, but longer-lived TCP connections like our redis clients don't
care too much for.
I managed to put together a configuration that works fine in local testing
(vagrant configured by test-kitchen), but for some reason when I try to
push this to staging, haproxy is refusing to start, complaining that it
can't bind to the keepalived-managed VIP. For the life of me I can't figure
out what the problem is, but hopefully someone here will be able to give me
some pointers? Thanks in advance for your help :)
The error message:
```bash
[root@redis02.stage ~]# journalctl -ln5 -u haproxy.service --no-pager
-- Logs begin at Wed 2015-05-20 22:35:37 UTC, end at Wed 2015-05-20
22:45:55 UTC. --
May 20 22:35:47 redis02.stage.iad01.treehouse systemd[1]: Starting HAProxy
Load Balancer...
May 20 22:35:47 redis02.stage.iad01.treehouse systemd[1]: Started HAProxy
Load Balancer.
May 20 22:35:47 redis02.stage.iad01.treehouse haproxy-systemd-wrapper[794]:
haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
May 20 22:35:47 redis02.stage.iad01.treehouse haproxy-systemd-wrapper[794]:
[ALERT] 139/223547 (801) : Starting proxy redis: cannot bind socket [
10.240.36.71:6379]
May 20 22:35:47 redis02.stage.iad01.treehouse haproxy-systemd-wrapper[794]:
haproxy-systemd-wrapper: exit, haproxy RC=256
```
version info:
```bash
[root@redis02.stage ~]# haproxy -vvv
HA-Proxy version 1.5.4 2014/09/02
Copyright 2000-2014 Willy Tarreau <w...@1wt.eu>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
```
the configuration:
```bash
[root@redis02.stage ~]# cat /etc/haproxy/haproxy.cfg
# Generated by Chef
# Changes will be overwritten!
global
user haproxy
group haproxy
stats socket /var/lib/haproxy/stats.sock
log /dev/log local0 info
maxconn 50000
defaults TCP
mode tcp
log global
option tcplog
option tcpka
source 10.240.36.71
listen redis
bind 10.240.36.71:6379
default-server on-marked-down shutdown-sessions
option tcp-check
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
server redis01.stage 10.240.36.27:6379 backup check inter 1000 rise 2
fall 5
server redis02.stage 10.240.36.63:6379 backup check inter 1000 rise 2
fall 5
```
listening services:
```bash
[root@redis02.stage ~]# netstat -lptn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 0.0.0.0:26379 0.0.0.0:* LISTEN
2449/redis-sentinel
tcp 0 0 10.240.36.63:6379 0.0.0.0:* LISTEN
2388/redis-server 1
tcp 0 0 127.0.0.1:3030 0.0.0.0:* LISTEN
930/ruby
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
782/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
919/master
tcp 0 0 127.0.0.1:2812 0.0.0.0:* LISTEN
784/monit
tcp6 0 0 :::26379 :::* LISTEN
2449/redis-sentinel
tcp6 0 0 :::22 :::* LISTEN
782/sshd
tcp6 0 0 ::1:25 :::* LISTEN
919/master
```
local addresses:
```bash
[root@redis02.stage ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
link/ether fa:16:3e:54:5f:49 brd ff:ff:ff:ff:ff:ff
inet 10.240.36.63/22 brd 10.240.39.255 scope global dynamic eth0
valid_lft 86280sec preferred_lft 86280sec
inet 10.240.36.71/22 brd 10.240.39.255 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe54:5f49/64 scope link
valid_lft forever preferred_lft forever
```
relevant sysctl?:
```bash
[root@redis02.stage ~]# sysctl net.ipv4.ip_nonlocal_bind
net.ipv4.ip_nonlocal_bind = 1
```
address seems to "work":
```bash
[root@redis02.stage ~]# ping -I 10.240.36.71 -c 2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.240.36.71 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=38 time=12.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=38 time=12.8 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.806/12.818/12.831/0.113 ms
[root@redis02.stage ~]# nc -l 10.240.36.71 6379
^C
```
cheers,
nathan w
