Hi,
I'm using haproxy on centos 6.5 KVM virtual machines to loadbalance
some ldap traffic. Both virtual servers (haproxy and ldap server) are
running on the same KVM host (for testing I disabled the other ldap
servers in the balanced setup).
Now I'm seeing in the error logs all the time connection resets during
transfer (SD state), even while the traffic is really low (still in
testing phase). I repackaged 1.5.3 for centos to see if this resolves
the issue, but it does not. I should add that the ldap client seems to
be working just fine.
I already tried playing with sysctl settings, but to no avail, so I
could use some help here :-)
Some info, settings and logs:
My basic haproxy config:
global
daemon
maxconn 4096
log 127.0.0.1 local3
stats socket /tmp/haproxy
defaults
#timeout connect 5000ms
#timeout client 50000ms
#timeout server 50000ms
timeout connect 3s
#timeout client 5m
#timeout server 5m
timeout client 3605s
timeout server 3605s
log global
option tcplog
option dontlog-normal
option redispatch
# Ldap
listen in-389
maxconn 4000
bind 172.18.235.96:389
mode tcp
balance leastconn
option ldap-check
option log-health-checks
option srvtcpka
server server1 a.b.c.d:389 maxconn 1024 weight 50 check
server server2 e.f.g.h:389 maxconn 1024 weight 50 check
server server3 i.j.k.l:389 maxconn 1024 weight 10 check
server server4 m.n.o.p:389 maxconn 1024 weight 10 check
echo "show info" | socat unix-connect:/tmp/haproxy stdio
Name: HAProxy
Version: 1.5.3
Release_date: 2014/07/25
Nbproc: 1
Process_num: 1
Pid: 28679
Uptime: 0d 0h06m03s
Uptime_sec: 363
Memmax_MB: 0
Ulimit-n: 8233
Maxsock: 8233
Maxconn: 4096
Hard_maxconn: 4096
CurrConns: 55
CumConns: 538
CumReq: 557
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 1
ConnRateLimit: 0
MaxConnRate: 28
SessRate: 1
SessRateLimit: 0
MaxSessRate: 28
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 69
Run_queue: 1
Idle_pct: 100
[root@bpmgt0002lb a540208]# echo "show pools" | socat
unix-connect:/tmp/haproxy stdio
Dumping pools usage. Use SIGQUIT to flush them.
- Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 3 users [SHARED]
- Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool channel (80 bytes) : 116 allocated (9280 bytes), 114 used, 1
users [SHARED]
- Pool task (112 bytes) : 71 allocated (7952 bytes), 70 used, 1 users [SHARED]
- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool connection (320 bytes) : 116 allocated (37120 bytes), 113
used, 1 users [SHARED]
- Pool hdr_idx (416 bytes) : 2 allocated (832 bytes), 1 used, 1 users [SHARED]
- Pool session (864 bytes) : 58 allocated (50112 bytes), 57 used, 1
users [SHARED]
- Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
- Pool buffer (16416 bytes) : 116 allocated (1904256 bytes), 114
used, 1 users [SHARED]
Total: 10 pools, 2009712 bytes allocated, 1974368 used.
echo "show stat" | socat unix-connect:/tmp/haproxy stdio (disabled
some backends for easier testing):
in-389,FRONTEND,,,57,60,4000,1050,1109359,55405656,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,1,0,28,,,,,,,,,,,0,0,0,,,0,0,0,0,,,,,,,,
in-389,server1,0,0,51,54,1024,1032,1096497,55364671,,0,,0,833,0,0,UP,50,1,0,0,0,1219,0,,1,2,1,,1032,,2,1,,28,L7OK,0,0,,,,,,,0,,,,0,833,,,,,1,Success,,0,0,0,12129,
in-389,server2,0,0,4,4,1024,12,8120,26848,,0,,0,8,0,0,MAINT,50,1,0,0,1,1174,1174,,1,2,2,,12,,2,0,,3,L7OK,0,0,,,,,,,0,,,,0,8,,,,,1179,,,0,0,0,2,
in-389,server3,0,0,1,1,1024,2,1356,3510,,0,,0,0,0,0,MAINT,10,1,0,0,1,1174,1174,,1,2,3,,2,,2,0,,1,L7OK,0,7,,,,,,,0,,,,0,0,,,,,1218,,,0,1,0,1,
in-389,server4,0,0,1,1,1024,4,3386,10627,,0,,0,2,0,0,MAINT,10,1,0,0,1,1174,1174,,1,2,4,,4,,2,0,,1,L7OK,0,7,,,,,,,0,,,,0,2,,,,,1215,,,0,1,0,2,
in-389,BACKEND,0,0,57,60,400,1050,1109359,55405656,0,0,,0,843,0,0,UP,50,1,0,,0,1219,0,,1,2,0,,1050,,1,1,,28,,,,,,,,,,,,,,0,843,0,0,0,0,1,,,0,0,0,12130,
My extra sysctl settings on the virtual loadbalancer:
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 5000
net.core.netdev_max_backlog = 5000
net.ipv4.ip_local_port_range = 1025 65000
net.core.rmem_max=12582912
net.core.wmem_max=12582912
net.ipv4.tcp_rmem= 10240 87380 12582912
net.ipv4.tcp_wmem= 10240 87380 12582912
Some example loglines:
[10/Sep/2014:15:21:52.236] in-389 in-389/server1 1/0/281 3356 SD
58/57/57/51/0 0/0
[10/Sep/2014:15:22:04.211] in-389 in-389/server1 1/0/281 3356 SD
58/57/57/51/0 0/0
The output of "netstat -i" shows no errors on both haproxy and ldap
servers (virtio network driver is being used).
However, the tcp resets are showing up in netstat.
For the haproxy server:
Tcp:
1634017 active connections openings
910828 passive connection openings
732 failed connection attempts
186392 connection resets received
126 connections established
165215192 segments received
158185858 segments send out
194179 segments retransmited
426 bad segments received.
1238664 resets sent
Udp:
734717 packets received
173 packets to unknown port received.
0 packet receive errors
1520025 packets sent
UdpLite:
TcpExt:
4639 invalid SYN cookies received
234010 TCP sockets finished time wait in fast timer
481844 delayed acks sent
5124 delayed acks further delayed because of locked socket
Quick ack mode was activated 6322 times
618224 packets directly queued to recvmsg prequeue.
1629109 packets directly received from backlog
47034427 packets directly received from prequeue
96825479 packets header predicted
49243 packets header predicted and directly queued to user
7643396 acknowledgments not containing data received
54024702 predicted acknowledgments
1868 times recovered from packet loss due to fast retransmit
68518 times recovered from packet loss due to SACK data
Detected reordering 24 times using FACK
Detected reordering 22 times using SACK
TCPDSACKUndo: 875
2874 congestion windows recovered after partial ack
73099 TCP data loss events
TCPLostRetransmit: 1588
20 timeouts after reno fast retransmit
620 timeouts after SACK recovery
202 timeouts in loss state
147422 fast retransmits
9739 forward retransmits
12133 retransmits in slow start
14567 other TCP timeouts
TCPRenoRecoveryFail: 372
373 sack retransmits failed
6260 DSACKs sent for old packets
3299 DSACKs received
876798 connections reset due to unexpected data
183198 connections reset due to early user close
685 connections aborted due to timeout
And for the ldap server:
Tcp:
21001 active connections openings
1263366 passive connection openings
297 failed connection attempts
948390 connection resets received
61 connections established
48712376 segments received
73981815 segments send out
7113 segments retransmited
19 bad segments received.
478141 resets sent
Udp:
176452 packets received
86 packets to unknown port received.
0 packet receive errors
1055678 packets sent
UdpLite:
TcpExt:
15894 invalid SYN cookies received
48 resets received for embryonic SYN_RECV sockets
17739 TCP sockets finished time wait in fast timer
319947 delayed acks sent
6394 delayed acks further delayed because of locked socket
Quick ack mode was activated 23 times
68065 packets directly queued to recvmsg prequeue.
10146 packets directly received from backlog
513207 packets directly received from prequeue
5543973 packets header predicted
426 packets header predicted and directly queued to user
4391742 acknowledgments not containing data received
35677710 predicted acknowledgments
1688 times recovered from packet loss due to SACK data
TCPDSACKUndo: 20
1103 congestion windows recovered after partial ack
1578 TCP data loss events
TCPLostRetransmit: 14
55 timeouts after SACK recovery
1 timeouts in loss state
2936 fast retransmits
135 forward retransmits
30 retransmits in slow start
2058 other TCP timeouts
1 sack retransmits failed
23 DSACKs sent for old packets
70 DSACKs received
2304 connections reset due to unexpected data
457369 connections reset due to early user close
30 connections aborted due to timeout
Any hints are very much appreciated. If more info is needed, let me know.
Franky
