Recently I've moved several servers from haproxy 1.7.x to 1.8.x I have
a setup with nghttpx handling h2 (haproxy connects to nghttpx via unix
socket which handles h2 and connects back to haproxy with plain
http/1.1 also through unix socket).
After the upgrade I wanted to switch to native h2 supported by
haproxy. Unfortunately, it seems that over time haproxy is
accumulating sockets in CLOSE_WAIT state. Currently, after 12h I have
5k connections in this state. All of them have non-zero Recv-Q and
zero Send-Q. netstat -ntpa shows something like this:
tcp 1 0 IP:443 IP:28032 CLOSE_WAIT 115495/haproxy
tcp 35 0 IP:443 IP:49531 CLOSE_WAIT 115495/haproxy
tcp 507 0 IP:443 IP:31938 CLOSE_WAIT 115495/haproxy
tcp 134 0 IP:443 IP:49672 CLOSE_WAIT 115495/haproxy
tcp 732 0 IP:443 IP:3180 CLOSE_WAIT 115494/haproxy
tcp 746 0 IP:443 IP:39731 CLOSE_WAIT 115494/haproxy
tcp 35 0 IP:443 IP:62986 CLOSE_WAIT 115495/haproxy
tcp 585 0 IP:443 IP:51318 CLOSE_WAIT 115493/haproxy
tcp 100 0 IP:443 IP:60449 CLOSE_WAIT 115493/haproxy
tcp 35 0 IP:443 IP:1274 CLOSE_WAIT 115494/haproxy
..
Those are all frontend connections. Reloading haproxy removes those
connections, but only after hard-stop-after kicks in and old processes
are killed. Disabling native h2 support and switching back to nghttpx
makes the problem disappear.
This kinda seems like the socket was closed on the writing side, but
the client has already sent something and everything is stuck. I was
not able to reproduce the problem by myself. Any ideas how to debug
this further?
haproxy -vv (Debian package rebuilt on stretch with USE_TFO):
HA-Proxy version 1.8.9-1~tsg9+1 2018/05/21
Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fdebug-prefix-map=/root/haproxy-1.8.9=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
config
global
log /dev/log daemon warning
log-send-hostname
chroot /var/lib/haproxy
maxconn 65536
user haproxy
group haproxy
daemon
nbproc 4
stats socket /var/run/haproxy/stats.socket user haproxy mode 0640
level user process 1
stats socket /var/run/haproxy/stats-1.socket user haproxy mode 0640
level admin process 1
stats socket /var/run/haproxy/stats-2.socket user haproxy mode 0640
level admin process 2
stats socket /var/run/haproxy/stats-3.socket user haproxy mode 0640
level admin process 3
stats socket /var/run/haproxy/stats-4.socket user haproxy mode 0640
level admin process 4
ssl-default-bind-ciphers
ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA
ssl-default-bind-options ssl-min-ver TLSv1.0
tune.ssl.cachesize 200000
tune.ssl.lifetime 24h
hard-stop-after 2h
unix-bind prefix /var/lib/haproxy/ mode 600 user haproxy group haproxy
defaults http
option dontlognull
option dontlog-normal
option redispatch
option tcp-smart-connect
option httplog
timeout client 60s
timeout connect 10s
timeout server 60s
timeout tunnel 10m
timeout client-fin 30s
timeout http-keep-alive 30s
timeout http-request 30s
log global
retries 3
backlog 16384
maxconn 65536
mode http
errorfile 403 /etc/haproxy/403.html
backend php
option httpchk GET /ping.php
balance roundrobin
cookie SOME_COOKIE insert indirect httponly
server hostname-1 IP:81 check inter 5000 rise 2 fall 5 weight 100 cookie 1
# more servers
frontend http
bind IP:80 transparent
bind IP:443 tfo transparent ssl alpn h2,http/1.1 curves X25519:P-256
tls-ticket-keys FILE crt FILE
http-request set-header X-Forwarded-For %ci unless LOCALHOST
http-request set-header X-Forwarded-Proto https unless { dst_port 80 }
http-request set-header X-Forwarded-Proto http if { dst_port 80 }
default_backend php
--
Janusz Dziemidowicz
