Hi! I faced a problem dealing with l4 (tcp mode) haproxy-based proxy over Graphite's component receiving metrics from clients and clients who are connecting just to send one or two Graphite-metrics and disconnecting right after.
It looks like this 1. Client connects to haproxy (SYN/SYN-ACK/ACK) 2. Client sends one line of metric 3. Haproxy acknowledges receiving this line (ACK to client) 4. Client disconnects (FIN, FIN-ACK) 5. Haproxy writes 1/-1/0/0 CC-termination state to log without even trying to connect to a backend and send client's data to it. 6. Metric is lost :( If the client is slow enough between steps 1 and 2 or it sends a bunch of metrics so haproxy has time to connect to a backend – everything works like a charm. How can I deal with these send-and-forget clients? Example. First column is a time delta in seconds between packets 0.000000 client haproxy TCP 100 58664 → 2024 [SYN] Seq=0 Win=65535 Len=0 MSS=1220 WS=64 TSval=904701415 TSecr=0 SACK_PERM=1 0.000015 haproxy client TCP 96 2024 → 58664 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=8840 SACK_PERM=1 TSval=276942420 TSecr=904701415 WS=2048 0.019105 client haproxy TCP 88 58664 → 2024 [ACK] Seq=1 Ack=1 Win=131264 Len=0 TSval=904701434 TSecr=276942420 0.000090 client haproxy TCP 151 58664 → 2024 [PSH, ACK] Seq=1 Ack=1 Win=131264 Len=63 TSval=904701434 TSecr=276942420 0.000012 haproxy client TCP 88 2024 → 58664 [ACK] Seq=1 Ack=64 Win=65536 Len=0 TSval=276942439 TSecr=904701434 0.000150 client haproxy TCP 88 58664 → 2024 [FIN, ACK] Seq=64 Ack=1 Win=131264 Len=0 TSval=904701434 TSecr=276942420 0.000058 haproxy client TCP 88 2024 → 58664 [FIN, ACK] Seq=1 Ack=65 Win=65536 Len=0 TSval=276942439 TSecr=904701434 haproxy -vv HA-Proxy version 2.2.8-1 2021/01/28 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2025. Known bugs: http://www.haproxy.org/bugs/bugs-2.2.8.html Running on: Linux 4.19.91-22 #1 SMP Wed Dec 25 14:25:55 UTC 2019 x86_64 Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -O2 -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_SYSTEMD=1 DEBUG = Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=32). Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.1 Built with network namespace support. Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.21 2016-01-12 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with gcc compiler version 5.4.0 20160609 Built with the Prometheus exporter as a service Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) fcgi : mode=HTTP side=BE mux=FCGI <default> : mode=HTTP side=FE|BE mux=H1 h2 : mode=HTTP side=FE|BE mux=H2 <default> : mode=TCP side=FE|BE mux=PASS Available services : prometheus-exporter Available filters : [SPOE] spoe [COMP] compression [TRACE] trace [CACHE] cache [FCGI] fcgi-app -- Best regards, Maksim Kupriianov
