Hi everyone, writing to get some help on a setup I am building with haproxy.
Part of the setup is a content inspection of the tcp payload (binary stream), for which the load balancing will be done. Testing with content inspection based on simple ACL pattern matches but also tried evaluating the payload in LUA scripts. Where the latter is my personal preference. In the end the incoming requests should be accepted/rejected, based on the payload evaluation result. My target is to process multiple hundreds of simultaneous requests at peak times, which *ALL* should undergo a payload inspection for the initial request. Scenario will also terminate TLS later on, but this should make no difference for the inspection (at least to my understanding) My problem is that the "req.payload(0,10)" fetch, which I am using for that purpose, does not seem to reliably have access to the payload at all times. So far I was not able to find out, what the cause of that could be. There were several mitigation hints on that problem, but somehow I am failing to get it to work. For troubleshooting I got down to a very simplistic setup, which just accesses the payload and prints it to the logfile. I am using apache benchmark "ab" to generate ingress traffic for larger batches. An apache server acts as a test backend. Please not this is just for testing purposes. The final protocol is *NOT* http. I think this is negligible atm(?) as the part I am focussing on is actual inboud/eval stuff, before the backend is contacted. So out of a 100 requests sent with "ab" about 10-50% of the requests are failing to display payload content. I also noticed, that localhost generated ab requests have a much higher chance of failing to print the payload. Have the strong feeling that the payload is trying to be accessed before its fully available to haproxy - even if its just a few bytes (testing with 2-8) Kind of lost here at the moment and I would really be grateful for any suggestions and help on that one. Is there a reasonable way to reliably "wait" for incoming requests w/o delaying the requests too much in the end? Best Regards Micha Below you can find the setup I came up with: # VERSIONS $ grep VERSION= /etc/os-release VERSION="18.04.4 LTS (Bionic Beaver)" $ grep 2.1 /etc/apt/sources.list.d/vbernat-ubuntu-haproxy-2_0-bionic.list deb http://ppa.launchpad.net/vbernat/haproxy-2.1/ubuntu bionic main $ haproxy -vv HA-Proxy version 2.1.2-1ppa1~bionic 2019/12/21 - https://haproxy.org/ Status: stable branch - will stop receiving fixes around Q1 2021. Known bugs: http://www.haproxy.org/bugs/bugs-2.1.2.html Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-HuTwKZ/haproxy-2.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1 Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=2). Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018 Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.3 Built with network namespace support. Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.31 2018-02-12 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with zlib version : 1.2.11 Running on zlib version : 1.2.11 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with the Prometheus exporter as a service Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 fcgi : mode=HTTP side=BE mux=FCGI <default> : mode=HTTP side=FE|BE mux=H1 <default> : mode=TCP side=FE|BE mux=PASS Available services : prometheus-exporter Available filters : [SPOE] spoe [CACHE] cache [FCGI] fcgi-app [TRACE] trace [COMP] compression # Standard apache from ubuntu repos, with a stripped down index.html $ dpkg -l apache2 | grep ii ii apache2 2.4.29-1ubuntu4.11 amd64 Apache HTTP Server # CONFIG (lua validation code not included, not in use at the moment) # NOTE!: removed any acl and lua usage for testing req.payload $ grep -v "#\|^$" haproxy.cfg global log stdout format raw local0 info lua-load /etc/haproxy/txn5.lua lua-load /etc/haproxy/getvar.lua tune.lua.forced-yield 10 tune.lua.maxmem 5 tune.lua.session-timeout 5s defaults timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend fe bind :8080 mode tcp default_backend be1 log global option tcplog tcp-request inspect-delay 5s tcp-request content set-var(txn.rawPayload) req.payload(0,2),hex log-format "[%t] payload=%[var(txn.rawPayload)]" backend be1 server local-apache localhost:80 check inter 2000 # TESTING: $ haproxy -f /etc/haproxy/haproxy.cfg $ ab -r -t 300 -n 100 -c 30 http://127.0.0.1:8080/ # OUTPUT haproxy logs: (...) [12/Feb/2020:09:48:31.600] payload=- [12/Feb/2020:09:48:31.600] payload=- [12/Feb/2020:09:48:31.600] payload=- [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.600] payload=4745 [12/Feb/2020:09:48:31.600] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.600] payload=4745 [12/Feb/2020:09:48:31.600] payload=- [12/Feb/2020:09:48:31.600] payload=- [12/Feb/2020:09:48:31.601] payload=4745 [12/Feb/2020:09:48:31.601] payload=4745 [12/Feb/2020:09:48:31.602] payload=- [12/Feb/2020:09:48:31.602] payload=- [12/Feb/2020:09:48:31.602] payload=- [12/Feb/2020:09:48:31.602] payload=- [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.598] payload=4745 [12/Feb/2020:09:48:31.602] payload=4745 [12/Feb/2020:09:48:31.602] payload=4745 [12/Feb/2020:09:48:31.602] payload=- [12/Feb/2020:09:48:31.602] payload=- [12/Feb/2020:09:48:31.603] payload=4745 [12/Feb/2020:09:48:31.603] payload=4745 [12/Feb/2020:09:48:31.603] payload=4745 (...) $ haproxy -f /etc/haproxy/haproxy.cfg | grep -oP "payload=.*" | sort | uniq -c 9 payload=- 91 payload=4745 $ haproxy -f /etc/haproxy/haproxy.cfg | grep -oP "payload=.*" | sort | uniq -c 26 payload=- 75 payload=4745 # payload first 2 bytes from http request is displayed correctly "GE"(T) $ echo -n "4745" | xxd -r -p GE
