Hello,
Sometimes PUT/POST request bodies don't get proxied. Headers are sent
through, server then waits for the body to be sent but the body data never
arrives and the socket times out after 5 minutes.
This seems to be a regression in 1.8, reverting to 1.7.9 fixes the issue.
haproxy version: 1.8.7. 3 backend servers. Using Lua on all the request
paths.
nbproc > 1 and not using threads.
Normally the setup handles 500k to 1.5m successful PUT/POST requests per
day.
About 5-10 or so end up failing as described above.
Some config settings:
option http-server-close
option log-health-checks
option log-separate-errors
option redispatch
retries 4
timeout client 150000
timeout connect 5000
timeout queue 5000
timeout server 3600000
More details about the failing requests:
Body size: about 30k-50k
Tw = 0
Tc = 0 in 90% and Tc=1 in 10% of the cases
Tq is about 5-10 on average (with some outliers)
actconn/feconn: 140 on average
beconn: 30 on average
Termination state: ---- for all requests
None are http/2
Things we tried which failed to fix the issue:
Upgraded to haproxy 1.8.8 + git up to
80e179128cfd78d95cdebf7195fd21299e7931b6
Disabled HTTP/2. None of the failing connections are HTTP/2, but was trying
to
test the theory that successful HTTP/2 connections somehow altered the
internal state of the server and affected those failing requests.
The only thing that fixed the issue was downgrading to haproxy 1.7.9
Thank you,
-Nick
/usr/sbin/haproxy -vv
HA-Proxy version 1.8.7 2018/04/07
Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS = USE_SLZ=1 USE_REGPARM=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1
USE_SYSTEMD=1 USE_STATIC_PCRE2=1 USE_PCRE2_JIT=1 USE_TFO=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
Running on OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE2 version : 10.30 2017-08-14
PCRE2 library supports JIT : yes
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
