We've found that just switching back to 1.7.9 makes this issue disappear. It's
a really hard one to pin down though. It's quite rare and we've had no luck
building a reliable reproducing script.
Is there anything more we can provide to give insight into why haproxy might
sometimes hang during request processing?
B.
> On 8 May 2018, at 16:54, Nick Vatamaniuc <vatam...@gmail.com> wrote:
>
> Hello,
>
> Sometimes PUT/POST request bodies don't get proxied. Headers are sent
> through, server then waits for the body to be sent but the body data never
> arrives and the socket times out after 5 minutes.
>
> This seems to be a regression in 1.8, reverting to 1.7.9 fixes the issue.
>
> haproxy version: 1.8.7. 3 backend servers. Using Lua on all the request paths.
> nbproc > 1 and not using threads.
>
> Normally the setup handles 500k to 1.5m successful PUT/POST requests per day.
> About 5-10 or so end up failing as described above.
>
> Some config settings:
>
> option http-server-close
> option log-health-checks
> option log-separate-errors
> option redispatch
> retries 4
> timeout client 150000
> timeout connect 5000
> timeout queue 5000
> timeout server 3600000
>
> More details about the failing requests:
>
> Body size: about 30k-50k
> Tw = 0
> Tc = 0 in 90% and Tc=1 in 10% of the cases
> Tq is about 5-10 on average (with some outliers)
> actconn/feconn: 140 on average
> beconn: 30 on average
> Termination state: ---- for all requests
> None are http/2
>
> Things we tried which failed to fix the issue:
>
> Upgraded to haproxy 1.8.8 + git up to 80e179128cfd78d95cdebf7195fd21299e7931b6
>
> Disabled HTTP/2. None of the failing connections are HTTP/2, but was trying to
> test the theory that successful HTTP/2 connections somehow altered the
> internal state of the server and affected those failing requests.
>
> The only thing that fixed the issue was downgrading to haproxy 1.7.9
>
> Thank you,
> -Nick
>
>
>
> /usr/sbin/haproxy -vv
>
> HA-Proxy version 1.8.7 2018/04/07
> Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org>
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
> -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
> OPTIONS = USE_SLZ=1 USE_REGPARM=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1
> USE_SYSTEMD=1 USE_STATIC_PCRE2=1 USE_PCRE2_JIT=1 USE_TFO=1
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> Built with OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
> Running on OpenSSL version : OpenSSL 1.1.0h 27 Mar 2018
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
> Built with Lua version : Lua 5.3.4
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Encrypted password support via crypt(3): yes
> Built with multi-threading support.
> Built with PCRE2 version : 10.30 2017-08-14
> PCRE2 library supports JIT : yes
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"),
> raw-deflate("deflate"), gzip("gzip")
> Built with network namespace support.
> Available polling systems :
> epoll : pref=300, test result OK
> poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
