Re: Uptick in 503s with HAProxy 1.8
Hi,
Follow up from my previous email, we have new findings and the findings are as
below.
We observed endpoints that returned 400 with a termination reason of CH in 1.7
now sometimes return a 503 in 1.8 with a termination of CC. Find below
breakdown of CC and CH with their http response codes in the past 3 days.
"host","http_status_code","proxy_termination_state","_count"
"haproxy01","400","CH--","5610"
"haproxy01","503","CC--","1405"
"haproxy02","400","CH--","6967"
"haproxy03","400","CH--","7147"
"haproxy04","400","CH--","6910"
"haproxy05","400","CH--","6943"
haproxy01 is running 1.8.14 HAProxy while the rest are running 1.7.11
Note the servers are all behind the same L3/L4 load-balancer so the nature of
traffic is entirely similar.
Is there any change between 1.7 and 1.8 that will explain this new logging
behavior.
Thanks.
Abejide Ayodele
It always seems impossible until it's done. --Nelson Mandela
On Thu, Nov 1, 2018 at 10:25 AM Àbéjídé Àyodélé
wrote:
>
> Hi Friends,
>
> We upgraded to HAProxy 1.8.14 from 1.7.11 and saw an uptick in the number of
> browser related 503 requests with termination reason CC. TC, TR and TW are
> consistent across these requests and are -1,-1,0 respectively, these requests
> are proxied to nginx, the only thing that has changed so far in our setup is
> upgrading from 1.7.11 to 1.8.14. Find below an example log entry:
>
> {"proxy_server_name":"http-in","proxy_tq":"826","proxy_tw":"0","proxy_tc":"-1","proxy_tr":"-1","proxy_tt":"826","http_status_code":"503","http_bytes_read":"0","http_captured_request_cookie":"-","http_captured_response_cookie":"-","proxy_termination_state":"CC--","proxy_active_connections":"131","proxy_frontend_connections":"104","proxy_backend_connections":"27","proxy_server_connections":"27","proxy_retries":"0","proxy_server_queue":"0","proxy_backend_queue":"0","http_method":"GET","http_request_url":"/favicon.ico","ssl_version":"TLSv1.2","ssl_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
>
> I can share our config if required, I am not including it in this email
> because of it's size and hope we can narrow down to specifics that will be
> easy to share.
>
> Do folks know if there any known things that could have changed between 1.7
> and 1.8 to have introduced this uptick?
>
> Thanks!
>
> Abejide Ayodele
> It always seems impossible until it's done. --Nelson Mandela
Re: PUT/POST request bodies sometimes are not proxied in version 1.8
I believe this is similar to our experience with:
https://www.mail-archive.com/haproxy@formilux.org/msg31645.html
Abejide Ayodele
It always seems impossible until it's done. --Nelson Mandela
On Thu, Nov 8, 2018 at 10:27 AM Nick Vatamaniuc wrote:
>
> Hello,
>
> I'd like to report an issue where sometimes PUT/POST request bodies do not
> get proxied. Headers are sent then the backend server waits to receive the
> body. It never arrives and after 5 minutes the socket times out. This
> happens in all the 1.8 releases up until 1.8.14 but it doesn't happen in 1.7.
> In fact one of the ways we mitigate the issue is by reverting back to 1.7.
>
> Normally the requests go through without an issue. We see about 1M+ requests
> per day. However about 5 or 10 of those end up failing as described. Is there
> anything we can do to debug the issue further, or were there any changes that
> might lead to this specific bug in 1.8.x?
>
> More details about our setup:
> Lua is used on all requests
> nbproc > 1
> Not using threads
> Request body sizes about 30-50KB
>
> Some config settings:
> option http-server-close
> option log-health-checks
> option log-separate-errors
> option redispatch
> retries 4
> timeout client 15
> timeout connect 5000
> timeout queue 5000
> timeout server 360
>
> $ /usr/sbin/haproxy -vv
> HA-Proxy version 1.8.14-52e4d43 2018/09/20
> Copyright 2000-2018 Willy Tarreau
>
> Build options :
> TARGET = linux2628
> CPU = generic
> CC = gcc
> CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
> -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
> OPTIONS = USE_SLZ=1 USE_REGPARM=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1
> USE_SYSTEMD=1 USE_STATIC_PCRE2=1 USE_PCRE2_JIT=1 USE_TFO=1
>
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
>
> Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
> Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
> Built with Lua version : Lua 5.3.4
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Encrypted password support via crypt(3): yes
> Built with multi-threading support.
> Built with PCRE2 version : 10.31 2018-02-12
> PCRE2 library supports JIT : yes
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"),
> raw-deflate("deflate"), gzip("gzip")
> Built with network namespace support.
>
> Available polling systems :
> epoll : pref=300, test result OK
>poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
>
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
>
> Thank you
> -Nick Vatamaniuc
Uptick in 503s with HAProxy 1.8
Hi Friends,
We upgraded to HAProxy 1.8.14 from 1.7.11 and saw an uptick in the number of
browser related 503 requests with termination reason CC. TC, TR and TW are
consistent across these requests and are -1,-1,0 respectively, these requests
are proxied to nginx, the only thing that has changed so far in our setup is
upgrading from 1.7.11 to 1.8.14. Find below an example log entry:
{"proxy_server_name":"http-in","proxy_tq":"826","proxy_tw":"0","proxy_tc":"-1","proxy_tr":"-1","proxy_tt":"826","http_status_code":"503","http_bytes_read":"0","http_captured_request_cookie":"-","http_captured_response_cookie":"-","proxy_termination_state":"CC--","proxy_active_connections":"131","proxy_frontend_connections":"104","proxy_backend_connections":"27","proxy_server_connections":"27","proxy_retries":"0","proxy_server_queue":"0","proxy_backend_queue":"0","http_method":"GET","http_request_url":"/favicon.ico","ssl_version":"TLSv1.2","ssl_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
I can share our config if required, I am not including it in this email
because of it's size and hope we can narrow down to specifics that will be
easy to share.
Do folks know if there any known things that could have changed between 1.7
and 1.8 to have introduced this uptick?
Thanks!
Abejide Ayodele
It always seems impossible until it's done. --Nelson Mandela
Re: Configuring HAProxy session limits
Thanks for your response! It clarified alot.
Configuring HAProxy session limits
Hi Friends, I am trying to bump session limits via the maxconn in the global section as below: cat /etc/haproxy/redacted-haproxy.cfg global maxconn 1 stats socket /var/run/redacted-haproxy-stats.sock user haproxy group haproxy mode 660 level operator expose-fd listeners frontend redacted-frontend mode tcp bind :2004 default_backend redacted-backend backend redacted-backend mode tcp balance leastconn hash-type consistent server redacted_0 redacted01.qa:8443 check agent-check agent-port 8080 weight 100 send-proxy server redacted-684994ccd-6rn9q 192.168.39.223:8443 check port 8443 weight 100 send-proxy server redacted-684994ccd-c88d9 192.168.46.66:8443 check port 8443 weight 100 send-proxy server redacted-canary-58ccdb7cf4-47f4m 192.168.53.47:8443 check port 8443 weight 100 send-proxy NOTE: I removed some portion of the config for conciseness sake. However this did not seem to have any impact on HAProxy after a reload as seen below: echo "show stat" | socat unix-connect:/var/run/redacted-haproxy-stats.sock stdio | cut -d"," -f7 slim 2000 200 I do not know where 2000 and 200 are coming from as I did not at any point configure that, the maxconn was previously 4096. A more detailed stats output is below: echo "show stat" | socat unix-connect:/var/run/redacted-haproxy-stats.sock stdio # pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses, redacted-frontend,FRONTEND,,,0,2,2000,3694,0,0,0,0,0,OPEN,1,2,00,3,0,9,,,0,0,0,,,0,0,0,0,tcp,,3,9,3694,,0,0, redacted-backend,redacted_0,0,0,0,1,,2,0,0,,0,,0,0,0,0,UP,94,1,0,0,0,1582,0,,1,3,1,,2,,2,0,,1,L4OK,,0,,,0,0,683,,via agent : up,0,0,0,0,L7OK,0,50,Layer4 check passed,Layer7 check passed,2,3,4,1,1,1,10.185.57.54:8443,,tcp redacted-backend,redacted-684994ccd-6rn9q,0,0,0,1,,46,0,0,,0,,0,0,0,0,UP,100,1,0,0,0,1582,0,,1,3,2,,46,,2,0,,1,L4OK,,0,,,0,0,6,,,0,0,0,1Layer4 check passed,,2,3,4192.168.39.223:8443,,tcp redacted-backend,redacted-684994ccd-c88d9,0,0,0,1,,45,0,0,,0,,0,0,0,0,UP,100,1,0,0,0,1582,0,,1,3,3,,45,,2,0,,1,L4OK,,0,,,0,0,12,,,0,0,0,0Layer4 check passed,,2,3,4192.168.46.66:8443,,tcp redacted-backend,redacted-canary-58ccdb7cf4-47f4m,0,0,0,1,,45,0,0,,0,,0,0,0,0,UP,100,1,0,0,0,1582,0,,1,3,4,,45,,2,0,,1,L4OK,,0,,,0,0,10,,,0,0,0,1Layer4 check passed,,2,3,4192.168.53.47:8443,,tcp redacted-backend,BACKEND,0,0,0,2,200,3694,0,0,0,0,,0,0,0,0,UP,394,4,0,,0,1582,0,,1,3,0,,138,,1,3,,9,,0,0,0,0,0,0,6,,,0,0,0,1,,tcp,leastconn,,, I need guidance on what I need to do to configure session limits correctly and also make it reflect in the exported metrics. Thanks! Abejide Ayodele It always seems impossible until it's done. --Nelson Mandela
