Hi Willy and Tim, > > >> Code 134 implies the worker was killed with SIGABRT. You could check > > >> whether there is a core dump. > > > > > > I don't have any core dumps. > > > > Check whether coredumps are enabled using `ulimit -c`, often they are > > disabled by default, because they could contain sensitive information. > > After the next crash you should be able to retrieve a backtrace using > > gdb. Possibly recompile haproxy with debug symbols for it to be useful. > > If it happens quickly, another option might be to attach gdb to the > process after it is started. But with multiple processes it's not very > convenient. > > A few more things on the core dumps : > - they are ignored if you have a chroot statement in the global section > - you need not to use "user/uid/group/gid" otherwise the system also > disables core dumps
I'm using chroot and user/group in my config, so I'm not able to share core dumps. > There are very few abort() calls in the code : > - some in the thread debugging code to detect recursive locks ; > - one in the cache applet which triggers on an impossible case very > likely resulting from cache corruption (hence a bug) > - a few inside the Lua library > - a few in the HPACK decompressor, detecting a few possible bugs there > > Except for Lua, all of them were added during 1.8, so depending on what the > configuration uses, there are very few possible candidates. I added my configuration in this mail. Hopefully this will narrow down the possible candidates. I did some more research to the memory warnings we encounter every few days. It seems like the haproxy processes use a lot of memory. Would haproxy with nbthreads share this memory? 1160 haproxy 20 0 1881720 1.742g 5504 S 83.9 11.5 1:53.38 haproxy 1045 haproxy 20 0 1880120 1.740g 5572 S 71.0 11.5 1:36.62 haproxy 1104 haproxy 20 0 1880376 1.741g 6084 R 64.6 11.5 1:46.29 haproxy 1079 haproxy 20 0 1881116 1.741g 5564 S 58.1 11.5 1:42.29 haproxy 1135 haproxy 20 0 1881240 1.741g 5564 S 58.1 11.5 1:49.85 haproxy 995 haproxy 20 0 1881852 1.742g 5584 R 38.7 11.5 1:30.05 haproxy 1020 haproxy 20 0 1881448 1.741g 5516 S 25.8 11.5 1:32.20 haproxy 4926 haproxy 20 0 1881008 1.718g 2176 S 6.5 11.3 3:11.74 haproxy 8526 haproxy 20 0 1878032 6516 1304 S 0.0 0.0 2:10.04 haproxy 8529 haproxy 20 0 1880336 5208 4 S 0.0 0.0 2:34.68 haproxy 11530 haproxy 20 0 1878748 6556 1392 S 0.0 0.0 2:25.94 haproxy 26938 haproxy 20 0 1882592 6032 892 S 0.0 0.0 3:56.79 haproxy 29577 haproxy 20 0 1880480 1.738g 3132 S 0.0 11.5 2:08.74 haproxy 31124 haproxy 20 0 1880776 1.740g 4284 S 0.0 11.5 2:58.84 haproxy 7548 root 20 0 1869896 1.731g 4456 S 0.0 11.4 1008:23 haproxy I'm using systemd to reload haproxy for new SSL certificates every few minutes. [Service] Environment=CONFIG=/etc/haproxy/haproxy.cfg EnvironmentFile=-/etc/default/haproxy ExecStartPre=/usr/sbin/haproxy -f ${CONFIG} -c -q ExecStart=/usr/sbin/haproxy -Ws -f ${CONFIG} -p /run/haproxy.pid $EXTRAOPTS ExecReload=/usr/sbin/haproxy -c -f ${CONFIG} ExecReload=/bin/kill -USR2 $MAINPID KillMode=mixed Restart=always Configuration: global log **removed hostname** syslog maxconn 32000 ulimit-n 65536 tune.maxrewrite 2048 user haproxy group haproxy daemon chroot /var/lib/haproxy nbproc 7 maxcompcpuusage 85 spread-checks 0 ssl-default-bind-options no-sslv3 stats socket /var/run/haproxy.sock mode 400 level admin process 1 stats socket /var/run/haproxy.sock.2 mode 400 level admin process 2 stats socket /var/run/haproxy.sock.3 mode 400 level admin process 3 stats socket /var/run/haproxy.sock.4 mode 400 level admin process 4 stats socket /var/run/haproxy.sock.5 mode 400 level admin process 5 stats socket /var/run/haproxy.sock.6 mode 400 level admin process 6 stats socket /var/run/haproxy.sock.7 mode 400 level admin process 7 master-worker no-exit-on-failure defaults log global timeout http-request 5s timeout connect 2s timeout client 125s timeout server 125s mode http option dontlog-normal option http-server-close option tcp-smart-connect frontend fe_http bind ipv4@:80 backlog 65534 bind ipv6@:80 backlog 65534 bind ipv4@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt /etc/haproxy/ssl/ crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534 bind ipv6@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt /etc/haproxy/ssl/ crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534 bind-process 1-7 tcp-request inspect-delay 5s tcp-request content accept if { req_ssl_hello_type 1 } option forwardfor acl secure dst_port 443 acl is_acme_request path_beg /.well-known/acme-challenge/ reqadd X-Forwarded-Proto:\ https if secure default_backend be_reservedpage use_backend be_acme if is_acme_request use_backend %[req.fhdr(host),lower,map_dom(/etc/haproxy/domain2backend.map)] compression algo gzip maxconn 32000 http-response set-header X-Balancer lb0 listen stats bind *:1936 bind-process 1 mode http stats enable stats uri / stats admin if TRUE backend be_acme bind-process 1 option httpchk HEAD /ping.php HTTP/1.1\r\nHost:\ **removed hostname** option http-server-close option http-pretend-keepalive server **removed hostname** **removed ip**:80 maxconn 200 inter 5000 check backend be_reservedpage bind-process 1 http-request set-header X-Forwarded-Host %[hdr(host)] http-request set-header Host **removed hostname** http-response set-header Cache-Control no-cache,\ no-store,\ must-revalidate server **removed name** **removed ip**:80 verify none check Thanks, Frank