Hi, I have seen this crash in production for several days with the official dockerized version 3.2.10, also upgraded to 3.2.16 with the same result. After analysing a few coredumps and applying the following small patch, it has been running in production so far without crashing.
The process aborted by watchdog. I have included the relevant parts which seem
to be only around idle connection cleanup and task scheduling. keep-alive is
enabled and http-reuse is set to safe, multi-threading config is default and
running on a 24-core machine.
relevant gdb output from the coredump:
[Current thread is 1 (Thread 0x7f18be7fc6c0 (LWP 17))]
#0 __GI_abort () at ./stdlib/abort.c:95
#1 0x00005627716a4720 in ha_panic () at src/debug.c:847
#2 0x00005627716a4795 in ha_panic () at src/debug.c:778
#3 0x000056277179ebbd in wdt_handler (sig=<optimized out>,
si=<optimized out>, arg=<optimized out>) at src/wdt.c:193
#4 <signal handler called>
#5 0x00005627715cc7af in _task_schedule (task=0x56277420a8d0,
when=285047, caller=caller@entry=0x562771899ee0 <_.4>) at
include/haproxy/task.h:725
#6 0x00005627715da609 in _task_schedule (task=<optimized out>,
when=<optimized out>, caller=0x562771899ee0 <_.4>) at
include/haproxy/task.h:711
#7 srv_add_to_idle_list (srv=0x562773f02d00, conn=<optimized out>,
is_safe=is_safe@entry=128) at src/server.c:7291
#8 0x00005627715a1684 in h1s_finish_detach
(h1s=h1s@entry=0x7f1898949730) at src/mux_h1.c:1170
#9 0x00005627715a1e28 in h1_detach (sd=<optimized out>) at src/mux_h1.c:4583
#10 0x00005627716946e8 in sc_detach_endp
(scp=scp@entry=0x7f18be7f0d98) at src/stconn.c:449
Another thread was waiting on the idle connection server lock while running
the same idle connection cleanup task:
#8 0x00005627715cbbd7 in __pl_wait_unlock_long (lock=0x5627718fa140
<idle_conn_srv_lock>, mask=18446744069414584320) at
include/import/plock.h:96
loops = 14005
ret = <optimized out>
m = 74862
ret = <optimized out>
m = <optimized out>
loops = <optimized out>
#9 pl_wait_unlock_long (lock=lock@entry=0x5627718fa140
<idle_conn_srv_lock>, mask=mask@entry=18446744069414584320) at
include/import/plock.h:123
No locals.
#10 0x00005627715dac35 in srv_cleanup_idle_conns (task=0x56277420a8d0,
context=<optimized out>, state=<optimized out>) at src/server.c:7321
__lk_r = 0x5627718fa140 <idle_conn_srv_lock>
__set_r = 4294967300
__msk_r = 18446744069414584320
srv = <optimized out>
eb = <optimized out>
i = <optimized out>
>From this it looked like one thread could call task_schedule(idle_conn_task)
from srv_add_to_idle_list() while idle_conn_srv_lock was still held, while
another thread was already running srv_cleanup_idle_conns() and waiting for
the same lock.
I tested this by patching docker image 3.2.16 and running it in production on
multiple nodes which are behind another layer of load balancers.
The attached patch is against current master and should be backported to 3.2
if the analysis is correct. Also, I think the severity is MAJOR, but
I'm not sure.
Sincerely,
Amin Shayan
haproxy-idle-task-schedule-master.patch
Description: Binary data
