Hi List,
With a build of 1.8.12 (and the 1.9 snapshot of 20180623 ) im getting
the 'old' haproxy process take up 100% cpu usage when using 3 threads in
the config and reloading with -sf parameter. I'm using FreeBSD.. (It
also happens with the 14-7 snapshot.)
It seems to happen after 1 thread quits, one of the others gets out of
control.
Most of the time it happens after the first reload.:
haproxy -f /var/etc/haproxy/haproxy.cfg -D
haproxy -f /var/etc/haproxy/haproxy.cfg -D -sf 19110
The main features i use are: ssl offloading / lua / threads
Only a little to no traffic passing through though, im seeing this
behavior also on my in-active production node, the strange part sofar
though is that i could not reproduce it yet on my test machine.
If someone has got a idea on how to patch or what direction to search
for a fix i'm happy to try.
If there is nothing obvious that can be spotted with the info from the
stack-traces of both 1.8 and 1.9 below ill try and dig further tomorrow
:).Thanks in advance for anyone's time :).
Regards,
PiBa-NL (Pieter)
p.s.
I CC'ed Christopher as he seems to have made the last 2 patches going
into 20180623. Im hoping he has a clue on what to do next :).
## Below stack traces are from 1.8.12.. ##
[2.4.3-RELEASE][admin@pfsense_3]/root: /usr/local/bin/gdb --pid 2136
/usr/local/sbin/haproxy
GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/haproxy...done.
Attaching to program: /usr/local/sbin/haproxy, process 2136
[New LWP 101099 of process 2136]
Reading symbols from /lib/libcrypt.so.5...(no debugging symbols
found)...done.
Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libssl.so.8...(no debugging symbols
found)...done.
Reading symbols from /lib/libcrypto.so.8...(no debugging symbols
found)...done.
Reading symbols from /usr/local/lib/liblua-5.3.so...(no debugging
symbols found)...done.
Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done.
[Switching to LWP 100345 of process 2136]
0x0000000800f0f91c in ?? () from /lib/libthr.so.3
(gdb) thread info
Invalid thread ID: info
(gdb) info thread
Id Target Id Frame
* 1 LWP 100345 of process 2136 0x0000000800f0f91c in ?? () from
/lib/libthr.so.3
2 LWP 101099 of process 2136 thread_sync_barrier (barrier=0x8bc4e0
<thread_exit_sync.barrier>) at src/hathreads.c:112
(gdb) bt full
#0 0x0000000800f0f91c in ?? () from /lib/libthr.so.3
No symbol table info available.
#1 0x0000000800f0bf97 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000000050bd4f in main (argc=6, argv=0x7fffffffec70) at
src/haproxy.c:3077
tids = 0x80249cf40
threads = 0x802487240
i = 2
old_sig = {__bits = {1073741824, 0, 0, 0}}
blocked_sig = {__bits = {4227856759, 4294967295, 4294967295,
4294967295}}
err = 0
retry = 200
limit = {rlim_cur = 2282, rlim_max = 2282}
errmsg =
"\000\354\377\377\377\177\000\000\250\354\377\377\377\177\000\000p\354\377\377\377\177\000\000\006\000\000\000\000\000\000\000\270\030\241\374+/\213\032`e\213\000\000\000\000\000h\354\377\377\377\177\000\000\250\354\377\377\377\177\000\000p\354\377\377\377\177\000\000\006\000\000\000\000\000\000\000\020\354\377\377\377\177\000\000r3\340\001\b\000\000\000\001\000\000"
pidfd = 27
(gdb) thread 2
[Switching to thread 2 (LWP 101099 of process 2136)]
#0 thread_sync_barrier (barrier=0x8bc4e0 <thread_exit_sync.barrier>) at
src/hathreads.c:112
112 src/hathreads.c: No such file or directory.
(gdb) bt full
#0 thread_sync_barrier (barrier=0x8bc4e0 <thread_exit_sync.barrier>) at
src/hathreads.c:112
old = 7
#1 0x00000000005ae23f in thread_exit_sync () at src/hathreads.c:151
barrier = 7
#2 0x000000000051260d in sync_poll_loop () at src/haproxy.c:2391
stop = 1
#3 0x0000000000512533 in run_poll_loop () at src/haproxy.c:2438
next = -1636293614
exp = -1636293614
#4 0x000000000050f10b in run_thread_poll_loop (data=0x80249cf48) at
src/haproxy.c:2470
start_lock = {lock = 0, info = {owner = 0, waiters = 0,
last_location = {function = 0x0, file = 0x0, line = 0}}}
ptif = 0x8af8c0 <per_thread_init_list>
ptdf = 0x800f067bc
#5 0x0000000800f01c05 in ?? () from /lib/libthr.so.3
No symbol table info available.
#6 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfdfd000
(gdb) quit
A debugging session is active.
Inferior 1 [process 2136] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/local/sbin/haproxy, process 2136
[2.4.3-RELEASE][admin@pfsense_3]/root: haproxy -vv
HA-Proxy version 1.8.12-fdc6c62 2018/07/13
Copyright 2000-2018 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -DDEBUG_THREAD -DDEBUG_MEMORY -pipe -g -fstack-protector
-fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-address-of-packed-member
-Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1
USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2m-freebsd 2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
## Below stack traces are from HA-Proxy version 1.9-dev0-ba86c6c
2018/06/22 .. ##
[2.4.3-RELEASE][root@pfsense_5]/root/bisect1.9: /usr/local/bin/gdb --pid
20321 /usr/local/sbin/haproxy
GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/haproxy...done.
Attaching to program: /usr/local/sbin/haproxy, process 20321
[New LWP 100321 of process 20321]
Reading symbols from /lib/libcrypt.so.5...(no debugging symbols
found)...done.
Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libssl.so.8...(no debugging symbols
found)...done.
Reading symbols from /lib/libcrypto.so.8...(no debugging symbols
found)...done.
Reading symbols from /usr/local/lib/liblua-5.3.so...(no debugging
symbols found)...done.
Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols
found)...done.
[Switching to LWP 100260 of process 20321]
0x00000000005b14d5 in thread_sync_barrier (barrier=0x8c1a58
<thread_exit_sync.barrier>) at src/hathreads.c:109
109 src/hathreads.c: No such file or directory.
(gdb) info thread
Id Target Id Frame
* 1 LWP 100260 of process 20321 0x00000000005b14d5 in
thread_sync_barrier (barrier=0x8c1a58 <thread_exit_sync.barrier>)
at src/hathreads.c:109
2 LWP 100321 of process 20321 thread_sync_barrier
(barrier=0x8c1a58 <thread_exit_sync.barrier>) at src/hathreads.c:109
(gdb) bt full
#0 0x00000000005b14d5 in thread_sync_barrier (barrier=0x8c1a58
<thread_exit_sync.barrier>) at src/hathreads.c:109
old = 2
#1 0x00000000005b191f in thread_exit_sync () at src/hathreads.c:148
barrier = 7
#2 0x000000000051548d in sync_poll_loop () at src/haproxy.c:2395
stop = 1
#3 0x00000000005153a9 in run_poll_loop () at src/haproxy.c:2439
next = -1522829164
exp = -1522831664
#4 0x00000000005124eb in run_thread_poll_loop (data=0x802440ff0) at
src/haproxy.c:2471
start_lock = {lock = 0, info = {owner = 0, waiters = 0,
last_location = {function = 0x0, file = 0x0, line = 0}}}
ptif = 0x8b4a70 <per_thread_init_list>
ptdf = 0x800f0c3ee <pthread_sigmask+46>
#5 0x000000000050f097 in main (argc=4, argv=0x7fffffffeb18) at
src/haproxy.c:3076
tids = 0x802440ff0
threads = 0x80248a300
i = 3
old_sig = {__bits = {1073741824, 0, 0, 0}}
blocked_sig = {__bits = {4227856759, 4294967295, 4294967295,
4294967295}}
err = 0
retry = 200
limit = {rlim_cur = 2282, rlim_max = 2282}
errmsg =
"\000\353\377\377\377\177\000\000@\353\377\377\377\177\000\000\030\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000>\035\311\224\066\332\275\222
\271\213\000\000\000\000\000\020\353\377\377\377\177\000\000@\353\377\377\377\177\000\000\030\353\377\377\377\177\000\000\004\000\000\000\000\000\000\000\260\352\377\377\377\177\000\000r3\340\001\b\000\000\000\001\000\000"
pidfd = -1
(gdb) thread 2
[Switching to thread 2 (LWP 100321 of process 20321)]
#0 thread_sync_barrier (barrier=0x8c1a58 <thread_exit_sync.barrier>) at
src/hathreads.c:109
109 in src/hathreads.c
(gdb) bt full
#0 thread_sync_barrier (barrier=0x8c1a58 <thread_exit_sync.barrier>) at
src/hathreads.c:109
old = 7
#1 0x00000000005b191f in thread_exit_sync () at src/hathreads.c:148
barrier = 7
#2 0x000000000051548d in sync_poll_loop () at src/haproxy.c:2395
stop = 1
#3 0x00000000005153a9 in run_poll_loop () at src/haproxy.c:2439
next = -1522831674
exp = -1522831674
#4 0x00000000005124eb in run_thread_poll_loop (data=0x802440ff4) at
src/haproxy.c:2471
start_lock = {lock = 0, info = {owner = 0, waiters = 0,
last_location = {function = 0x0, file = 0x0, line = 0}}}
ptif = 0x8b4a70 <per_thread_init_list>
ptdf = 0x800f0b7bc
#5 0x0000000800f06c05 in ?? () from /lib/libthr.so.3
No symbol table info available.
#6 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfffe000
(gdb)
[2.4.3-RELEASE][root@pfsense_5]/root/bisect1.9: haproxy -vv
HA-Proxy version 1.9-dev0-ba86c6c 2018/06/22
Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -DDEBUG_THREAD -DDEBUG_MEMORY -pipe -g -fstack-protector
-fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-address-of-packed-member
-Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1
USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2m-freebsd 2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
