Hi List,
Ive got a weird issue.. and im not sure where/how to continue digging at
the moment...
Using nbthread=1 nbproc=1, a few lua scripts, ssl offloading / http
traffic.. Only a few connections < 100...
Sometimes haproxy starts using 100% cpu usage.. After a few days..
(Makes it hard to debug..)
Currently running with version 'HA-Proxy version 1.9-dev0-48d92ee
2018/07/30'
Ive ran some commands against the haproxy socket like 'show activity',
as can be seen there are lots of loops and tasks in just a second of time.
[2.4.3-RELEASE][root@pfsense_3]/root:
/usr/local/pkg/haproxy/haproxy_socket.sh show activity
show activity thread_id: 0
date_now: 1533754729.799405
loops: 828928664
wake_cache: 845396
wake_tasks: 827400248
wake_signal: 0
poll_exp: 828245644
poll_drop: 17451
poll_dead: 0
poll_skip: 0
fd_skip: 0
fd_lock: 0
fd_del: 0
conn_dead: 0
stream: 101147
empty_rq: 1242050
long_rq: 0
[2.4.3-RELEASE][root@pfsense_3]/root:
/usr/local/pkg/haproxy/haproxy_socket.sh show activity
show activity thread_id: 0
date_now: 1533754731.084664
loops: 829000230
wake_cache: 845398
wake_tasks: 827471812
wake_signal: 0
poll_exp: 828317210
poll_drop: 17452
poll_dead: 0
poll_skip: 0
fd_skip: 0
fd_lock: 0
fd_del: 0
conn_dead: 0
stream: 101149
empty_rq: 1242050
long_rq: 0
Other than that ive tried to attach gdb and step through / log some
functions.. it passes through.
With a gdb 'command' file like bellow i created a little log of function
breakpoints hit:
set pagination off
set height 0
set logging on
delete
rbreak haproxy.c:.
rbreak session.c:.
rbreak hlua.c:.
rbreak task.c:.
commands 1-99999
cont
end
cont
Which got me a log with the following content.. as you can see it
'seems' to be looping over the same task multiple times.., which might
not even be a problem.??. the t=0x802545a60 expires and wakes up ,
expires and wakes up.?.:
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
<rqueue_local>) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
<rqueue_local>) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
<rqueue_local>) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
<rqueue_local>) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
Breakpoint 51, session_expire_embryonic (t=0x802545a60,
context=0x8024483a0, state=513) at src/session.c:389
389 src/session.c: No such file or directory.
Breakpoint 249, __task_queue (task=0x802545a60) at src/task.c:185
185 src/task.c: No such file or directory.
Breakpoint 253, wake_expired_tasks () at src/task.c:209
209 in src/task.c
Breakpoint 250, __task_wakeup (t=0x802545a60, root=0x8ced50
<rqueue_local>) at src/task.c:72
72 in src/task.c
Breakpoint 41, sync_poll_loop () at src/haproxy.c:2378
2378 src/haproxy.c: No such file or directory.
Breakpoint 252, process_runnable_tasks () at src/task.c:275
275 src/task.c: No such file or directory.
haproxy -vv
HA-Proxy version 1.9-dev0-48d92ee 2018/07/30
Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -DDEBUG_THREAD -DDEBUG_MEMORY -pipe -g -fstack-protector
-fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-address-of-packed-member
-Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1
USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2m-freebsd 2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
But im not sure what to do next.. Today it happened again and i tried to
run the gdb log command with all haproxy source files.. but then it
stopped working completely..(or at least so slow it didnt properly
respond anymore..) so i had to abort and restart..
Is there any extra info i can gather next time? A different gdb command
script to try and run?
Should i try a newer version? (i did have it with a previous build from
a week earlier also.., not sure if it happened before that..)
Could it be because of the openssl version mismatch? (not sure how easy
it is for me to compile it against the 'correct' version.. i never
seemed to have issues with that before..)
Hoping someone has an idea how to debug it further/differently or
perhaps create a patch that might provide extra information when it
occurs again.?.
Regards,
PiBa-NL (Pieter)
