Hi Frederic,
Op 28-8-2018 om 11:27 schreef Frederic Lecaille:
On 08/27/2018 10:46 PM, PiBa-NL wrote:
Hi Frederic, Oliver,
Thanks for your investigations :).
I've made a little reg-test (files attached). Its probably not
'correct' to commit as-is, but should be enough to get a
reproduction.. I hope..
changing it to nbthread 1 makes it work every time..(that i tried)
The test actually seems to show a variety of issues.
## Every once in a while it takes like 7 seconds to run a test..
During which cpu usage is high..
do you think we can reproduce this 200% CPU usage issue after having
disabled ssl
With ssl 'disabled' i can run the test 500 times without a single failure..
As for the cpu usage issue it does not seem to reproduce 'easily' when
running inside varnishtest.. But that might also be because it dumps its
core most of the time..
Using the same config that varnishtest generated, and then changing the
ports to :80 (for frontend) and 81 (for stats) then manually running
haproxy -f /tmp/vtc.132.456/h1/cfg after a few curl requests curl hangs
waiting for haproxy's response which is running 100% cpu..
Below 2 backtraces one of 100% cpu usage, and one of a core dump. Does
that help? Do you need the actual core+binary?
Regards,
PiBa-NL (Pieter)
#############
Using 100% cpu:
(gdb) info thread
Id Target Id Frame
* 1 LWP 101573 of process 28901 0x0000000801e11e3a in _kevent () from
/lib/libc.so.7
2 LWP 100816 of process 28901 0x0000000801e11e3a in _kevent ()
from /lib/libc.so.7
3 LWP 101309 of process 28901 0x000000080187a71d in ?? () from
/usr/local/lib/liblua-5.3.so
(gdb) thread 3
[Switching to thread 3 (LWP 101309 of process 28901)]
#0 0x000000080187a71d in ?? () from /usr/local/lib/liblua-5.3.so
(gdb) bt full
#0 0x000000080187a71d in ?? () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#1 0x000000080187acd7 in ?? () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#2 0x000000080187b108 in ?? () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#3 0x0000000801873e30 in lua_gc () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#4 0x0000000000438e45 in hlua_ctx_resume (lua=0x8024dbf80,
yield_allowed=1) at src/hlua.c:1186
ret = 0
msg = 0x5a5306 <nsec_now+22> "Hiu\360"
trace = 0x7fffdfdfcc00 ""
#5 0x000000000044887a in hlua_applet_http_fct (ctx=0x8024d4a80) at
src/hlua.c:6716
si = 0x803081840
strm = 0x803081500
res = 0x803081570
rule = 0x80242d6e0
px = 0x8024c4400
hlua = 0x8024dbf80
blk1 = 0x7fffdfdfcca0 ""
len1 = 34397581057
blk2 = 0x803081578 ""
len2 = 34410599800
---Type <return> to continue, or q <return> to quit---
ret = 0
#6 0x00000000005a78a7 in task_run_applet (t=0x80242db40,
context=0x8024d4a80, state=16385) at src/applet.c:49
app = 0x8024d4a80
si = 0x803081840
#7 0x00000000005a49a6 in process_runnable_tasks () at src/task.c:384
t = 0x80242db40
state = 16385
ctx = 0x8024d4a80
process = 0x5a77f0 <task_run_applet>
t = 0x80242db40
max_processed = 200
#8 0x000000000051a6b2 in run_poll_loop () at src/haproxy.c:2386
next = -2118609833
exp = -2118610700
#9 0x0000000000517672 in run_thread_poll_loop (data=0x8024843c8) at
src/haproxy.c:2451
start_lock = {lock = 0, info = {owner = 0, waiters = 0,
last_location = {function = 0x0, file = 0x0, line = 0}}}
ptif = 0x8c1980 <per_thread_init_list>
ptdf = 0x800f177cc
#10 0x0000000800f12bc5 in ?? () from /lib/libthr.so.3
No symbol table info available.
#11 0x0000000000000000 in ?? ()
No symbol table info available.
Backtrace stopped: Cannot access memory at address 0x7fffdfdfd000
##########
Core dump:
gdb --core haproxy.core /usr/local/sbin/haproxy
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Core was generated by `haproxy -f /tmp/vtc.28884.6c5c88f3/h1/cfg'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libcrypt.so.5...done.
Loaded symbols for /lib/libcrypt.so.5
Reading symbols from /lib/libz.so.6...done.
Loaded symbols for /lib/libz.so.6
Reading symbols from /lib/libthr.so.3...done.
Loaded symbols for /lib/libthr.so.3
Reading symbols from /usr/lib/libssl.so.8...done.
Loaded symbols for /usr/lib/libssl.so.8
Reading symbols from /lib/libcrypto.so.8...done.
Loaded symbols for /lib/libcrypto.so.8
Reading symbols from /usr/local/lib/liblua-5.3.so...done.
Loaded symbols for /usr/local/lib/liblua-5.3.so
Reading symbols from /lib/libm.so.5...done.
Loaded symbols for /lib/libm.so.5
Reading symbols from /lib/libc.so.7...done.
Loaded symbols for /lib/libc.so.7
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0 0x000000080187b828 in lua_yieldk () from /usr/local/lib/liblua-5.3.so
(gdb) info thread
3 process 101183 0x0000000801e11e3a in _kevent () from /lib/libc.so.7
2 process 101573 0x0000000801e11e3a in _kevent () from /lib/libc.so.7
* 1 process 101226 0x000000080187b828 in lua_yieldk () from
/usr/local/lib/liblua-5.3.so
(gdb) bt full
#0 0x000000080187b828 in lua_yieldk () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#1 0x000000080187a8e7 in lua_yieldk () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#2 0x000000080187afa8 in lua_yieldk () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#3 0x00000008018881f6 in lua_close () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#4 0x0000000801879270 in lua_resume () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#5 0x0000000801877f26 in lua_getinfo () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#6 0x0000000801878ef7 in lua_resume () from /usr/local/lib/liblua-5.3.so
No symbol table info available.
#7 0x0000000000438af3 in hlua_ctx_resume (lua=0x803086000,
yield_allowed=1) at src/hlua.c:1089
ret = 50819904
msg = 0x5a8a24 "▒\200"
trace = 0x7fffdfdfcc60 "▒▒▒▒▒\177"
#8 0x000000000044887a in hlua_applet_http_fct (ctx=0x803063380) at
src/hlua.c:6716
si = (struct stream_interface *) 0x803077340
strm = (struct stream *) 0x803077000
res = (struct channel *) 0x803077070
rule = (struct act_rule *) 0x80242d5a0
---Type <return> to continue, or q <return> to quit---
px = (struct proxy *) 0x8024c4400
hlua = (struct hlua *) 0x803086000
blk1 = 0x7fffdfdfcca0 ""
len1 = 34397581057
blk2 = 0x803077078 ""
len2 = 34410557560
ret = 0
#9 0x00000000005a78a7 in task_run_applet (t=0x80307c0a0,
context=0x803063380, state=16385) at src/applet.c:49
app = (struct appctx *) 0x803063380
si = (struct stream_interface *) 0x803077340
#10 0x00000000005a49a6 in process_runnable_tasks () at src/task.c:384
t = (struct task *) 0x80307c0a0
state = 16385
ctx = (void *) 0x803063380
process = (struct task *(*)(struct task *, void *, unsigned
short)) 0x5a77f0 <task_run_applet>
t = (struct task *) 0x80307c0a0
max_processed = 200
#11 0x000000000051a6b2 in run_poll_loop () at src/haproxy.c:2386
next = -2118242384
exp = -2118243378
#12 0x0000000000517672 in run_thread_poll_loop (data=0x8024843c8) at
src/haproxy.c:2451
start_lock = {lock = 0, info = {owner = 0, waiters = 0,
last_location = {function = 0x0, file = 0x0, line = 0}}}
ptif = (struct per_thread_init_fct *) 0x8c1980
---Type <return> to continue, or q <return> to quit---
ptdf = (struct per_thread_deinit_fct *) 0x800f177cc
#13 0x0000000800f12bc5 in pthread_create () from /lib/libthr.so.3
No symbol table info available.
#14 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)
