bneradt commented on issue #11131:
URL:
https://github.com/apache/trafficserver/issues/11131#issuecomment-2045917503
I reproduced this locally by running the ramp test in a loop with ATS run
via gdb:
```gdb
(gdb) bt
#0 __pthread_kill_implementation (threadid=281474795599200,
signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x0000fffff6f522f8 [PAC] in __pthread_kill_internal (signo=6,
threadid=<optimized out>) at pthread_kill.c:78
#2 0x0000fffff6f05800 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
#3 0x0000fffff6ef0288 [PAC] in __GI_abort () at abort.c:79
#4 0x00000000008beae0 [PAC] in ink_abort (message_format=0xd1a908 "%s:%d:
failed assertion `%s`") at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/tscore/ink_error.cc:99
#5 0x00000000008bac50 in _ink_assert (expression=0xd35a30 "event ==
HTTP_TUNNEL_EVENT_DONE || event == VC_EVENT_INACTIVITY_TIMEOUT", file=0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
line=2986)
at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/tscore/ink_assert.cc:35
#6 0x0000000000920edc in HttpSM::tunnel_handler (this=0xffffc466f710,
event=103, data=0xffffc42605b0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2986
#7 0x000000000091f350 in HttpSM::main_handler (this=0xffffc466f710,
event=103, data=0xffffc42605b0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2622
#8 0x000000000088a7ac in Continuation::handleEvent (this=0xffffc466f710,
event=103, data=0xffffc42605b0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/include/iocore/eventsystem/Continuation.h:228
#9 0x0000000000c4aea4 in write_signal_and_update (event=103,
vc=0xffffc42602e0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:121
#10 0x0000000000c4b430 in write_signal_done (event=103, nh=0xfffff5e1cba0,
vc=0xffffc42602e0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:166
#11 0x0000000000c4c834 in write_to_net_io (nh=0xfffff5e1cba0,
vc=0xffffc42602e0, thread=0xfffff5e1c010) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:522
#12 0x0000000000c4bfa4 in write_to_net (nh=0xfffff5e1cba0,
vc=0xffffc42602e0, thread=0xfffff5e1c010) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:356
#13 0x0000000000c4df9c in UnixNetVConnection::net_write_io
(this=0xffffc42602e0, nh=0xfffff5e1cba0, lthread=0xfffff5e1c010) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/UnixNetVConnection.cc:862
#14 0x0000000000c7b7b8 in NetHandler::process_ready_list
(this=0xfffff5e1cba0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/NetHandler.cc:286
#15 0x0000000000c7bab4 in NetHandler::waitForActivity (this=0xfffff5e1cba0,
timeout=4965250) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/net/NetHandler.cc:364
#16 0x0000000000cbf164 in EThread::execute_regular (this=0xfffff5e1c010) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/eventsystem/UnixEThread.cc:300
#17 0x0000000000cbf300 in EThread::execute (this=0xfffff5e1c010) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/eventsystem/UnixEThread.cc:350
#18 0x0000000000cbd9dc in spawn_thread_internal (a=0x11deca0) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/iocore/eventsystem/Thread.cc:68
#19 0x0000fffff6f50584 in start_thread (arg=0xfffff7fb9760) at
pthread_create.c:444
#20 0x0000fffff6fbfd4c [PAC] in thread_start () at
../sysdeps/unix/sysv/linux/aarch64/clone3.S:76
(gdb) f 6
#6 0x0000000000921014 in HttpSM::tunnel_handler (this=0xffffc53434b0,
event=103, data=0xffffc5866260) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2989
2989 ink_assert(event == HTTP_TUNNEL_EVENT_DONE || event ==
VC_EVENT_INACTIVITY_TIMEOUT);
(gdb) p event
$4 = 103
(gdb) p (int)TS_EVENT_VCONN_WRITE_COMPLETE
$5 = 103
```
Thus the crash happens while processing an unexpected `WRITE_COMPLETE` event.
Checking the `history` value of the HttpSM shows this:
```gdb
(gdb) p history
$1 = {history = {
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd34450 "state_api_callback", line = 1292}, event = 60000, reentrancy
= 2},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd34488 "state_api_callout", line = 1332}, event = 60000, reentrancy =
2},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd33a20 "state_read_client_request_header", line = 523}, event = 100,
reentrancy = 3},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd39128 "set_next_state", line = 7933}, event = 34463, reentrancy = 3},
{location = {file = 0xd50b60
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpCacheSM.cc",
func = 0xd50bf8 "state_cache_open_read", line = 106}, event = 1104, reentrancy
= -31073},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd36c58 "do_cache_lookup_and_read", line = 4954}, event = 22808,
reentrancy = 3},
{location = {file = 0xd50b60
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpCacheSM.cc",
func = 0xd50bf8 "state_cache_open_read", line = 106}, event = 1103, reentrancy
= -31073},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd35348 "state_cache_open_read", line = 2534}, event = 1103,
reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd39128 "set_next_state", line = 7893}, event = 34463, reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd39128 "set_next_state", line = 8047}, event = 34463, reentrancy = 1},
{location = {file = 0xd50b60
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpCacheSM.cc",
func = 0xd50d80 "state_cache_open_write", line = 188}, event = 1108,
reentrancy = -31073},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd351f0 "state_cache_open_write", line = 2413}, event = 1108,
reentrancy = 2},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd39128 "set_next_state", line = 7942}, event = 34463, reentrancy = 2},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd34a38 "state_read_server_response_header", line = 1873}, event =
104, reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd38478 "setup_internal_transfer", line = 6908}, event = 34463,
reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd35a08 "tunnel_handler", line = 2984}, event = 103, reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd38708 "setup_server_transfer", line = 7143}, event = 34463,
reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd35af8 "tunnel_handler_server", line = 3038}, event = 2302,
reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd35d30 "tunnel_handler_ua", line = 3409}, event = 103, reentrancy =
0},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd35a08 "tunnel_handler", line = 2984}, event = 2301, reentrancy = 1},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd34450 "state_api_callback", line = 1292}, event = 60000, reentrancy
= 2},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd34488 "state_api_callout", line = 1332}, event = 60000, reentrancy =
2},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd38b10 "kill_this", line = 7520}, event = 34463, reentrancy = 0},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd38b10 "kill_this", line = 7520}, event = 34463, reentrancy = 0},
{location = {file = 0xd33838
"/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc",
func = 0xd38b10 "kill_this", line = 7520}, event = 34463, reentrancy = 0},
{location = {file = 0xefbeaddeefbeadde <error: Cannot access memory at
address 0xefbeaddeefbeadde>, func = 0xefbeaddeefbeadde <error: Cannot access
memory at address 0xefbeaddeefbeadde>, line = -272716322}, event = 44510,
reentrancy = -4162} <repeats 40 times>},
history_pos = 16}
```
Note that the final events are `kill_this`. We don't expect `WRITE_COMPLETE`
events anymore after kill_this is called. Apparently. The `this` HttpSM, while
killed, is not yet deleted, so this is not technically a use after free yet:
```gdb
(gdb) f 6
#6 0x0000000000921014 in HttpSM::tunnel_handler (this=0xffffc53434b0,
event=103, data=0xffffc5866260) at
/home/bneradt/src/ts_asf_master_fix_ramp_test_failure/src/proxy/http/HttpSM.cc:2989
2989 ink_assert(event == HTTP_TUNNEL_EVENT_DONE || event ==
VC_EVENT_INACTIVITY_TIMEOUT);
(gdb) p magic
$11 = 65261
(gdb) p (int)HTTP_SM_MAGIC_ALIVE
$12 = 65261
(gdb) p kill_this_async_done
$13 = false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
