[ https://issues.apache.org/jira/browse/TS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335371#comment-14335371 ]
Susan Hinrichs commented on TS-3405: ------------------------------------ Saw a similar stack after a seg fault in my dev build. In that case contp (Http2ClientSession?) had been freed but FetchSM still had a reference to it. It was processing EVENT_EOS. The GDB prints are below. I'll give Ryo's patch a try. (gdb) bt #0 0x000000000050caf8 in Continuation::handleEvent (this=0x7fffc82842d0, event=-4, data=0x7fffc8039650) at ../iocore/eventsystem/I_Continuation.h:146 #1 0x000000000050ac43 in FetchSM::InvokePluginExt (this=0x7fffc8039650, fetch_event=104) at FetchSM.cc:301 #2 0x000000000050b4e6 in FetchSM::process_fetch_read (this=0x7fffc8039650, event=104) at FetchSM.cc:465 #3 0x000000000050b7c5 in FetchSM::fetch_handler (this=0x7fffc8039650, event=104, edata=0x7fffdc0727a8) at FetchSM.cc:514 #4 0x000000000050cafa in Continuation::handleEvent (this=0x7fffc8039650, event=104, data=0x7fffdc0727a8) at ../iocore/eventsystem/I_Continuation.h:146 #5 0x000000000054bd42 in PluginVC::process_read_side (this=0x7fffdc072698, other_side_call=false) at PluginVC.cc:640 #6 0x000000000054a220 in PluginVC::main_handler (this=0x7fffdc072698, event=1, data=0x7fffe01e8580) at PluginVC.cc:206 #7 0x000000000050cafa in Continuation::handleEvent (this=0x7fffdc072698, event=1, data=0x7fffe01e8580) at ../iocore/eventsystem/I_Continuation.h:146 #8 0x00000000007a876e in EThread::process_event (this=0x7ffff69e9010, e=0x7fffe01e8580, calling_code=1) at UnixEThread.cc:144 #9 0x00000000007a8c5d in EThread::execute (this=0x7ffff69e9010) at UnixEThread.cc:238 #10 0x00000000007a7c74 in spawn_thread_internal (a=0x11280d0) at Thread.cc:88 #11 0x000000351e807851 in start_thread () from /lib64/libpthread.so.0 #12 0x000000351e4e890d in clone () from /lib64/libc.so.6 (gdb) frame 1 #1 0x000000000050ac43 in FetchSM::InvokePluginExt (this=0x7fffc8039650, fetch_event=104) at FetchSM.cc:301 301 contp->handleEvent(TS_FETCH_EVENT_EXT_BODY_DONE, this); (gdb) print this->contp[0] $14 = {<force_VFPT_to_top> = {_vptr.force_VFPT_to_top = 0x7fffc8283c51}, handler = (int (Continuation::*)(Continuation *, int, void *)) 0xefbeaddeefbeadde, this adjustment -1171307680053154338, handler_name = 0xefbeaddeefbeadde <Address 0xefbeaddeefbeadde out of bounds>, mutex = {m_ptr = 0xefbeaddeefbeadde}, link = {<SLink<Continuation>> = { next = 0xefbeaddeefbeadde}, prev = 0xefbeaddeefbeadde}} (gdb) > Memory use after free in HTTP/2 > ------------------------------- > > Key: TS-3405 > URL: https://issues.apache.org/jira/browse/TS-3405 > Project: Traffic Server > Issue Type: Bug > Components: HTTP/2 > Reporter: Bryan Call > Fix For: 5.3.0 > > Attachments: fix-h2.patch > > > From Leif running on docs.trafficserver.apache.org: > > {code} > traffic_server: using root directory '/opt/ats' > ================================================================= > ==31101==ERROR: AddressSanitizer: heap-use-after-free on address > 0x61800000c888 at pc 0x4f3558 bp 0x2aaf10c88930 sp 0x2aaf10c88928 > READ of size 8 at 0x61800000c888 thread T2 ([ET_NET 1]) > #0 0x4f3557 in Continuation::handleEvent(int, void*) > ../iocore/eventsystem/I_Continuation.h:146 > #1 0x4f3557 in FetchSM::InvokePluginExt(int) > /usr/local/src/trafficserver/proxy/FetchSM.cc:301 > #2 0x4f3a7a in FetchSM::process_fetch_read(int) > /usr/local/src/trafficserver/proxy/FetchSM.cc:465 > #3 0x4f5112 in FetchSM::fetch_handler(int, void*) > /usr/local/src/trafficserver/proxy/FetchSM.cc:514 > #4 0x59f1b7 in Continuation::handleEvent(int, void*) > ../iocore/eventsystem/I_Continuation.h:146 > #5 0x59f1b7 in PluginVC::process_read_side(bool) > /usr/local/src/trafficserver/proxy/PluginVC.cc:640 > #6 0x5abcb9 in PluginVC::main_handler(int, void*) > /usr/local/src/trafficserver/proxy/PluginVC.cc:206 > #7 0xc821fe in Continuation::handleEvent(int, void*) > /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146 > #8 0xc821fe in EThread::process_event(Event*, int) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144 > #9 0xc84819 in EThread::execute() > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:238 > #10 0xc80e18 in spawn_thread_internal > /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:88 > #11 0x2aaf0b083df2 in start_thread (/lib64/libpthread.so.0+0x7df2) > #12 0x2aaf0c8ec1ac in clone (/lib64/libc.so.6+0xf61ac) > 0x61800000c888 is located 8 bytes inside of 816-byte region > [0x61800000c880,0x61800000cbb0) > freed by thread T0 ([ET_NET 0]) here: > #0 0x2aaf08c131c7 in __interceptor_free > ../../.././libsanitizer/asan/asan_malloc_linux.cc:62 > #1 0x7b7d42 in Http2ClientSession::do_io_close(int) > /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:194 > #2 0x7b7d42 in Http2ClientSession::main_event_handler(int, void*) > /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:237 > #3 0xc1351f in Continuation::handleEvent(int, void*) > ../../iocore/eventsystem/I_Continuation.h:146 > #4 0xc1351f in read_signal_and_update > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140 > #5 0xc1351f in read_signal_done > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185 > #6 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*) > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939 > #7 0xbbabf8 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) > /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:596 > #8 0xbda09c in NetHandler::mainNetEvent(int, Event*) > /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513 > #9 0xc85089 in Continuation::handleEvent(int, void*) > /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146 > #10 0xc85089 in EThread::process_event(Event*, int) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144 > #11 0xc85089 in EThread::execute() > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268 > #12 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826 > #13 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4) > previously allocated by thread T0 ([ET_NET 0]) here: > #0 0x2aaf08c1393b in __interceptor_posix_memalign > ../../.././libsanitizer/asan/asan_malloc_linux.cc:130 > #1 0x2aaf09afd2f9 in ats_memalign > /usr/local/src/trafficserver/lib/ts/ink_memory.cc:96 > #2 0x7cd804 in ClassAllocator<Http2ClientSession>::alloc() > ../../lib/ts/Allocator.h:124 > #3 0x7cd804 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*, > IOBufferReader*) > /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:57 > #4 0x7cd3c4 in Http2SessionAccept::mainEvent(int, void*) > /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:69 > #5 0xbc2fae in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) > /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:101 > #6 0xc1351f in Continuation::handleEvent(int, void*) > ../../iocore/eventsystem/I_Continuation.h:146 > #7 0xc1351f in read_signal_and_update > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140 > #8 0xc1351f in read_signal_done > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185 > #9 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*) > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939 > #10 0xbbba59 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) > /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:489 > #11 0xbda09c in NetHandler::mainNetEvent(int, Event*) > /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513 > #12 0xc85089 in Continuation::handleEvent(int, void*) > /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146 > #13 0xc85089 in EThread::process_event(Event*, int) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144 > #14 0xc85089 in EThread::execute() > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268 > #15 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826 > #16 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4) > Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here: > #0 0x2aaf08be286a in __interceptor_pthread_create > ../../.././libsanitizer/asan/asan_interceptors.cc:183 > #1 0xc81aa5 in ink_thread_create ../../lib/ts/ink_thread.h:148 > #2 0xc81aa5 in Thread::start(char const*, unsigned long, void* > (*)(void*), void*) > /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:103 > #3 0xc8a026 in EventProcessor::start(int, unsigned long) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:140 > #4 0x498d0b in main /usr/local/src/trafficserver/proxy/Main.cc:1636 > #5 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4) > SUMMARY: AddressSanitizer: heap-use-after-free > ../iocore/eventsystem/I_Continuation.h:146 Continuation::handleEvent(int, > void*) > Shadow bytes around the buggy address: > 0x0c307fff98c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff98d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff98e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff98f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa > 0x0c307fff9900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > =>0x0c307fff9910: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Heap right redzone: fb > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack partial redzone: f4 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Contiguous container OOB:fc > ASan internal: fe > ==31101==ABORTING > traffic_server: using root directory '/opt/ats' > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)