[ https://issues.apache.org/jira/browse/TS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335444#comment-14335444 ]
Susan Hinrichs commented on TS-3405: ------------------------------------ Yes, that is the same thing I was seeing. this->ua_session is null. I tried adding a null check in send_goaway_frame (see below), but now I'm in a state where ATS is kind of spinning. I'm running with two clients with three browsers between them. My ATS box is running at 170% CPU (with 2 CPUs on the VM I think). It isn't a tight spin. Generally two threads are active but it varies between stops. Looking at send_goaway_frame more closely, I see that it does a assert that ua_session is not NULL at the beginning of the function. I assume that in the process of handling the HTTP2_SESSION_EVENT_FINI, ua_session gets set to null. {code} void Http2ConnectionState::send_goaway_frame(Http2StreamId id, Http2ErrorCode ec) { Http2Frame frame(HTTP2_FRAME_TYPE_GOAWAY, 0, 0); Http2Goaway goaway; ink_assert(this->ua_session != NULL); goaway.last_streamid = id; goaway.error_code = ec; frame.alloc(buffer_size_index[HTTP2_FRAME_TYPE_GOAWAY]); http2_write_goaway(goaway, frame.write()); frame.finalize(HTTP2_GOAWAY_LEN); handleEvent(HTTP2_SESSION_EVENT_FINI, NULL); if (this->ua_session != NULL) { // xmit event MUTEX_LOCK(lock, this->ua_session->mutex, this_ethread()); this->ua_session->handleEvent(HTTP2_SESSION_EVENT_XMIT, &frame); } } {code} > Memory use after free in HTTP/2 > ------------------------------- > > Key: TS-3405 > URL: https://issues.apache.org/jira/browse/TS-3405 > Project: Traffic Server > Issue Type: Bug > Components: HTTP/2 > Reporter: Bryan Call > Fix For: 5.3.0 > > Attachments: fix-h2.patch > > > From Leif running on docs.trafficserver.apache.org: > > {code} > traffic_server: using root directory '/opt/ats' > ================================================================= > ==31101==ERROR: AddressSanitizer: heap-use-after-free on address > 0x61800000c888 at pc 0x4f3558 bp 0x2aaf10c88930 sp 0x2aaf10c88928 > READ of size 8 at 0x61800000c888 thread T2 ([ET_NET 1]) > #0 0x4f3557 in Continuation::handleEvent(int, void*) > ../iocore/eventsystem/I_Continuation.h:146 > #1 0x4f3557 in FetchSM::InvokePluginExt(int) > /usr/local/src/trafficserver/proxy/FetchSM.cc:301 > #2 0x4f3a7a in FetchSM::process_fetch_read(int) > /usr/local/src/trafficserver/proxy/FetchSM.cc:465 > #3 0x4f5112 in FetchSM::fetch_handler(int, void*) > /usr/local/src/trafficserver/proxy/FetchSM.cc:514 > #4 0x59f1b7 in Continuation::handleEvent(int, void*) > ../iocore/eventsystem/I_Continuation.h:146 > #5 0x59f1b7 in PluginVC::process_read_side(bool) > /usr/local/src/trafficserver/proxy/PluginVC.cc:640 > #6 0x5abcb9 in PluginVC::main_handler(int, void*) > /usr/local/src/trafficserver/proxy/PluginVC.cc:206 > #7 0xc821fe in Continuation::handleEvent(int, void*) > /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146 > #8 0xc821fe in EThread::process_event(Event*, int) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144 > #9 0xc84819 in EThread::execute() > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:238 > #10 0xc80e18 in spawn_thread_internal > /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:88 > #11 0x2aaf0b083df2 in start_thread (/lib64/libpthread.so.0+0x7df2) > #12 0x2aaf0c8ec1ac in clone (/lib64/libc.so.6+0xf61ac) > 0x61800000c888 is located 8 bytes inside of 816-byte region > [0x61800000c880,0x61800000cbb0) > freed by thread T0 ([ET_NET 0]) here: > #0 0x2aaf08c131c7 in __interceptor_free > ../../.././libsanitizer/asan/asan_malloc_linux.cc:62 > #1 0x7b7d42 in Http2ClientSession::do_io_close(int) > /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:194 > #2 0x7b7d42 in Http2ClientSession::main_event_handler(int, void*) > /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:237 > #3 0xc1351f in Continuation::handleEvent(int, void*) > ../../iocore/eventsystem/I_Continuation.h:146 > #4 0xc1351f in read_signal_and_update > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140 > #5 0xc1351f in read_signal_done > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185 > #6 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*) > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939 > #7 0xbbabf8 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) > /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:596 > #8 0xbda09c in NetHandler::mainNetEvent(int, Event*) > /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513 > #9 0xc85089 in Continuation::handleEvent(int, void*) > /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146 > #10 0xc85089 in EThread::process_event(Event*, int) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144 > #11 0xc85089 in EThread::execute() > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268 > #12 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826 > #13 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4) > previously allocated by thread T0 ([ET_NET 0]) here: > #0 0x2aaf08c1393b in __interceptor_posix_memalign > ../../.././libsanitizer/asan/asan_malloc_linux.cc:130 > #1 0x2aaf09afd2f9 in ats_memalign > /usr/local/src/trafficserver/lib/ts/ink_memory.cc:96 > #2 0x7cd804 in ClassAllocator<Http2ClientSession>::alloc() > ../../lib/ts/Allocator.h:124 > #3 0x7cd804 in Http2SessionAccept::accept(NetVConnection*, MIOBuffer*, > IOBufferReader*) > /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:57 > #4 0x7cd3c4 in Http2SessionAccept::mainEvent(int, void*) > /usr/local/src/trafficserver/proxy/http2/Http2SessionAccept.cc:69 > #5 0xbc2fae in SSLNextProtocolTrampoline::ioCompletionEvent(int, void*) > /usr/local/src/trafficserver/iocore/net/SSLNextProtocolAccept.cc:101 > #6 0xc1351f in Continuation::handleEvent(int, void*) > ../../iocore/eventsystem/I_Continuation.h:146 > #7 0xc1351f in read_signal_and_update > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:140 > #8 0xc1351f in read_signal_done > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:185 > #9 0xc1351f in UnixNetVConnection::readSignalDone(int, NetHandler*) > /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:939 > #10 0xbbba59 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) > /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:489 > #11 0xbda09c in NetHandler::mainNetEvent(int, Event*) > /usr/local/src/trafficserver/iocore/net/UnixNet.cc:513 > #12 0xc85089 in Continuation::handleEvent(int, void*) > /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:146 > #13 0xc85089 in EThread::process_event(Event*, int) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:144 > #14 0xc85089 in EThread::execute() > /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:268 > #15 0x498f96 in main /usr/local/src/trafficserver/proxy/Main.cc:1826 > #16 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4) > Thread T2 ([ET_NET 1]) created by T0 ([ET_NET 0]) here: > #0 0x2aaf08be286a in __interceptor_pthread_create > ../../.././libsanitizer/asan/asan_interceptors.cc:183 > #1 0xc81aa5 in ink_thread_create ../../lib/ts/ink_thread.h:148 > #2 0xc81aa5 in Thread::start(char const*, unsigned long, void* > (*)(void*), void*) > /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:103 > #3 0xc8a026 in EventProcessor::start(int, unsigned long) > /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:140 > #4 0x498d0b in main /usr/local/src/trafficserver/proxy/Main.cc:1636 > #5 0x2aaf0c817af4 in __libc_start_main (/lib64/libc.so.6+0x21af4) > SUMMARY: AddressSanitizer: heap-use-after-free > ../iocore/eventsystem/I_Continuation.h:146 Continuation::handleEvent(int, > void*) > Shadow bytes around the buggy address: > 0x0c307fff98c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff98d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff98e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff98f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa > 0x0c307fff9900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > =>0x0c307fff9910: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > 0x0c307fff9960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd > Shadow byte legend (one shadow byte represents 8 application bytes): > Addressable: 00 > Partially addressable: 01 02 03 04 05 06 07 > Heap left redzone: fa > Heap right redzone: fb > Freed heap region: fd > Stack left redzone: f1 > Stack mid redzone: f2 > Stack right redzone: f3 > Stack partial redzone: f4 > Stack after return: f5 > Stack use after scope: f8 > Global redzone: f9 > Global init order: f6 > Poisoned by user: f7 > Contiguous container OOB:fc > ASan internal: fe > ==31101==ABORTING > traffic_server: using root directory '/opt/ats' > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)