[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987611#comment-13987611
]
Yunkai Zhang commented on TS-2767:
--
Ah, sorry, I'm also not an STL expert:(
Totally agree to remove STL, they just be used for quickly implementation.
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
> Attachments: ts2767.diff
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987334#comment-13987334
]
Sudheer Vinukonda commented on TS-2767:
---
coz, std::string.clear() does not free the memory - it will logically set the
string to an empty string, but, not release the memory. You may write a simple
test program to confirm this behavior. The only way you could guarantee freeing
the memory from std::string object is by swapping it out with an empty string
object. In any case, as noted by Leif above, STL constructs use heap memory and
are not efficient. Their use should be avoided during call processing for
better performance.
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
> Attachments: ts2767.diff
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_sessi
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987269#comment-13987269
]
Yunkai Zhang commented on TS-2767:
--
Why not use {{string.clear()}} ? I bet it should be more efficient than
{{string.swap()}}
{code}
void
SpdyRequest::clear()
{
...
url.clear();
...
}
{code}
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
> Attachments: ts2767.diff
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHa
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986897#comment-13986897
]
ASF subversion and git services commented on TS-2767:
-
Commit b3a98049645c6f9a238eac7a9c319be2ff435383 in trafficserver's branch
refs/heads/master from [~bcall]
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=b3a9804 ]
added TS-2767 to changes
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
> Attachments: ts2767.diff
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986894#comment-13986894
]
ASF subversion and git services commented on TS-2767:
-
Commit 966f4a55a343283231dbbf7d8fe2ff75858182e2 in trafficserver's branch
refs/heads/master from [~sudheerv]
[ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=966f4a5 ]
TS-2767: ATS Memory Leak related to SPDY
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
> Attachments: ts2767.diff
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAn
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986806#comment-13986806
]
Sudheer Vinukonda commented on TS-2767:
---
Just to clarify - the reason the std::string objects are not being released
(before the fix) is due to the usage of freelist in ATS. The destructor of
SpdyRequest doesn't do anything (other than put back the object in the
freelist), so, the destructor of string objects never get called.
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
> Attachments: ts2767.diff
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986328#comment-13986328
]
Sudheer Vinukonda commented on TS-2767:
---
Thanks, Leif - Agree, will open a new jira to rewrite some aspects of the
current spdy implementation (e.g replace TS API w/ core API, replace
std::string w/ char* etc).
In the mean time, the fix wrt the memory leak seems to hold good so far. The
memory seems to be stable (10g RES + 13g VIRT) for the past 5+ hours. Will
continue to monitor further.
On a side note though, the host running master ATS seems to consume higher
memory than a host running 4.0.2, by about 2g on average (there might be a
slower leak, when SPDY is disabled - remember seeing master ATS (w/ spdy
disabled) grow upto 19g compare to 8g on a 4.0.2 host, when run for more than 2
days). Need to run longer and analyze further.
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
> Fix For: 5.0.0
>
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*)
[jira] [Commented] (TS-2767) ATS Memory Leak related to SPDY
[
https://issues.apache.org/jira/browse/TS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986303#comment-13986303
]
Leif Hedstrom commented on TS-2767:
---
We really need to go back and fix the code using std::string, and other STL
constructs in our critical path. That's not cool.
> ATS Memory Leak related to SPDY
> ---
>
> Key: TS-2767
> URL: https://issues.apache.org/jira/browse/TS-2767
> Project: Traffic Server
> Issue Type: Bug
> Components: SPDY
>Reporter: Sudheer Vinukonda
> Labels: SPDY_ATS, yahoo
>
> During production testing of SPDY, we noticed ATS's memory quickly grows to
> about 20g RES and 31g VIRT and eventually becomes unresponsive. dstat shows a
> lot of disk/paging activity.
> Running Valgrind on a single request (using spdycat) shows the below possible
> leaks.
> {code}
> ==23000== 103 bytes in 1 blocks are possibly lost in loss record 3,133 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD19A: std::string::_Rep::_M_clone(std::allocator
> const&, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CD5EB: std::string::reserve(unsigned long) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CDABE: std::string::append(std::string const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D28DE: spdy_fetcher_launch(SpdyRequest*, TSFetchMethod)
> (basic_string.h:2165)
> ==23000==by 0x5D379D: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (SpdyCallbacks.cc:294)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==
> ==23000== 128 bytes in 1 blocks are possibly lost in loss record 3,256 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x5D3C8E: std::vector,
> std::allocator >
> >::_M_insert_aux(__gnu_cxx::__normal_iterator std::string>*, std::vector,
> std::allocator > > >,
> std::pair const&) (new_allocator.h:89)
> ==23000==by 0x5D32CF: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_vector.h:741)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(int, Event*)
> (UnixNet.cc:384)
> ==23000==by 0x71059E: EThread::process_event(Event*, int)
> (I_Continuation.h:146)
> ==23000==by 0x710DCA: EThread::execute() (UnixEThread.cc:269)
> ==23000==
> ==23000== 173 bytes in 5 blocks are possibly lost in loss record 3,303 of
> 3,823
> ==23000==at 0x4C285BC: operator new(unsigned long)
> (vg_replace_malloc.c:298)
> ==23000==by 0x72CC3C8: std::string::_Rep::_S_create(unsigned long,
> unsigned long, std::allocator const&) (in
> /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCDE4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x72CCF32: std::basic_string,
> std::allocator >::basic_string(char const*, std::allocator
> const&) (in /usr/lib64/libstdc++.so.6.0.13)
> ==23000==by 0x5D32A2: spdy_on_ctrl_recv_callback(spdylay_session*,
> spdylay_frame_type, spdylay_frame*, void*) (stl_pair.h:101)
> ==23000==by 0x715E9F: spdylay_session_on_syn_stream_received
> (spdylay_session.c:1782)
> ==23000==by 0x716BA6: spdylay_session_mem_recv (spdylay_session.c:2246)
> ==23000==by 0x716DD8: spdylay_session_recv (spdylay_session.c:2828)
> ==23000==by 0x5D0E57: spdy_default_handler(tsapi_cont*, TSEvent, void*)
> (SpdySM.cc:263)
> ==23000==by 0x6EC1A6: UnixNetVConnection::readSignalAndUpdate(int)
> (I_Continuation.h:146)
> ==23000==by 0x6DC779: SSLNetVConnection::net_read_io(NetHandler*,
> EThread*) (SSLNetVConnection.cc:294)
> ==23000==by 0x6E39F1: NetHandler::mainNetEvent(
