I am trying to resolve a memory leak issue with ATS 7.0 version.
We are using ATS from the CDN perspective and developed 3 plugins.
DevOps complains that traffic_server process gradually consumes lot of memory
and does release back which is a sign of memory leak.
I tried with Valgrind but it didn’t work because of LUA and one of our ATS
community member suggested to use Address Sanitizer.
I followed exactly the steps mentioned Debugging Traffic Server using ASAN
<https://cwiki.apache.org/confluence/download/attachments/56066455/summit_asan.pptx?version=1&modificationDate=1429916307000&api=v2>.
I have build the traffic server with CXXFLAGS=-fno-omit-frame-pointer
-fsanitize=address
Verified that traffic_server is build with ASAN library by 'ldd
bin/traffic_server’
Restarted the traffic server with 'service nm-trafficserver restart'.
I have adopted three approaches to catch the memory leak
Approach 1:
Just running the traffic server(with -f option) and continuously sending HTTP
requests using wrk (from another machine) by attaching
to a gdb .
' gdb $(pidof traffic_server)’
break __asan_report_error.
Issue: I didn’t hit this function from the memory leak perspective.
Approach 2:
I have run the traffic server with PROXY_AUTO_EXIT and sending the http
requests
PROXY_AUTO_EXIT=30 ASAN_OPTIONS=detect_leaks=1:verbosity=2 bin/traffic_server
-f
Issue: Tool didn’t complain any thing about the memory leak.
Approach 3:
I have created simple code explicitly introducing the memory leak
#include <stdlib.h>
void dummy() {
malloc(20);
}
int main() {
dummy();
return 0;
}
Compiled with g++ -fsanitize=address add.cpp
Executed like ASAN_OPTIONS=detect_leaks=1 ./a.out
Issue: Tool didn’t detect this memory leak.
Production System Configuration:
16Gb Ram with 8 cores VM.
gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Did any one encounter this ASAN issue on Ubuntu 14.04 ? If so, how did they
resolve it ?
Any other suggestions how to narrow down the memory leak issue with ATS ?
Regards
Vamsi
PS: I am new to the community and new to Apache traffic server open source
module too.
