Daniel, et. al.: First, thanks for responding. Second, the MemScope output I included doesn't show order of execution, unfortunately, and I apologize for the confusion. It's not true that Curl_add_buffer_init() is called AFTER Curl_add_buffer(). The _init() routine is executed first, as you'd expect.
I've done additional analysis, to identify the source lines that are being called. I don't know anything about the internals of the cURL library (yet), so I have no intuition about what might be going on here. But here's what I do now know: * The 20 byte buffer being leaked is allocated at line 1188 (using 7.21.3 code) of http.c - the one call to realloc() in Curl_add_buffer(). * This same line 1188 is executed three times during a single curlMlti_perform(). The first time, the calling sequence is Curl_do(line 5336)->Curl_http(line 2537)->Curl_add_buffer(line 1188), and 20 bytes are allocated - new_size is 20 - and this is the 20 bytes that is being leaked. * The second time line 1188 is executed the calling sequence is Curl_do(line 5336)->Curl_http(line 2541)->Curl_add_bufferf(line 1145)->Curl_add_buffer(line 188), new_size is 122, and this allocation is properly freed. * The third time line 1188 is executed the calling sequence is again Curl_do(line 5336)->Curl_http(line 2874)->Curl_add_buffer(line 1188), new_size is 488, and again is properly freed. It is always that first 20 byte allocation that is getting leaked. Does this help clarify the situation at all? Matt Fisher ------- On Thu, 17 Feb 2011, Fisher, Matt wrote: > * Wind River's MemScope tool is used to analyze the allocation and freeing > of dynamic memory blocks. MemScope logs every single dynamic memory > allocation & free, so when a test run is complete it is easy to see any > blocks that have not been freed. It's a pity it doesn't properly record source+line as well (hello valgrind) as then it'd be slightly easier to fully understand this... Can you figure out exactly which Curl_add_buffer() call that does this 20 byte malloc? > Curl_http;1;0;20;20 <--- curl/http.c::Curl_http() > . Curl_add_buffer;1;0;20;20 <--- curl/http.c::Curl_add_buffer() > . . malloc;1;0;20;20 <--- VxWorks malloc() > . Curl_http;1;1;12;0 > . . Curl_add_buffer_init;1;1;12;0 > . . . calloc;1;1;12;0 > . Curl_http;1;1;488;0 > . . Curl_add_buffer;1;1;488;0 What looks suspicious in this flow is that Curl_add_buffer_init() is called _after_ that first 20-byte Curl_add_buffer() call. It feels like that might should be investigated closer. If you can just set a break-point in there and see... -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Fisher, Matt Sent: Thursday, February 17, 2011 12:33 PM To: [email protected] Subject: Apparent memory leak deep in cURL library; requesting assistance I am trying to understand why a 20-byte memory buffer is being leaked every time my code issues an XMLRPC client call. I have ported XMLRPC-C 1.61.31(stable) to Wind River VxWorks 6.4, and am using the client functionality only. The XMLRPC server resides on a remote CPU. Configuration/setup: * XMLRPC-C library 1.16.31, downloaded from http://xmlrpc-c.sourceforge.net/ and ported as needed to build & run using Wind River Workbench (GNU C) and VxWorks 6.4. Supporting sub-libraries (util, libutil, curl_transport, expat) are also included in the Workbench project and built. * cURL library 7.21.3, downloaded from http://curl.haxx.se/ and built with XMLRPC-C into a single downloadable kernel module (library). * Application-specific functions sitting above XMLRPC-C that make use of the global client and basic client-side APIs (xmlrpc_client_call2, xmlrpc_read_int, etc.). * All application-level code is written in C and follows the code examples and documented instructions found at http://xmlrpc-c.sourceforge.net/. I made no functional changes to the cURL library. I simply took the .c and .h files found in the /include/curl and /lib directories of the downloaded source archive and built them into a library using GNU C under Workbench. The only changes I made were to add conditional compile statements to header file include paths, to meet the specific project build needs. Summary of testing: * Application code and XMLRPC/cURL library is running on a simulated VxWorks target using Wind River's VxSim capability. Code is debugged and analyzed using the tools supplied with Wind River Workbench, version 2.6. Simulated VxWorks target has networking capability. * XMLRPC server is a remote Linux host accessed over the internet. * My application has over 60 API functions, each of which issues one or two XMLRPCs to the remote XMLRPC server. All API functions work correctly. I have successfully retrieved scalar values, strings, structures, and arrays of structures from the remote server and processed them correctly in the application-level code. * API functions are run in a loop, and each tested API function has been run one time, ten times, and 100 times to assess dynamic memory behavior. * Wind River's MemScope tool is used to analyze the allocation and freeing of dynamic memory blocks. MemScope logs every single dynamic memory allocation & free, so when a test run is complete it is easy to see any blocks that have not been freed. Summary of results: * For every test run, regardless of top-level API function tested and regardless of the number of iterations the API function is run, there is a 20-byte memory leak per RPC. API functions that issue one RPC leak 20 bytes, those that issue two RPCs leak 40 bytes. Perfectly repeatable results, 100% consistent. * The memory leak occurs deep within the cURL library code (stack trace below), when the function http.c::Curl_http() calls curl_add_buffer() and the CURLM state is CURLM_STATE_DO. This occurs during the xmlrpc_curl_transport.c::performRpc() execution. MemScope stack trace: What follows is a text dump of the MemScope output, for a single execution of one API function (that makes a single RPC). This trace shows where the memory leak is observed. The set of four semi-colon-delimited integers after each function name are MemScope data. The only important integer is the last one - it is the current number of bytes allocated & not freed. The last integer of every line should be 0 upon completion of task execution. This trace was taken after the test had fully completed. I have annotated the lowest five levels of the stack trace to call attention to the source of the 20-byte leak. This repeatable leak is a threat to system stability. . . issm_running_trial;11;10;1539;20 . . . xmlrpc_client_call;11;10;1539;20 . . . . clientCall_va;11;10;1539;20 . . . . . xmlrpc_client_call2;8;7;1539;20 . . . . . . call;8;7;1539;20 . . . . . . . performRpc;8;7;1539;20 . . . . . . . . performCurlTransaction;8;7;1539;20 . . . . . . . . . finishCurlMulti;8;7;1539;20 . . . . . . . . . . doCurlWork;8;7;1539;20 . . . . . . . . . . . curlMulti_perform;8;7;1539;20 . . . . . . . . . . . . curl_multi_perform;8;7;1539;20 . . . . . . . . . . . . . multi_runsingle;4;3;535;20 <--- curl/multi.c::multi_runsingle() . . . . . . . . . . . . . . Curl_do;4;3;535;20 <--- curl/url.c::Curl_do() . . . . . . . . . . . . . . . Curl_http;1;0;20;20 <--- curl/http.c::Curl_http() . . . . . . . . . . . . . . . . Curl_add_buffer;1;0;20;20 <--- curl/http.c::Curl_add_buffer() . . . . . . . . . . . . . . . . . malloc;1;0;20;20 <--- VxWorks malloc() . . . . . . . . . . . . . . . Curl_http;1;1;12;0 . . . . . . . . . . . . . . . . Curl_add_buffer_init;1;1;12;0 . . . . . . . . . . . . . . . . . calloc;1;1;12;0 . . . . . . . . . . . . . . . Curl_http;1;1;488;0 . . . . . . . . . . . . . . . . Curl_add_buffer;1;1;488;0 . . . . . . . . . . . . . . . . . malloc;1;1;488;0 . . . . . . . . . . . . . . . Curl_http;1;1;15;0 . . . . . . . . . . . . . . . . strdup;1;1;15;0 . . . . . . . . . . . . . . . . . malloc;1;1;15;0 . . . . . . . . . . . . . multi_runsingle;2;2;20;0 . . . . . . . . . . . . . multi_runsingle;1;1;964;0 . . . . . . . . . . . . . multi_runsingle;1;1;20;0 . . . . . xmlrpc_client_call2;3;3;80;0 I would appreciate any guidance, suggestions, conclusions (or questions for additional information) regarding this memory leak. Is this an actual leak in the released code that needs to be fixed? Or is there a probable error with the way I've built and use XMLRPC/cURL that is manifesting as this leak? Thank you for any feedback you are willing to give. Matt Fisher [email protected] ------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html ------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
