Hello Michael and all,
I believe I have located the source of the memory leak. I implemented a fix in
my source tree, but due to my lack of familiarity with APR, I am not sure
whether there are any issues with my change. The problem lies in the fact that
the etch_queue mutexes are created in the context of the g_etch_main_pool, the
global apr_pool, and therefore will not be deallocated until
etch_runtime_shutdown() is called.
The patch from svn diff is shown below. If there are no issues with this
change, I would like to check it in and will close the JIRA issue I submitted.
Thanks, and all the best,
--thomas
Index: common/etch_mutex.c
===================================================================
--- common/etch_mutex.c (revision 1173358)
+++ common/etch_mutex.c (working copy)
@@ -68,7 +68,7 @@
ETCH_ASSERT(newmutex != NULL);
apr_thread_mutex_lock(g_etch_main_pool_mutex);
- apr_status = apr_thread_mutex_create(&apr_mutex, flags, g_etch_main_pool);
+ apr_status = apr_thread_mutex_create(&apr_mutex, flags, pool);
apr_thread_mutex_unlock(g_etch_main_pool_mutex);
if(apr_status != APR_SUCCESS) {
char temp[1024];
Index: transport/etch_plain_mailbox.c
===================================================================
--- transport/etch_plain_mailbox.c (revision 1173351)
+++ transport/etch_plain_mailbox.c (working copy)
@@ -110,13 +110,14 @@
do
{
+ if (NULL == (queue = new_queue(capacity))) break;
+
// TODO: pool
- status = etch_mutex_create(&mutex, ETCH_MUTEX_UNNESTED, NULL);
+ status = etch_mutex_create(&mutex, ETCH_MUTEX_UNNESTED,
queue->subpool);
if(status != ETCH_SUCCESS) {
// error
break;
}
- if (NULL == (queue = new_queue(capacity))) break;
/* - - - - - - - - - - - - - - -
* i_mailbox
-----Original Message-----
From: Thomas Marsh
Sent: Thursday, September 15, 2011 2:41 PM
To: '[email protected]'
Subject: RE: Etch/C Memory Consumption
Hello Michael,
I have started to investigate the memory consumption issue in more detail, and
can confirm it is leaking around 32 bytes per call of say_hello(). That is
about the size of the User helloworld_object and the user->name wchar_t which
is allocated in the client loop, but clearly must be a copy of that data if
this is the issue. The leak traceback is below (for a run with a loop length of
1000):
32,768 bytes in 4 blocks are possibly lost in loss record 567 of 567
at malloc (vg_replace_malloc.c:195)
by apr_pool_create_ex (apr_pools.c:344)
by new_queue (etch_queue.c:59)
by new_mailbox_a (etch_plain_mailbox.c:119)
by new_mailbox (etch_plain_mailbox.c:82)
by pmboxmgr_transport_call (etch_plain_mailbox_manager.c:543)
by tcpdelsvc_begincall (etch_transport.c:525)
by etchremote_begincall (etch_remote.c:129)
by helloworld_remote_begin_server_say_hello (helloworld_remote_server.c:217)
by helloworld_remote_server_say_hello (helloworld_remote_server.c:276)
by main (helloworld_client_main.c:183)
The leak size at this location varies depending on the number of times I
iterate over the say_hello() invocation. Here are the leak sizes from this
routine at various loop durations:
500 -> 16,384 bytes lost
1,000 -> 32,768 bytes lost
5,000 -> 155,648 bytes lost
10,000 -> 319,488 bytes lost
However, this memory is released upon a call to etch_runtime_shutdown(), so you
cannot see the leak unless you interrupt the program execution (or watch the
memory grow...).
As for my platform:
- Linux CentOS 5.4 (32 bit i686)
- libapr-1.4.5
- libapr-util-1.3.12
- libapr-iconv-1.2.1
- etch (latest from SVN)
- gcc 4.1.2
I hope that the traceback may give you some clue as to the source of the leak.
Otherwise, I will dig deeper when I find more time to focus on this again.
Viele Gruesse,
--thomas
-----Original Message-----
From: Michael Fitzner [mailto:[email protected]]
Sent: Wednesday, September 14, 2011 6:34 AM
To: [email protected]
Subject: AW: Etch/C Memory Consumption
Hi Thomas,
Your code should be correct and no memory leaks in this part. Could you provide
me with some more information about what systems do you use (Linux, Windows)
and a call stack of you memory leak if available. I will also try to reproduce
your test.
Thanks
Michael
-----Ursprüngliche Nachricht-----
Von: Thomas Marsh [mailto:[email protected]]
Gesendet: Dienstag, 13. September 2011 16:47
An: [email protected]
Betreff: RE: Etch/C Memory Consumption
Hello Martijn,
Thanks for your response. However, this is not the source of the leak. There is
a clear etch_object_destroy in the helloworld_remote_begin_server_say_hello()
which deallocates any parameters to the methods (meaning you will get a
segfault if you try to reuse the parameters which now no longer point to valid
memory). This usage semantic is also clearly stated in the C Binding notes
>From http://incubator.apache.org/etch/c-binding-tips-tricks.html:
The C Binding for Etch has the following memory management rules:
Implementation side: Parameters of functions have to be destroyed by
the function implementation using etch_object_destroy.
Caller side: Result Objects have to be freed by the caller using
etch_object_destroy. Parameters of calls will be freed by the runtime
automatically.
To reiterate, program memory usage in this very simple usage scenario _must_ be
stable. I suspect an ever growing hash table, or some other similar culprit
within the runtime. Can any of the Etch/C binding developers comment?
Thanks, and best regards,
--thomas
-----Original Message-----
From: Martijn Dashorst [mailto:[email protected]]
Sent: Tuesday, September 13, 2011 7:23 AM
To: [email protected]
Subject: Re: Etch/C Memory Consumption
And user->name = new_stringw(L"User"); will do so as well
Martijn
On Tue, Sep 13, 2011 at 1:52 PM, Martijn Dashorst <[email protected]>
wrote:
> user = new_helloworld_user()
>
> will allocate memory for each pass through the loop.
>
> Martijn
>
> On Mon, Sep 12, 2011 at 11:19 PM, Thomas Marsh <[email protected]>
> wrote:
>> Hello all,
>>
>> I have a question about memory consumption in the Etch/C runtime based on
>> behavior we are seeing within our C client. I have modified the C
>> implementation of the HelloWorld example in the distribution to run an
>> infinite loop of requests:
>>
>> In the main() routine of helloworld_client_main.c:
>>
>> ...
>> while (1) {
>> user = new_helloworld_user();
>> user->id = 5;
>> user->name = new_stringw(L"User");
>> result = remote->say_hello(remote,
>> user);
>> if (is_etch_exception(result)) {
>> ...
>> }
>> printf("%S\n", result->v.valw);
>> etch_object_destroy(result);
>> }
>> ...
>>
>> While running this, I see that the memory consumption of the client
>> continually grows. (In this example, it grows by about 1 mB every 5
>> seconds.) I cannot see the memory leak when testing with valgrind, so it
>> would suggest that the leak is in Etch managed memory which is cleared at
>> exit.
>>
>> My understanding is that the call to remote->say_hello() should delegate
>> responsibility of deallocation of the parameters to the Etch runtime, and
>> that the client code is only responsible for deallocating the result object.
>> The memory use should be stable within this tight loop. Can anyone comment
>> on the potential cause of the memory consumption?
>>
>> Thanks, and best regards,
>>
>> --thomas
>>
>
>
>
> --
> Become a Wicket expert, learn from the best: http://wicketinaction.com
>
--
Become a Wicket expert, learn from the best: http://wicketinaction.com
