Hi Thomas, Thanks for further investigations. I will check your patch and give feedback. It would be still great if you could attach the patch file to Jira https://issues.apache.org/jira/browse/ETCH-176
Cheers Michael 2011/9/20 Thomas Marsh <[email protected]>: > Hello Michael and all, > > I believe I have located the source of the memory leak. I implemented a fix > in my source tree, but due to my lack of familiarity with APR, I am not sure > whether there are any issues with my change. The problem lies in the fact > that the etch_queue mutexes are created in the context of the > g_etch_main_pool, the global apr_pool, and therefore will not be deallocated > until etch_runtime_shutdown() is called. > > The patch from svn diff is shown below. If there are no issues with this > change, I would like to check it in and will close the JIRA issue I submitted. > > Thanks, and all the best, > > --thomas > > Index: common/etch_mutex.c > =================================================================== > --- common/etch_mutex.c (revision 1173358) > +++ common/etch_mutex.c (working copy) > @@ -68,7 +68,7 @@ > ETCH_ASSERT(newmutex != NULL); > > apr_thread_mutex_lock(g_etch_main_pool_mutex); > - apr_status = apr_thread_mutex_create(&apr_mutex, flags, > g_etch_main_pool); > + apr_status = apr_thread_mutex_create(&apr_mutex, flags, pool); > apr_thread_mutex_unlock(g_etch_main_pool_mutex); > if(apr_status != APR_SUCCESS) { > char temp[1024]; > Index: transport/etch_plain_mailbox.c > =================================================================== > --- transport/etch_plain_mailbox.c (revision 1173351) > +++ transport/etch_plain_mailbox.c (working copy) > @@ -110,13 +110,14 @@ > > do > { > + if (NULL == (queue = new_queue(capacity))) break; > + > // TODO: pool > - status = etch_mutex_create(&mutex, ETCH_MUTEX_UNNESTED, NULL); > + status = etch_mutex_create(&mutex, ETCH_MUTEX_UNNESTED, > queue->subpool); > if(status != ETCH_SUCCESS) { > // error > break; > } > - if (NULL == (queue = new_queue(capacity))) break; > > /* - - - - - - - - - - - - - - - > * i_mailbox > > -----Original Message----- > From: Thomas Marsh > Sent: Thursday, September 15, 2011 2:41 PM > To: '[email protected]' > Subject: RE: Etch/C Memory Consumption > > Hello Michael, > > I have started to investigate the memory consumption issue in more detail, > and can confirm it is leaking around 32 bytes per call of say_hello(). That > is about the size of the User helloworld_object and the user->name wchar_t > which is allocated in the client loop, but clearly must be a copy of that > data if this is the issue. The leak traceback is below (for a run with a loop > length of 1000): > > 32,768 bytes in 4 blocks are possibly lost in loss record 567 of 567 > at malloc (vg_replace_malloc.c:195) > by apr_pool_create_ex (apr_pools.c:344) > by new_queue (etch_queue.c:59) > by new_mailbox_a (etch_plain_mailbox.c:119) > by new_mailbox (etch_plain_mailbox.c:82) > by pmboxmgr_transport_call (etch_plain_mailbox_manager.c:543) > by tcpdelsvc_begincall (etch_transport.c:525) > by etchremote_begincall (etch_remote.c:129) > by helloworld_remote_begin_server_say_hello > (helloworld_remote_server.c:217) > by helloworld_remote_server_say_hello (helloworld_remote_server.c:276) > by main (helloworld_client_main.c:183) > > The leak size at this location varies depending on the number of times I > iterate over the say_hello() invocation. Here are the leak sizes from this > routine at various loop durations: > > 500 -> 16,384 bytes lost > 1,000 -> 32,768 bytes lost > 5,000 -> 155,648 bytes lost > 10,000 -> 319,488 bytes lost > > However, this memory is released upon a call to etch_runtime_shutdown(), so > you cannot see the leak unless you interrupt the program execution (or watch > the memory grow...). > > As for my platform: > - Linux CentOS 5.4 (32 bit i686) > - libapr-1.4.5 > - libapr-util-1.3.12 > - libapr-iconv-1.2.1 > - etch (latest from SVN) > - gcc 4.1.2 > > I hope that the traceback may give you some clue as to the source of the > leak. Otherwise, I will dig deeper when I find more time to focus on this > again. > > Viele Gruesse, > > --thomas > > -----Original Message----- > From: Michael Fitzner [mailto:[email protected]] > Sent: Wednesday, September 14, 2011 6:34 AM > To: [email protected] > Subject: AW: Etch/C Memory Consumption > > Hi Thomas, > Your code should be correct and no memory leaks in this part. Could you > provide me with some more information about what systems do you use (Linux, > Windows) and a call stack of you memory leak if available. I will also try to > reproduce your test. > > Thanks > Michael > > -----Ursprüngliche Nachricht----- > Von: Thomas Marsh [mailto:[email protected]] > Gesendet: Dienstag, 13. September 2011 16:47 > An: [email protected] > Betreff: RE: Etch/C Memory Consumption > > Hello Martijn, > > Thanks for your response. However, this is not the source of the leak. There > is a clear etch_object_destroy in the > helloworld_remote_begin_server_say_hello() which deallocates any parameters > to the methods (meaning you will get a segfault if you try to reuse the > parameters which now no longer point to valid memory). This usage semantic is > also clearly stated in the C Binding notes > > From http://incubator.apache.org/etch/c-binding-tips-tricks.html: > > The C Binding for Etch has the following memory management rules: > > Implementation side: Parameters of functions have to be destroyed by > the function implementation using etch_object_destroy. > Caller side: Result Objects have to be freed by the caller using > etch_object_destroy. Parameters of calls will be freed by the runtime > automatically. > > To reiterate, program memory usage in this very simple usage scenario _must_ > be stable. I suspect an ever growing hash table, or some other similar > culprit within the runtime. Can any of the Etch/C binding developers comment? > > Thanks, and best regards, > > --thomas > > -----Original Message----- > From: Martijn Dashorst [mailto:[email protected]] > Sent: Tuesday, September 13, 2011 7:23 AM > To: [email protected] > Subject: Re: Etch/C Memory Consumption > > And user->name = new_stringw(L"User"); will do so as well > > Martijn > > On Tue, Sep 13, 2011 at 1:52 PM, Martijn Dashorst > <[email protected]> wrote: >> user = new_helloworld_user() >> >> will allocate memory for each pass through the loop. >> >> Martijn >> >> On Mon, Sep 12, 2011 at 11:19 PM, Thomas Marsh <[email protected]> >> wrote: >>> Hello all, >>> >>> I have a question about memory consumption in the Etch/C runtime based on >>> behavior we are seeing within our C client. I have modified the C >>> implementation of the HelloWorld example in the distribution to run an >>> infinite loop of requests: >>> >>> In the main() routine of helloworld_client_main.c: >>> >>> ... >>> while (1) { >>> user = new_helloworld_user(); >>> user->id = 5; >>> user->name = new_stringw(L"User"); >>> result = remote->say_hello(remote, >>> user); >>> if (is_etch_exception(result)) { >>> ... >>> } >>> printf("%S\n", result->v.valw); >>> etch_object_destroy(result); >>> } >>> ... >>> >>> While running this, I see that the memory consumption of the client >>> continually grows. (In this example, it grows by about 1 mB every 5 >>> seconds.) I cannot see the memory leak when testing with valgrind, so it >>> would suggest that the leak is in Etch managed memory which is cleared at >>> exit. >>> >>> My understanding is that the call to remote->say_hello() should delegate >>> responsibility of deallocation of the parameters to the Etch runtime, and >>> that the client code is only responsible for deallocating the result >>> object. The memory use should be stable within this tight loop. Can anyone >>> comment on the potential cause of the memory consumption? >>> >>> Thanks, and best regards, >>> >>> --thomas >>> >> >> >> >> -- >> Become a Wicket expert, learn from the best: http://wicketinaction.com >> > > > > -- > Become a Wicket expert, learn from the best: http://wicketinaction.com >
