Re: [Dbmail-dev] dbmail-imapd memory leak - one more time...
Aaron Stone wrote: This might sound stupid, but is your dbmail.conf unusually large? Like maybe some filesystem corruption concatenated it together with a 71MB file from someplace else? It would be a possibility if the amount of wasted memory was not tied directly to the amount of messages processed. I got only 17MB (not a typo) with 1000 messages. I can't imagine that g_key_file_new is allocating such a large slab; I assume that this is just the first function to request the slab, but you did set the environment variables which turn off the slab allocator, right? No, the only time it was turned off was in http://rabbit.us/pool/misc/dbmail_bug584_memcheck_svn2574_always_malloc.tar.bz2, everything else was with the thing turned on. I heavily modified the script, now it runs 6 tests total - with 1000 2000 and 3000 messages, both with and without the allocator. Memory consumption is logged also at 10 second intervals. It is running right now but will take quite a while, I should have result around 1pm CET. Since the amount of data keeps growing, would you like me to compress the thing with rar, or a larger bzip2 is preferable? Or do I scrap the level 5 logs altogether? Thanks Peter ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak - one more time...
On Thu, 2007-05-17 at 12:34 +0200, Peter Rabbitson wrote: Aaron Stone wrote: This might sound stupid, but is your dbmail.conf unusually large? Like maybe some filesystem corruption concatenated it together with a 71MB file from someplace else? It would be a possibility if the amount of wasted memory was not tied directly to the amount of messages processed. I got only 17MB (not a typo) with 1000 messages. Just making sure that we're chasing the right issues. Given that we have seen by inspection areas of code that aren't doing what we thought they did (stupid g_list_foreach going only along the next- link of a doubly linked list!) I wanted to rule out other weirdness. I can't imagine that g_key_file_new is allocating such a large slab; I assume that this is just the first function to request the slab, but you did set the environment variables which turn off the slab allocator, right? No, the only time it was turned off was in http://rabbit.us/pool/misc/dbmail_bug584_memcheck_svn2574_always_malloc.tar.bz2, everything else was with the thing turned on. I heavily modified the script, now it runs 6 tests total - with 1000 2000 and 3000 messages, both with and without the allocator. Memory consumption is logged also at 10 second intervals. It is running right now but will take quite a while, I should have result around 1pm CET. Since the amount of data keeps growing, would you like me to compress the thing with rar, or a larger bzip2 is preferable? Or do I scrap the level 5 logs altogether? bzip2 is fine with me, the data set is still a manageable size. I bet we could scrap the level 5 logs. All that matters is pinpointing the functions that originate the lost memory allocations. Aaron ___ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak - one more time...
Peter,
On second look please test this patch against rev 2579 vanila:
Index: dbmail-mailbox.c
===
--- dbmail-mailbox.c(revision 2579)
+++ dbmail-mailbox.c(working copy)
@@ -1313,7 +1313,7 @@
GTree * dbmail_mailbox_get_set(struct DbmailMailbox *self, const char
*set, gboolean uid)
{
- GList *ids = NULL, *sets = NULL;
+ GList *topids, *ids = NULL, *sets = NULL;
GString *t;
char *rest;
u64_t i, l, r, lo = 0, hi = 0;
@@ -1330,12 +1330,13 @@
TRACE(TRACE_DEBUG,[%s] uid [%d], set, uid);
if (uid) {
- ids = g_tree_keys(self-ids);
- ids = g_list_last(ids);
+ topids = g_tree_keys(self-ids);
+ ids = g_list_last(topids);
hi = *((u64_t *)ids-data);
- ids = g_list_first(ids);
+ ids = g_list_first(topids);
lo = *((u64_t *)ids-data);
- g_list_free(g_list_first(ids));
+ //g_list_free(g_list_first(ids));
+ g_list_free(topids);
} else {
lo = 1;
hi = g_tree_nnodes(self-ids);
I think it was cause you tried to use my patch from the old code and
Paul's changes might have not helped. let me know
Thanks,
Leif
Peter Rabbitson wrote:
Leif Jackson wrote:
Peter try 2579 with my changes from the previous message.
Same. Logs at
http://rabbit.us/pool/misc/bug584_memcheck_svn2579_LJpatch.tar.bz2
Peter
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
Re: [Dbmail-dev] dbmail-imapd memory leak - one more time...
On Wed, May 16, 2007, Peter Rabbitson [EMAIL PROTECTED] said:
Leif Jackson wrote:
Peter,
On second look please test this patch against rev 2579 vanila:
Index: dbmail-mailbox.c
===
--- dbmail-mailbox.c(revision 2579)
+++ dbmail-mailbox.c(working copy)
@@ -1313,7 +1313,7 @@
GTree * dbmail_mailbox_get_set(struct DbmailMailbox *self, const char
*set, gboolean uid)
{
- GList *ids = NULL, *sets = NULL;
+ GList *topids, *ids = NULL, *sets = NULL;
GString *t;
char *rest;
u64_t i, l, r, lo = 0, hi = 0;
@@ -1330,12 +1330,13 @@
TRACE(TRACE_DEBUG,[%s] uid [%d], set, uid);
if (uid) {
- ids = g_tree_keys(self-ids);
- ids = g_list_last(ids);
+ topids = g_tree_keys(self-ids);
+ ids = g_list_last(topids);
hi = *((u64_t *)ids-data);
- ids = g_list_first(ids);
+ ids = g_list_first(topids);
lo = *((u64_t *)ids-data);
- g_list_free(g_list_first(ids));
+ //g_list_free(g_list_first(ids));
+ g_list_free(topids);
} else {
lo = 1;
hi = g_tree_nnodes(self-ids);
I think it was cause you tried to use my patch from the old code and
Paul's changes might have not helped. let me know
Sigh... still same. Here is the relevant part:
==5570== 71,330,912 bytes in 287,652 blocks are still reachable in loss
record 26 of 26
==5570==at 0x40226DB: memalign (vg_replace_malloc.c:332)
==5570==by 0x4022735: posix_memalign (vg_replace_malloc.c:425)
==5570==by 0x415E0A3: (within /usr/lib/libglib-2.0.so.0.1200.12)
==5570==by 0x415EAA3: g_slice_alloc (in
/usr/lib/libglib-2.0.so.0.1200.12)
==5570==by 0x4145725: g_list_prepend (in
/usr/lib/libglib-2.0.so.0.1200.12)
==5570==by 0x4140AF7: (within /usr/lib/libglib-2.0.so.0.1200.12)
==5570==by 0x4140B49: g_key_file_new (in
/usr/lib/libglib-2.0.so.0.1200.12)
==5570==by 0x41C5B8C: config_read (config.c:46)
==5570==by 0x41DEE3B: DoConfig (serverparent.c:233)
==5570==by 0x41E02AB: serverparent_getopt (serverparent.c:144)
==5570==by 0x8052435: main (imapd.c:45)
==5570==
Let me know if you need the full logs as before. Off to bed now,
hopefully we can work on this further tomorrow.
Thank you all for the effort!
This might sound stupid, but is your dbmail.conf unusually large? Like
maybe some filesystem corruption concatenated it together with a 71MB file
from someplace else?
I can't imagine that g_key_file_new is allocating such a large slab; I
assume that this is just the first function to request the slab, but you
did set the environment variables which turn off the slab allocator,
right?
Aaron
___
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://twister.fastxs.net/mailman/listinfo/dbmail-dev
