Aaron Stone wrote: > There are really two problems here: > > o During the course of a single imap session, the server must > clean up after every major imap operation. A client who remains > connected for a few hours and does some heavy imap must not > kill the server. > > o After a client disconnects, the server must really really free > all of the memory that it allocated. The likelihood of one process > handling back-to-back major imap sessions is not high. We don't want > each process growing to 100MB of 'hey i might need this' memory > when really most sessions are just a crapload of 'noop's to get > status updates.
Yes, those are my problems. > Current SVN should have all of the necessary fixes in place to survive > this. We may or may not do a 2.2.5rc3 release. Probably a good idea, > though. > And here comes my problem - it does not :( As a matter of fact the latest svn behaves exactly as the rc2 version. I just completed a test run with 10000 messages being synchronized 3 time back to back, with only 3 children so the same daemons will be reused again and again. I did this without valgrind as it the memory goes usage goes too high and the dev box starts trashing. The second run bombed in the middle, after one of the daemons balooned to 3GB. I can provide valgrind logs again, but it will most likely take a day of running. > Thanks for being extremely helpful with testing. We cannot commit to > making the next release perfect, but we can definitely try to make sure > that all major issus are resolved. Let me restate my question from the last mail - am I understanding correctly that this is not considered a major issue anymore? Yes the test test case might be harsh, but every _morning_ the place where I wanted to deploy dbmail moves an average of 2000 messages between 2 boxes. In this situation the risk of memory issues is too high for me. I have no problem with helping you guys track this further, but unless there is an agreement that the behavior I am experiencing is plain wrong, I probably have to look for other alternatives to dbmail. Let me know what do you guys think. Peter ============= Here is the memory state right before the bomb: Fri May 18 10:52:35 CEST 2007 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND dbmail 25574 0.0 0.1 6160 956 ? S 10:15 0:00 /usr/local/sbin/dbmail-imapd dbmail 25576 27.2 70.6 3138348 639112 ? S 10:15 10:08 /usr/local/sbin/dbmail-imapd dbmail 25577 0.7 0.4 14088 4516 ? S 10:15 0:16 /usr/local/sbin/dbmail-imapd ============ relevant parts of the level5 trace: ... May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+288): Executing command uid... May 18 10:52:40 Debug:[mailbox] dbmail-mailbox.c,dbmail_mailbox_get_set(+1330): [322363] uid [1] May 18 10:52:40 Debug:[misc] misc.c,g_tree_merge(+1267): (0x7560d8e0) (0x754742e0): a[0] [OR] b[1] -> a[1] May 18 10:52:40 Debug:[sql] dbmysql.c,db_query(+290): query [SELECT seen_flag, answered_flag, deleted_flag, flagged_flag, draft_flag, recent_flag, DATE_FORMAT( .Out of memory (Needed 57148 bytes) May 18 10:52:40 Debug:[imap] imapcommands.c,_ic_store(+1636): unable to retrieve msginfo (process:25576): GLib-CRITICAL **: g_tree_lookup: assertion `tree != NULL' failed May 18 10:52:40 Warning:[imap] imapcommands.c,_do_store(+1464): unable to lookup msginfo struct for [322363] May 18 10:52:40 Debug:[imapsession] dbmail-imapsession.c,dbmail_imap_session_printf(+1424): RESPONSE: [15106 OK UID STORE completed ] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+305): Finished command uid [0] .... May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+288): Executing command uid... May 18 10:52:40 Debug:[mailbox] dbmail-mailbox.c,dbmail_mailbox_get_set(+1330): [322366] uid [1] May 18 10:52:40 Debug:[misc] misc.c,g_tree_merge(+1267): (0x755d7a68) (0x755d7af0): a[0] [OR] b[1] -> a[1] May 18 10:52:40 Debug:[sql] dbmysql.c,db_query(+290): query [SELECT seen_flag, answered_flag, deleted_flag, flagged_flag, draft_flag, recent_flag, DATE_FORMAT( .Out of memory (Needed 57148 bytes) May 18 10:52:40 Debug:[imap] imapcommands.c,_ic_store(+1636): unable to retrieve msginfo (process:25576): GLib-CRITICAL **: g_tree_lookup: assertion `tree != NULL' failed May 18 10:52:40 Warning:[imap] imapcommands.c,_do_store(+1464): unable to lookup msginfo struct for [322366] May 18 10:52:40 Debug:[imapsession] dbmail-imapsession.c,dbmail_imap_session_printf(+1424): RESPONSE: [15108 OK UID STORE completed ] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+305): Finished command uid [0] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+175): COMMAND: [15109 UID FETCH 322366 FLAGS] .... May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+288): Executing command uid... May 18 10:52:40 Debug:[mailbox] dbmail-mailbox.c,dbmail_mailbox_get_set(+1330): [322369] uid [1] May 18 10:52:40 Debug:[misc] misc.c,g_tree_merge(+1267): (0x83854e0) (0x8465648): a[0] [OR] b[1] -> a[1] May 18 10:52:40 Debug:[sql] dbmysql.c,db_query(+290): query [SELECT seen_flag, answered_flag, deleted_flag, flagged_flag, draft_flag, recent_flag, DATE_FORMAT( .Out of memory (Needed 48984 bytes) May 18 10:52:40 Debug:[imap] imapcommands.c,_ic_store(+1636): unable to retrieve msginfo (process:25576): GLib-CRITICAL **: g_tree_lookup: assertion `tree != NULL' failed May 18 10:52:40 Warning:[imap] imapcommands.c,_do_store(+1464): unable to lookup msginfo struct for [322369] May 18 10:52:40 Debug:[imapsession] dbmail-imapsession.c,dbmail_imap_session_printf(+1424): RESPONSE: [15110 OK UID STORE completed ] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+305): Finished command uid [0] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+175): COMMAND: [15111 UID FETCH 322369 FLAGS] .... May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+288): Executing command uid... May 18 10:52:40 Debug:[mailbox] dbmail-mailbox.c,dbmail_mailbox_get_set(+1330): [322372] uid [1] May 18 10:52:40 Debug:[misc] misc.c,g_tree_merge(+1267): (0x8106af0) (0x8106a68): a[0] [OR] b[1] -> a[1] May 18 10:52:40 Debug:[sql] dbmysql.c,db_query(+290): query [SELECT seen_flag, answered_flag, deleted_flag, flagged_flag, draft_flag, recent_flag, DATE_FORMAT( .Out of memory (Needed 40820 bytes) May 18 10:52:40 Debug:[imap] imapcommands.c,_ic_store(+1636): unable to retrieve msginfo (process:25576): GLib-CRITICAL **: g_tree_lookup: assertion `tree != NULL' failed May 18 10:52:40 Warning:[imap] imapcommands.c,_do_store(+1464): unable to lookup msginfo struct for [322372] May 18 10:52:40 Debug:[imapsession] dbmail-imapsession.c,dbmail_imap_session_printf(+1424): RESPONSE: [15112 OK UID STORE completed ] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+305): Finished command uid [0] .... May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+288): Executing command uid... May 18 10:52:40 Debug:[mailbox] dbmail-mailbox.c,dbmail_mailbox_get_set(+1330): [322375] uid [1] May 18 10:52:40 Debug:[misc] misc.c,g_tree_merge(+1267): (0x755c7378) (0x755c72f0): a[0] [OR] b[1] -> a[1] May 18 10:52:40 Debug:[sql] dbmysql.c,db_query(+290): query [SELECT seen_flag, answered_flag, deleted_flag, flagged_flag, draft_flag, recent_flag, DATE_FORMAT( .Out of memory (Needed 24492 bytes) May 18 10:52:40 Debug:[imap] imapcommands.c,_ic_store(+1636): unable to retrieve msginfo (process:25576): GLib-CRITICAL **: g_tree_lookup: assertion `tree != NULL' failed May 18 10:52:40 Warning:[imap] imapcommands.c,_do_store(+1464): unable to lookup msginfo struct for [322375] May 18 10:52:40 Debug:[imapsession] dbmail-imapsession.c,dbmail_imap_session_printf(+1424): RESPONSE: [15114 OK UID STORE completed ] May 18 10:52:40 Info:[imap] imap4.c,IMAPClientHandler(+305): Finished command uid [0] .... May 18 10:52:40 Message:[server] pool.c,scoreboard_state(+584): Scoreboard state: children [1/2], spares [0 (1 - 1)] May 18 10:52:40 Message:[server] pool.c,child_register(+293): register child [26607] May 18 10:52:40 Info:[server] pool.c,child_register(+318): initializing child_state [26607] using slot [0] May 18 10:52:40 Info:[serverchild] serverchild.c,CreateChild(+194): signal handler placed, going to perform task now May 18 10:52:40 Debug:[db] dbmodule.c,db_load_driver(+56): library_directory is [/usr/local/lib/dbmail] _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://twister.fastxs.net/mailman/listinfo/dbmail-dev
