A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=1024 ====================================================================== Reported By: rmoesbergen Assigned To: ====================================================================== Project: DBMail Issue ID: 1024 Category: IMAP daemon Reproducibility: always Severity: major Priority: normal Status: new target: ====================================================================== Date Submitted: 09-Sep-13 10:15 CEST Last Modified: 21-Sep-13 15:47 CEST ====================================================================== Summary: Imap daemon leaks extensive amounts of memory Description: During load tests we discovered that the dbmail-imap daemon leaks memory eventually causing the machine(s) that it runs on to OOM and start killing processes. Memory usage seems dependant on the amount of messages in the database, but only ever increases when more messages are sent and received, even though the messages are deleted after retrieval.
This bug is probably masked in a default installation due to the fact that the default logrotate script restarts all dbmail daemons every day, resetting memory usage. We've modified this script because this restart would cause unwanted downtime (we have very strict SLA's with our customers) ====================================================================== ---------------------------------------------------------------------- (0003576) paul (administrator) - 09-Sep-13 17:07 http://www.dbmail.org/mantis/view.php?id=1024#c3576 ---------------------------------------------------------------------- I've passed along the valgrind results to the libzdb author, and he urges you to upgrade libzdb: """ Please use the last version of libzdb (2.12). I see you use libzdb.so.8.0.3 in the log you sent and there has indeed been memory problems related to MySQL, but those should have been fixed back in 2.11.2 """ ---------------------------------------------------------------------- (0003577) rmoesbergen (reporter) - 10-Sep-13 13:56 http://www.dbmail.org/mantis/view.php?id=1024#c3577 ---------------------------------------------------------------------- I've built a new libzdb package (2.12), rebuilt dbmail against it and re-tested with valgrind. The zdb leaks are indeed gone, see attached vg-2.log. However, the unlimited growth of the imapd process is still there, unfortunately. ---------------------------------------------------------------------- (0003578) rmoesbergen (reporter) - 20-Sep-13 15:26 http://www.dbmail.org/mantis/view.php?id=1024#c3578 ---------------------------------------------------------------------- Should I try the 'master' branch to see if this is fixed there? I see some memory related commits that aren't in 3_1..? ---------------------------------------------------------------------- (0003579) paul (administrator) - 21-Sep-13 15:47 http://www.dbmail.org/mantis/view.php?id=1024#c3579 ---------------------------------------------------------------------- No point in testing master. My assumption is that there is no real leakage, but some severe memory fragmentation. Real leakage would have been exposed by valgrind. You might try: 1) DM_POOL=yes dbmail-imapd to see of using the memory pools helps. However, I expect it will decrease fragmentation, but will actually increase the memory footprint if you serve many concurrent clients since each client will get it's own memory arena. 2) build with jemalloc support. Maybe that will reduce fragmentation. 3) try a more recent kernel and glibc version if possible since I've seen drastic improvements on linux when I moved from a 2.6 kernel to a 3.2 kernel. 4) setup a load-balancing proxy in front of two or more imapd instances. This will allow you to recycle imapd when it becomes too big without suffering downtime. Nginx and ha-proxy are two possible solutions here. Issue History Date Modified Username Field Change ====================================================================== 09-Sep-13 10:15 rmoesbergen New Issue 09-Sep-13 10:15 rmoesbergen File Added: vg.log 09-Sep-13 17:07 paul Note Added: 0003576 10-Sep-13 13:54 rmoesbergen File Added: vg-2.log 10-Sep-13 13:56 rmoesbergen Note Added: 0003577 20-Sep-13 15:26 rmoesbergen Note Added: 0003578 21-Sep-13 15:47 paul Note Added: 0003579 ====================================================================== _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
