A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=1024 ====================================================================== Reported By: rmoesbergen Assigned To: ====================================================================== Project: DBMail Issue ID: 1024 Category: IMAP daemon Reproducibility: always Severity: major Priority: normal Status: new target: ====================================================================== Date Submitted: 09-Sep-13 10:15 CEST Last Modified: 25-Sep-13 14:22 CEST ====================================================================== Summary: Imap daemon leaks extensive amounts of memory Description: During load tests we discovered that the dbmail-imap daemon leaks memory eventually causing the machine(s) that it runs on to OOM and start killing processes. Memory usage seems dependant on the amount of messages in the database, but only ever increases when more messages are sent and received, even though the messages are deleted after retrieval.
This bug is probably masked in a default installation due to the fact that the default logrotate script restarts all dbmail daemons every day, resetting memory usage. We've modified this script because this restart would cause unwanted downtime (we have very strict SLA's with our customers) ====================================================================== ---------------------------------------------------------------------- (0003576) paul (administrator) - 09-Sep-13 17:07 http://www.dbmail.org/mantis/view.php?id=1024#c3576 ---------------------------------------------------------------------- I've passed along the valgrind results to the libzdb author, and he urges you to upgrade libzdb: """ Please use the last version of libzdb (2.12). I see you use libzdb.so.8.0.3 in the log you sent and there has indeed been memory problems related to MySQL, but those should have been fixed back in 2.11.2 """ ---------------------------------------------------------------------- (0003577) rmoesbergen (reporter) - 10-Sep-13 13:56 http://www.dbmail.org/mantis/view.php?id=1024#c3577 ---------------------------------------------------------------------- I've built a new libzdb package (2.12), rebuilt dbmail against it and re-tested with valgrind. The zdb leaks are indeed gone, see attached vg-2.log. However, the unlimited growth of the imapd process is still there, unfortunately. ---------------------------------------------------------------------- (0003578) rmoesbergen (reporter) - 20-Sep-13 15:26 http://www.dbmail.org/mantis/view.php?id=1024#c3578 ---------------------------------------------------------------------- Should I try the 'master' branch to see if this is fixed there? I see some memory related commits that aren't in 3_1..? ---------------------------------------------------------------------- (0003579) paul (administrator) - 21-Sep-13 15:47 http://www.dbmail.org/mantis/view.php?id=1024#c3579 ---------------------------------------------------------------------- No point in testing master. My assumption is that there is no real leakage, but some severe memory fragmentation. Real leakage would have been exposed by valgrind. You might try: 1) DM_POOL=yes dbmail-imapd to see of using the memory pools helps. However, I expect it will decrease fragmentation, but will actually increase the memory footprint if you serve many concurrent clients since each client will get it's own memory arena. 2) build with jemalloc support. Maybe that will reduce fragmentation. 3) try a more recent kernel and glibc version if possible since I've seen drastic improvements on linux when I moved from a 2.6 kernel to a 3.2 kernel. 4) setup a load-balancing proxy in front of two or more imapd instances. This will allow you to recycle imapd when it becomes too big without suffering downtime. Nginx and ha-proxy are two possible solutions here. ---------------------------------------------------------------------- (0003588) rmoesbergen (reporter) - 23-Sep-13 17:26 http://www.dbmail.org/mantis/view.php?id=1024#c3588 ---------------------------------------------------------------------- 1) Tested. With this setting on, memory increases even faster, but only increases, mem usage still never goes down. 2) Tested, no difference with 'standard' dbmail. 3) I'm already on 3.2.0 (ubuntu 12.04) Are you sure it's fragmentation? If find that kind of hard to believe... We're talking hundreds om megabytes of memory in a few minutes here... Wouldn't there be much more software with this problem if that was the issue? I suspect something is being cached/referenced and kept in memory when it shouldn't. I guess the only option I have left now is option 4, restart the processes (we already do load balancing over multiple servers). ---------------------------------------------------------------------- (0003590) paul (administrator) - 23-Sep-13 17:39 http://www.dbmail.org/mantis/view.php?id=1024#c3590 ---------------------------------------------------------------------- I'm not sure of anything without a proper valgrind proof of leakage. Hundreds of mbs of leakage is hard to believe, though I will take your word for it. No-one else has reported anything like it though, despite extensive load testing. Perhaps you can generate the exact IMAP commands used, and give me an idea of the size and complexity of the typical test message used. Are the imapd connections using IDLE? Small leaks can quickly explode if they occur often enough, so maybe if I'll be able to replay your imap commands I can find something.... anything. ---------------------------------------------------------------------- (0003597) rmoesbergen (reporter) - 24-Sep-13 10:14 http://www.dbmail.org/mantis/view.php?id=1024#c3597 ---------------------------------------------------------------------- Ok, here's a transcript of a typical session. The messages used are small, around 400 - 500 bytes, but there's a lot of messages... (6250 per run). The transcript below is for 1 message. Possible "issue" is that the software uses long-running IMAP connections that repeat NOOP and SEARCH commands in a loop. However: if there are no messages sent/received, no memory increase occurs, only during actual message retrieval. Disconnecting all IMAP sessions after doing a testrun does not lead to imapd freeing memory. A001 SELECT "INBOX" A001 OK [READ-WRITE] SELECT completed[CR][LF] A002 NOOP A002 OK NOOP completed[CR][LF] .... every 5 seconds .... After 25 NOOP's with no mailbox status info: A105 STATUS INBOX (MESSAGES) * STATUS "INBOX" (MESSAGES 0)[CR][LF] A105 OK STATUS completed[CR][LF] .... A message arrives A152 NOOP * 2 EXISTS[CR][LF] * 2 RECENT[CR][LF] A152 OK NOOP completed[CR][LF] A153 SEARCH NOT Deleted * SEARCH 1 2[CR][LF] A153 OK SEARCH completed[CR][LF] A154 FETCH 1 (FLAGS BODY[HEADER]) * 1 FETCH (FLAGS (\Recent) BODY[HEADER] {483}[CR][LF] Return-Path: <m...@intercommit.nl>[CR][LF] Delivery-date: Tue, 24 Sep 2013 08:18:33 +0100[CR][LF] Received: id 1VOMtF-0005fa-Az for <>; Tue, 24 Sep 2013 08:18:33 +0100[CR][LF] From: <m...@intercommit.nl>[CR][LF]> To: <m...@intercommit.nl>[CR][LF]> Subject: Bericht 4038122[CR][LF]> Message-ID: 138000737100...@edicon.go-secure.nl[CR][LF]> X-IcitMsgID: ThuZeqwTQj2RFo4YfEZjTg[CR][LF]> X-mtype: [CR][LF]> X-Mailer: InterCommIT Edicon+ - 1.8.30 GNU/Linux-AMD64 DX[CR][LF]> MIME-Version: 1.0[CR][LF]> Content-Type: multipart/mixed;[CR][LF]> boundary="=_7yIZwROu3P2Gg1dbRjkU"[CR][LF]> [CR][LF] * 1 FETCH (FLAGS (\Seen \Recent))[CR][LF] A154 OK FETCH completed[CR][LF] A155 FETCH 1 BODY[1.MIME] * 1 FETCH (BODY[1.MIME] {157}[CR][LF] Content-Type: application/octet-stream[CR][LF] Content-Transfer-Encoding: base64;conversions=gzip[CR][LF] Content-Disposition: attachment;[CR][LF] filename="msg4038122.dat"[CR][LF] [CR][LF] A155 OK FETCH completed[CR][LF] A156 FETCH 1 (BODY[1]) * 1 FETCH (BODY[1] {428}[CR][LF] .... Message body is received (428 bytes, base64 encoded) A156 OK FETCH completed[CR][LF] A157 FETCH 1 BODY[2.MIME] * 1 FETCH (BODY[2.MIME] NIL)[CR][LF] A157 OK FETCH completed[CR][LF] .... No more attachments ... A158 NOOP * 2 RECENT[CR][LF] A158 OK NOOP completed[CR][LF] A159 STORE 1 +FLAGS.SILENT (\Deleted) A159 OK STORE completed[CR][LF] .... Same sequence for message nr 2 ... .... No more messages found, so expunge: A164 EXPUNGE * 2 EXPUNGE[CR][LF] * 1 EXPUNGE[CR][LF] A164 OK EXPUNGE completed[CR][LF] .... Then it returns to polling every 5 seconds with the following loop: A702 NOOP A702 OK NOOP completed[CR][LF] A703 SEARCH NOT Deleted * SEARCH[CR][LF] A703 OK SEARCH completed[CR][LF] A704 EXPUNGE A704 OK EXPUNGE completed[CR][LF] Done. ---------------------------------------------------------------------- (0003598) paul (administrator) - 25-Sep-13 14:22 http://www.dbmail.org/mantis/view.php?id=1024#c3598 ---------------------------------------------------------------------- I've added a test-script that emulates your behaviour. It did uncover a leak that had been fixed in the master branch already. Please test bfd0ac138a4780da8f5683c2aa7194b001510e0f Issue History Date Modified Username Field Change ====================================================================== 09-Sep-13 10:15 rmoesbergen New Issue 09-Sep-13 10:15 rmoesbergen File Added: vg.log 09-Sep-13 17:07 paul Note Added: 0003576 10-Sep-13 13:54 rmoesbergen File Added: vg-2.log 10-Sep-13 13:56 rmoesbergen Note Added: 0003577 20-Sep-13 15:26 rmoesbergen Note Added: 0003578 21-Sep-13 15:47 paul Note Added: 0003579 23-Sep-13 17:26 rmoesbergen Note Added: 0003588 23-Sep-13 17:39 paul Note Added: 0003590 24-Sep-13 10:14 rmoesbergen Note Added: 0003597 25-Sep-13 14:22 paul Note Added: 0003598 ====================================================================== _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
