Hi Michael and Gilles, Sorry for the delay in responding.
> >> I've been trying to sync a larger IMAP mailbox (~6Gb including all > >> folders) and imapsync constantly gets terminated by the kernel > >> OOM-killer because it takes up all available memory (1024 mb). Michael, do you know what the size is of largest message that is being sync'd? I saw you mentioned maybe 60MB so I can try that out as a test case I guess. > > This maybe a well known problem coming directly from > > libmail-imapclient-perl. The problem doesn't come with large > > mailbox but with large messages. imapsync uses memory nearly 17 > > times the largest message size. There is also an old memory issue > > on freebsd systems. Mail::IMAPClient has some major inefficiencies and I suppose imapsync ends up doubling those because of the "from" and "to" objects each creating copies of the messages (and because methods used keep data in memory instead of moving them to files... it's a trade-off for sure). I'll try to look a bit more at this problem over the next few weeks. I've had a few complaints about memory usage. I am not convinced we have a leak but we definitely waste resources. I'll have to do some leg work to find something that helps expose the root of the problem(s) if there are any outside of the obvious inefficiencies. > Or can imapsync workaround this limitation in Mail:IMAPClient > somehow? Gilles, you said... I try but I partially succeeded. I went to decrease 17 to 4 times but I got errors and I gave up. I need to read carefully the IMAP RFC before going on again. Any chance you want to share some details on changes you made and/or attempted? Or were they simply changing of which methods you used and not changes working around Mail::IMAPClient behavior? I know you've chosen to not share code anymore, but I hope you will reciprocate if/when appropriate. In theory append_file() should be more (memory) efficient than append()/append_string() on large messages. Likewise for message_string() vs message_to_file(). Last I checked, imapsync still uses append_string() which causes the entire message contents to get stored in memory (several times over perhaps). It might be worth while to use append_string() for messages up to size X and then switch to append_file()/message_file() when the messages are > size X if those cases can easily be identified by imapsync. Otherwise, imapsync will always be straining system memory for large files. For everyone's reference, the bug filed on CPAN for Mail::IMAPClient is: Bug #61292: Memory consumption with message_string() and append() https://rt.cpan.org/Public/Bug/Display.html?id=61292 Phil -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org