Re: [Dovecot] Dovecot 1.1.11 failing to accept SSL/TLS connections
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, 19 Mar 2009, Timo Sirainen wrote: It's the dovecot process that has trouble with the fd limits, because it needs one fd for each child process (for logging). Dunno if it is related or helpful, I use v1.10 still. When I had login_max_processes_count too low, once in a while the dovecot process had lots of socket fd's (displayed with lsof in Linux), for which there was no counterpart in the system (lsof -U). The number of fd's did not decrease, when the number of logged in users decreased (aka over night) - I made a graph for monitoring fd's and users count. Doing a reload solved the problem just temporarily, we have peeks of user logins at specific parts of the year, so the problem came up every 4 or 5 days in these phases, but not in months otherwise. Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBScNIx3WSIuGy1ktrAQLy5wgApxdglt2/d6CBxwiIiKN+DXCJPPrK1HqC 1ELyWAg2eR6jb9xo1DprJWZgs/7fFMQZvcg4vBMPpsr9AZl3O22tTaDLNz9MglN+ vYIh2IXFqNzOPY9NXSM3v5S7Y7WqwRALfmMXcbQ1UGLn9CdnsPDvAaqEY26dWSee Fyjvgnxl035o2glvIJ5ppDwjOSnPL2hEYL34CprjzglLjzNuyh2l/oK/xhOadhwa Dswnr30uKZYpIVcVFHVom+y2P1X66noitaKZsTjAMgmJtKrHsFRahP6EQ+0qYGOf KTHyWBmStzxZazTTWrMuWdJwxn/B+W4i4x6nTK+5y+/wGx1/kcjW5A== =NUIC -END PGP SIGNATURE-
[Dovecot] update of current quota usage
Hello, I use Dovecot 1.1.11 and store the current quota usage in a Database. As far I can see it, the quota usage is updated each time dovecot performs an action on a mail. For example if a new mail is stored in the inbox, the current quota usage is increased. If a mail is deleted the quota usage is decreased. If there are still existing E-Mails in the inbox, before quota is turned on or if I perfom changes on the mail storage myself (for example purging trash or spam, or something) the current quota usage is NOT updated automatically by dovecot. For example I copy some mailboxes to my server, then the current quota for this mailboxes is still zero. Is it correct that dovecot can't do this for me? If yes, it seems I have to write my own (simple) update script and run it sometimes croned (because I don't really trust the actual quota implementation) and/or I have to update the current quota usage each time I perform actions myself. This workaround is easy by using a DB storage, but what about other implementations? Wouldn't it be nice if dovecot could do this or provide a tool for it? Thank you! Philipp
[Dovecot] Maildir and mail files question
If an incoming mail is saved in 2 mail locations via dovecot deliver, and its wished that if deleted from box a should also be deleted from box b ... wil it be safe to delete based upon the W=x part of the file name So far my tests indicate that It could be true ... Thanks Harry
Re: [Dovecot] Maildir and mail files question
Harry Lachanas escreveu: If an incoming mail is saved in 2 mail locations via dovecot deliver, and its wished that if deleted from box a should also be deleted from box b ... wil it be safe to delete based upon the W=x part of the file name So far my tests indicate that It could be true ... W= represents the size of the message. While it is relatively unlikely that two messages will have the exact same size, it certainly can happen. -- Eduardo M Kalinowski edua...@kalinowski.com.br
[Dovecot] ESORT broken in Dovecot 1.2
Looks like ESORT is broken (just compiled 1.2 from hg 10 minutes ago). Here is the command I'm using and the corresponding error message: C: 5 UID SORT RETURN (ALL COUNT) (ARRIVAL) US-ASCII UNSEEN S: 5 BAD Error in IMAP command UID: Invalid sort argument. The corresponding UID SORT command (without the RETURN syntax) works fine: 5 UID SORT (ARRIVAL) US-ASCII UNSEEN * SORT 5 OK Sort completed (0.000 secs). michael
Re: [Dovecot] Maildir and mail files question
On Fri, 2009-03-20 at 13:23 +0200, Harry Lachanas wrote: If an incoming mail is saved in 2 mail locations via dovecot deliver, and its wished that if deleted from box a should also be deleted from box b ... wil it be safe to delete based upon the W=x part of the file name Use Message-ID: header primarily and perhaps also include W=xxx part. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] update of current quota usage
On Fri, 2009-03-20 at 11:43 +0100, fl...@pbartels.info wrote: For example I copy some mailboxes to my server, then the current quota for this mailboxes is still zero. Is it correct that dovecot can't do this for me? Yes. It's too much work for Dovecot to try to figure out all the different things you might be doing under it. You can anyway easily make Dovecot recalculate the quota: delete the user's quota usage row from the database. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] update of current quota usage
Timo Sirainen t...@iki.fi wrote: On Fri, 2009-03-20 at 11:43 +0100, fl...@pbartels.info wrote: For example I copy some mailboxes to my server, then the current quota for this mailboxes is still zero. Is it correct that dovecot can't do this for me? Yes. It's too much work for Dovecot to try to figure out all the different things you might be doing under it. You can anyway easily make Dovecot recalculate the quota: delete the user's quota usage row from the database. Well, thats a very simple solution. Also for my example. If I copy some mailboxes to the server I just remove the quota usage row and dovecot recalculates it. I can do this too, if I remove some spam messages or purge the trash... What did you mean with too much work? (It's able to do it and you said yes it can't but it can as you said later.) Thank you, Philipp
Re: [Dovecot] Panic: Trying to allocate 2147483648 bytes
Timo Sirainen wrote: On Thu, 2009-03-19 at 15:25 -0400, David Halik wrote: So far everything is working smoothly, but when someone does a search through directory with a large number of emails, dovecot dies and prints the following message: [ID 107833 mail.crit] Panic: Trying to allocate 2147483648 bytes So you can easily reproduce this? Everytime and on different systems with a similar setup. Since I can't do debugging on the production server, I'm using a server with a similar setup. The only difference is probably physical memory and swap size. I'm now getting a different size in the panic error, but you get the idea: Mar 20 13:32:10 er0.rutgers.edu IMAP(dhalik): : [ID 107833 mail.crit] Panic: Trying to allocate 2147483648 bytes Also, I've been keeping an eye on top while this was running and it seems that Dovecot is growing at an amazing rate, which is why it runs out of room I guess: 8081 dhalik 1 580 1030M 871M run 0:33 53.29% imap This was while doing a full Pine text search for a word in a folder with about 1300 emails. Shortly after it died. Keep in mind that I am only able to reproduce this problem with the homedir on NFS and INDEX=MEMORY on. The same setup on local disk had no issues. I grabbed a backtrace if it is helpful: Yes, very helpful! Is it possible to also examine variable values? I'd love to, but it seems that is all the I can get out of Sun Studio unless you know some tricks. I verified that I am building with full debug symbols (-g -xs) and that there is no optimization. Could you try applying this patch: http://hg.dovecot.org/dovecot-1.1/rev/98a59fd61d7f and also the attached patch. What error does it fail in? I applied both patches, rebuilt dovecot 1.1.12 with full debug, and I'm not seeing anything different. Was there supposed to be a new error? It still exhibits the same behavior and eventually core dumps when it runs out of room, same as before. Here's the most recent traceback: http://pastebin.com/f4deddc23 I appreciate the help Timo. -- David Halik System Administrator OIT-CSS Rutgers University dha...@jla.rutgers.edu
Re: [Dovecot] update of current quota usage
Timo Sirainen t...@iki.fi wrote: On Fri, 2009-03-20 at 18:47 +0100, fl...@pbartels.info wrote: Yes. It's too much work for Dovecot to try to figure out all the different things you might be doing under it. You can anyway easily make Dovecot recalculate the quota: delete the user's quota usage row from the database. Well, thats a very simple solution. Also for my example. If I copy some mailboxes to the server I just remove the quota usage row and dovecot recalculates it. I can do this too, if I remove some spam messages or purge the trash... If you removed spam messages and purged trash via IMAP you wouldn't have to worry about updating quota. I mean if you do this directly on the server using a croned script. What did you mean with too much work? I mean that currently quota is updated only while changes are actually done. To be able to figure out when mailbox changes were done outside Dovecot it would need to keep some kind of state which messages/mailboxes were added to quota and which weren't. Index files help somewhat in implementing that, but it's still a lot of extra code. Yes as I see it quota usage is only done if something is done via IMAP, or perhaps if you fetch mail via POP?! (not tested). And then it is not recalculated, it's just updated like if you remove a file the quota is decreased of filesize, etc. That's ok if you're only using IMAP and if you can trust that all works fine. But there is also the behavior that dovecot calculates the usage if its not yet set and that is useful. I think the best solution for me is just to remove the row after I made some changes in the mailbox and then dovecot will recalc. the quota usage at the next action, like reciving a mail... So you just have to know that you have to remove the quota usage informations after you did something in the mailbox and then dovecot recalculates it for you. (I thought I must write a script that summarizes the usage and update the database...) Thank you
Re: [Dovecot] Panic: Trying to allocate 2147483648 bytes
On Fri, 2009-03-20 at 13:57 -0400, David Halik wrote: Also, I've been keeping an eye on top while this was running and it seems that Dovecot is growing at an amazing rate, which is why it runs out of room I guess: 8081 dhalik 1 580 1030M 871M run 0:33 53.29% imap OK, so it isn't really the kind of bug I first thought it would be. Instead it's allocating/leaking way too much memory. This was while doing a full Pine text search for a word in a folder with about 1300 emails. Shortly after it died. Keep in mind that I am only able to reproduce this problem with the homedir on NFS and INDEX=MEMORY on. The same setup on local disk had no issues. Can you test what exactly is the difference? INDEX=MEMORY or being in NFS? I don't see how either of those would make a difference here. To me it's currently looking like Solaris iconv() is returning some weird results in some condition. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] update of current quota usage
On Fri, 2009-03-20 at 19:10 +0100, fl...@pbartels.info wrote: If you removed spam messages and purged trash via IMAP you wouldn't have to worry about updating quota. I mean if you do this directly on the server using a croned script. Yes, but I meant that you could rewrite your script in a way that it uses IMAP to do the actualy work, instead of modifying the filesystem directly. For example with v1.2 you could do it easily: printf 1 select Trash 2 search return (save) before 01-jan-2009 3 store $ +flags.silent \deleted 4 expunge 5 logout | dovecot --exec-mail imap v1.1 doesn't have the return (save) and $ feature though, so it would need a more interactive script. What did you mean with too much work? I mean that currently quota is updated only while changes are actually done. To be able to figure out when mailbox changes were done outside Dovecot it would need to keep some kind of state which messages/mailboxes were added to quota and which weren't. Index files help somewhat in implementing that, but it's still a lot of extra code. Yes as I see it quota usage is only done if something is done via IMAP, or perhaps if you fetch mail via POP?! (not tested). Quota is updated whenever Dovecot is the one doing changes and quota plugin is loaded. So assuming you've added mail_plugins=quota inside pop3 section, Dovecot pop3 updates the quota. And then it is not recalculated, it's just updated like if you remove a file the quota is decreased of filesize, etc. Yes. Recalculating quota all the time would be way too much unnecessary work. signature.asc Description: This is a digitally signed message part
[Dovecot] Single Instance Storage still on the roadmap?
Hey Timo, I was curious where Single Instance Storage fits in on the roadmap, now that you are maybe settled in in your new digs... Is it still coming soon? Has it been pushed back by your new employers priorities? Its real high on mine, so if has been back-burnered, I'll need to start looking for alternatives... Thanks, -- Best regards, Charles
Re: [Dovecot] Single Instance Storage still on the roadmap?
On Fri, 2009-03-20 at 15:13 -0400, Charles Marcus wrote: I was curious where Single Instance Storage fits in on the roadmap, now that you are maybe settled in in your new digs... What's your use case? People mailing 10 MB pdfs to every...@company.com, or just reducing disk usage because random people receive the same attachment at different times? Is it still coming soon? Has it been pushed back by your new employers priorities? Multi-msgs-per-file dbox is getting close to ready. Adding single instance storage to it probably wouldn't be a huge job, but I don't think it's a very high priority. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Single Instance Storage still on the roadmap?
On 3/20/2009 3:24 PM, Timo Sirainen wrote: I was curious where Single Instance Storage fits in on the roadmap, now that you are maybe settled in in your new digs... What's your use case? People mailing 10 MB pdfs to every...@company.com, or just reducing disk usage because random people receive the same attachment at different times? Both. We get emails from vendors that contain large (sometimes 30 or 40MB) PDFs, and they send them to 5 or 50 of our employees. The employees then will email them back and forth, or send them to different people outside the company, sometimes many times. I spent a few hours doing a random audit of someones mail store (with them sitting there with me), and found cases where the same 40MB attachment existed 20 or more times, most of them in the Sent folder, but often with copies in different client folders. Our users are dumb... Is it still coming soon? Has it been pushed back by your new employers priorities? Multi-msgs-per-file dbox is getting close to ready. Adding single instance storage to it probably wouldn't be a huge job, but I don't think it's a very high priority. Ok, thanks for the update, even though its not what I wanted to hear... :)
Re: [Dovecot] Panic: Trying to allocate 2147483648 bytes
On Fri, 2009-03-20 at 15:04 -0400, David Halik wrote: But just to be sure I moved my homedir to local disk temporarily, and you are totally right. The problem happened again! So now I'm stumped. It looks like it is neither NFS or INDEX, but an issue with Solaris as you suggested. It just seems odd that I can't reproduce it on one of the servers, yet the other two do it every single time. As far as I know the servers have the same libraries as well: Maybe you could try copying the libraries and using LD_LIBRARY_PATH (or whatever it is in Solaris) to force using those libraries instead of the default ones? You could also try what it logs with this patch when crashing: diff -r 5deb3ee1d655 src/lib-charset/charset-iconv.c --- a/src/lib-charset/charset-iconv.c Fri Mar 20 18:23:12 2009 -0400 +++ b/src/lib-charset/charset-iconv.c Fri Mar 20 18:25:37 2009 -0400 @@ -145,6 +145,7 @@ /* force buffer to grow */ used = dest-used; size = buffer_get_size(dest) - used + 1; + i_warning(charset grows: used=%u, add=%u, (int)dest-used, (int)size); (void)buffer_append_space_unsafe(dest, size); buffer_set_used_size(dest, used); } signature.asc Description: This is a digitally signed message part
