[Dovecot] index synchronization during local delivery
Hi there, Now our system will take dovecot as the whole imap back end, and keep the LDA of ours, But our system has our own UID generation mechanism (in dovecot it's incremental from 1), which will be done while local delivery, So when local deliveries, it will first generate the UID, move the mail into maildir, then synchronize the mailbox index, which is part of dovecot LDA does. The question is what API or something else about index synchronization I can integrate into our own LDA program easily? Seems mailbox_copy() would be the possibility? Thanks.
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
Quoting Timo Sirainen : On 14.2.2012, at 5.19, Michael M Slusarz wrote: b login (SAVEDSTATE 1234567890) user pass I guess the drawback for this approach is that you are explicitly breaking the LOGIN definition. No breaking, extending :) Heh. I should know better after reading about 1000 RFCs last year... Anyway, to play devil's advocate, another drawback of this approach is that in the (fairly remote) chance that the LOGIN command is extended in the future by an RFC, there is no guarantee that this syntax would be used. michael
Re: [Dovecot] 2.1.rc5 Cannot delete folders from trash
On 2/12/2012 9:00 AM, Timo Sirainen wrote: On 12.2.2012, at 15.27, Mike Constabel wrote: Another Question to this Option: # tb-extra-mailbox-sep: # With mbox storage a mailbox can contain either mails or submailboxes, # but not both. Thunderbird separates these two by forcing server to # accept '/' suffix in mailbox names in subscriptions list. I thought this is not needed because I use mdbox. But mbox here means also mdbox and sdbox? Yes. I updated the comment: http://hg.dovecot.org/dovecot-2.1/rev/1a0ab868957b Should I use all 3 workarounds if I don't know which clients are in use? I don't know if delay-newmail is still relevant. I'm fighting the same issue. With the current release (2.1rc6) no errors are logged - but I'm having problems deleted folders from trash. I've enabled tb-extra-mailbox-sep. Two folders that were previously in Trash were successfully removed. I deleted four more - after several Thunderbird restarts, compacts, refreshes, I have killed three of them - there is still one folder that won't "die". Thunderbird reports the mailbox doesn't exist, yet it continues to appear. -- Daniel
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
Quoting Timo Sirainen : On 13.2.2012, at 23.32, Timo Sirainen wrote: Perhaps a way for (trusted) clients to be able to do this? :) a logout save * OK [SAVEDSTATE 1234567890] * BYE logged out a OK Actually, this could probably be safely implemented by sending all of the state to the client as a string: * OK [SAVEDSTATE base64-encoded-state] There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works). Given that it is not *that* expensive to re-create the state, I don't think the ability to recreate state from several days ago would be worth the effort of storing in the log/index files. For a disconnected client (e.g. webmail), there's going to necessarily need to be startup costs of initializing the session anyway so re-creating the state can be buried in this other work. Re-creating becomes more important when users are doing actions they expect immediate (or at least very fast) responses for. Things like listing messages in a mailbox, viewing a message, or polling mailboxes. Saving 50ms per request becomes important in these situations since re-creating state now takes 10% of the total request time. So I don't think states don't need a terribly long lifetime. I almost think of something like an internal Dovecot IDLE queue - after a logout is received, state is stored for 30 minutes after which it is discarded. (Although not knowing anything about internal Dovecot state, don't know if this is overly resource intensive.) As for base-64-encoded state: if other IMAP servers ever wanted to implement a similar protocol trying to coordinate the data structure would be a nightmare. Keeping it to something like a MODSEQ value would hide the state ID -> data abstraction entirely within dovecot. And would allow you to change your mind in the future if you come up with a better way to represent state. michael
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On 14.2.2012, at 5.19, Michael M Slusarz wrote: >> b login (SAVEDSTATE 1234567890) user pass > > I guess the drawback for this approach is that you are explicitly breaking > the LOGIN definition. No breaking, extending :) > And you don't allow reviving the state if using the AUTHENTICATE command. Just as easy: b authenticate (savedstate ..) plain .. > The alternative would be to have an additional pre-authentication command > that sets the desired state. But that adds the overhead of sending/parsing > another command. Yeah. Although not bad either, since you can pipeline that command + login/authenticate.
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
Quoting Timo Sirainen : On 13.2.2012, at 22.24, Michael M Slusarz wrote: http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html These posts neglect the fact that you don't need to issue a CAPABILITY command if the connection is reused either, so that's an additional advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive user of the imapproxy instance for this feature to work correctly. Somewhat topical, since Timo was just mentioning support for some sort of IMAP state save/restore feature possibly making it into 2.2. Perhaps a way for (trusted) clients to be able to do this? :) a logout save * OK [SAVEDSTATE 1234567890] * BYE logged out a OK ... b login (SAVEDSTATE 1234567890) user pass * OK [RESTOREDSTATE 1234567890] b ok vs. b login (SAVEDSTATE 1234567890) user pass * NO [UNKNOWNSTATE 1234567890] b ok I guess the drawback for this approach is that you are explicitly breaking the LOGIN definition. And you don't allow reviving the state if using the AUTHENTICATE command. The alternative would be to have an additional pre-authentication command that sets the desired state. But that adds the overhead of sending/parsing another command. michael
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
Quoting Jan-Frode Myklebust : BTW: do you also have information on the state of select caching in the up-imapproxy? I got some very negative comments when googling it, and the changelog didn't suggest there had been any improvements since.. I wouldn't trust it. IIRC, it was added years ago and given the syntax changes to the SELECT/EXAMINE call since then (e.g. QRESYNC extentsion), it is doubtful the code has been updated to handle these situations. And in a server-caching IMAP server like dovecot, all that SELECT information is not expensive anyway, so any gains would be miniscule. michael
Re: [Dovecot] Boehm
On 14.2.2012, at 5.04, Daniel L. Miller wrote: > As the Boehm GC is an available option, do I take that to mean it's > beneficial for some users but not most? Eh. Probably not.
[Dovecot] Boehm
As the Boehm GC is an available option, do I take that to mean it's beneficial for some users but not most? -- Daniel
Re: [Dovecot] v2.1.rc6 released
On 2/13/2012 7:00 PM, Timo Sirainen wrote: On 14.2.2012, at 4.54, Daniel L. Miller wrote: With this configure line: ./configure --with-ldap --with-bzlip --with-zlib --with-solr --with-lucene lucene-wrapper.cc:23: fatal error: CLucene.h: No such file or directory Base on this error, I suppose you don't have CLucene installed.. The 2.3.2 version is somewhere around in http://clucene.sourceforge.net/ Ah! I would've thought configure would catch that though - before make did. Ok. -- Daniel
Re: [Dovecot] v2.1.rc6 released
On 14.2.2012, at 4.54, Daniel L. Miller wrote: > With this configure line: > ./configure --with-ldap --with-bzlip --with-zlib --with-solr --with-lucene > > lucene-wrapper.cc:23: fatal error: CLucene.h: No such file or directory Base on this error, I suppose you don't have CLucene installed.. The 2.3.2 version is somewhere around in http://clucene.sourceforge.net/
Re: [Dovecot] v2.1.rc6 released
On 2/12/2012 1:01 PM, Timo Sirainen wrote: http://dovecot.org/releases/2.1/rc/dovecot-2.1.rc6.tar.gz http://dovecot.org/releases/2.1/rc/dovecot-2.1.rc6.tar.gz.sig I've finally finished all of my email backlog. If you haven't received an answer to some question/bugreport, resend the mail. This is hopefully the last v2.1 RC. If I don't receive any (serious) bug reports about this release in next few days, I'll just change the version number to v2.1.0 (and maybe update man pages, some are still missing..) With this configure line: ./configure --with-ldap --with-bzlip --with-zlib --with-solr --with-lucene I get this error on make: make[4]: Entering directory `/opt/dovecot-2.1.rc6/src/plugins/fts-lucene' /bin/bash ../../../libtool --tag=CC --mode=link gcc -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arh -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2-module -avoid-version -o lib20_doveadm_s_lucene_plugin.la -rpath /usr/local/lib/dovecot/doveadm doveadm-fts-lucene.lo -lrt /bin/bash ../../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/lib -I../../../src/lib-mail -I..//../src/lib-index -I../../../src/lib-storage -I../../../src/plugins/fts -I../../../src/doveadm -g -O2 -MT lucene-wrapper.lo -MD -MP -MF eps/lucene-wrapper.Tpo -c -o lucene-wrapper.lo lucene-wrapper.cc libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/lib -I../../../src/lib-mail -I../../../src/lib-index -I../../../src/l-storage -I../../../src/plugins/fts -I../../../src/doveadm -g -O2 -MT lucene-wrapper.lo -MD -MP -MF .deps/lucene-wrapper.Tpo -c lucene-wraer.cc -fPIC -DPIC -o .libs/lucene-wrapper.o lucene-wrapper.cc:23: fatal error: CLucene.h: No such file or directory compilation terminated. make[4]: *** [lucene-wrapper.lo] Error 1 make[4]: *** Waiting for unfinished jobs -- Daniel
Re: [Dovecot] Synchronization error in NFS
On 14.2.2012, at 4.39, Andy YB Hu wrote: > Another question is about the director failover. In > http://blog.dovecot.org/2010/05/new-director-service-in-v20-for-nfs.html > you said: "The main complexity here comes from how to handle proxy server > failures in different situations. Those are less interesting to describe > and I haven't yet implemented all of it, so let's just assume that in > future it all works perfectly." So currently there is no good way to > handle the director failover, or any 3rd party solution like poolmon by > Brad Davidson for health monitoring of backend servers? Your load balancer handles director downtimes so it connects only to directors that are up. Directors themselves figure out when one of them is down (by either explicit disconnection or timeout). That's the theory, and so far no one's told me it doesn't work that way. For the backend servers there's still no automation though. You'll need to explicitly tell director to stop trying to connect some specific backend. The poolmon is intended for that. It would be possible to implement this directly in the director itself, but so far it hasn't really been a priority since the companies who have paid for it have wanted to implement it internally themselves..
Re: [Dovecot] Synchronization error in NFS
Another question is about the director failover. In http://blog.dovecot.org/2010/05/new-director-service-in-v20-for-nfs.html you said: "The main complexity here comes from how to handle proxy server failures in different situations. Those are less interesting to describe and I haven't yet implemented all of it, so let's just assume that in future it all works perfectly." So currently there is no good way to handle the director failover, or any 3rd party solution like poolmon by Brad Davidson for health monitoring of backend servers? Thanks. Timo Sirainen Sent by: To dovecot-bounces@d Dovecot Mailing List ovecot.org cc 02/12/2012 12:06 Subject PMRe: [Dovecot] Synchronization error in NFS Please respond to Dovecot Mailing List On 10.2.2012, at 10.08, Andy YB Hu wrote: > Searched the archive, and got the doveadm -a director-admin for local > doveadm access, and -a host: port for remote doveadm access. You can give -c parameter also to doveadm (and all other Dovecot programs as well). > And you said: http://dovecot.org/list/dovecot/2010-July/050731.html > Now in my director, I have configed the userdb passwd, but the same error > occur: > doveadm(root): Error: User listing returned failure > doveadm(root): Error: user listing failed > > And in the log: Feb 10 07:45:25 auth: Error: Trying to iterate users, but > userdbs don't support it I'm guessing that it's connecting to wrong Dovecot's auth process. Use doveadm -c instead of -a. <><><>
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On 13.2.2012, at 23.32, Timo Sirainen wrote: > Perhaps a way for (trusted) clients to be able to do this? :) > > a logout save > * OK [SAVEDSTATE 1234567890] > * BYE logged out > a OK Actually, this could probably be safely implemented by sending all of the state to the client as a string: * OK [SAVEDSTATE base64-encoded-state] There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works).
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On 13.2.2012, at 22.24, Michael M Slusarz wrote: > http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html > http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html > > These posts neglect the fact that you don't need to issue a CAPABILITY > command if the connection is reused either, so that's an additional > advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive user > of the imapproxy instance for this feature to work correctly. > > Somewhat topical, since Timo was just mentioning support for some sort of > IMAP state save/restore feature possibly making it into 2.2. Perhaps a way for (trusted) clients to be able to do this? :) a logout save * OK [SAVEDSTATE 1234567890] * BYE logged out a OK ... b login (SAVEDSTATE 1234567890) user pass * OK [RESTOREDSTATE 1234567890] b ok vs. b login (SAVEDSTATE 1234567890) user pass * NO [UNKNOWNSTATE 1234567890] b ok
[Dovecot] Released Pigeonhole v0.2.6 for Dovecot v2.0.18
Hello Dovecot users, Pigeonhole v0.2.5 does not compile against the new Dovecot v2.0.18 release. That is why a release a new version now. Other changes are included as well, of which the one regarding the include extension requires special attention; Changelog v0.2.6: * This release fixes unintentional behavior of the include extension. Included script names with a name like "name.sieve" would implicitly map to a script file called "name.sieve" and not "name.sieve.sieve". Keep in mind that the .sieve file extension has no meaning from within the Sieve language. A Sieve script is always stored with an appended .sieve file extension, also when the name already ends with a .sieve suffix. IMPORTANT: Some installations have relied on this unintentional feature, so check your script includes for issues before upgrading. * Matched changes regarding auth_verbose setting in Dovecot. This means that this release will only compile against Dovecot v2.0.18. - Fixed problem in ManageSieve that caused it to omit a WARNINGS response code when the uploaded script compiled with warnings. - Made sure that locations of Sieve error never report `line 0'. - Fixed potential segfault occurring when interpreter initialization fails. The release is available as follows: http://www.rename-it.nl/dovecot/2.0/dovecot-2.0-pigeonhole-0.2.6.tar.gz http://www.rename-it.nl/dovecot/2.0/dovecot-2.0-pigeonhole-0.2.6.tar.gz.sig Refer to http://pigeonhole.dovecot.org and the Dovecot v2.0 wiki for more information. Have fun testing this new release and don't hesitate to notify me when there are any problems. Regards, -- Stephan Bosch step...@rename-it.nl
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On Mon, Feb 13, 2012 at 01:24:25PM -0700, Michael M Slusarz wrote: > > Except you are most likely NOT leveraging the truly interesting part > of imapproxy - the ability to restore the IMAP connection state via > the XPROXYREUSE status response. This is a significant performance > improvement since it also reduces processing load on the client side > (everything before/including authentication needs to be done whether > using imapproxy or not, so there is no client-side savings for these > commands). Thanks for this info, good to know. I'll check with inverse/sogo if this is something they use/intend to use.. > > additional advantage. Note that the XPROXYREUSE-enabled MUA must be > the exclusive user of the imapproxy instance for this feature to > work correctly. Not a problem. Assuming it doesn't also need to be the only imap user of the account/folder. BTW: do you also have information on the state of select caching in the up-imapproxy? I got some very negative comments when googling it, and the changelog didn't suggest there had been any improvements since.. -jf
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On Mon, Feb 13, 2012 at 09:57:31PM +0200, Timo Sirainen wrote: > > > > $ doveconf maildir_very_dirty_syncs > > maildir_very_dirty_syncs = yes > > > > but I don't think this gave the advantage I was expecting.. Was > > expecting this to move most iops to the index-luns, but the maildir > > luns seems just as busy. > > This setting should get rid of almost all readdir() calls. If it doesn't, > something's not working right. With maildir_very_dirty_syncs = yes: ReadMB/s WriteMB/s F_open f_close reads writes rdirinode 1.5 0.0 96 92 514 73 9 7 1.2 0.0 59 43 367 18 4 76 1.7 0.0 66 61 477 67 2 6 1.2 0.0 54 50 348 31 1 145 3.0 0.0 113 90 860 59 7 8 2.9 0.0 107 99 840 58 5 11 4.0 0.0 131 101 111777 2 65 With maildir_very_dirty_syncs = no (same node, a few seconds later): ReadMB/s WriteMB/s F_open f_close reads writes rdirinode 4.6 0.9 125 91 1161109641 6 2.3 0.7 200 170 697 127 86 16 1.1 0.6 124 99 406 61 48 109 2.7 0.1 212 144 755 114 74 15 2.7 0.0 159 133 818 70 78 194 0.8 1.2 86 73 225 60 16 9 1.9 0.0 124 116 573 53 30 6 So it seems to be working, good :-) -jf
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
Quoting Jan-Frode Myklebust : We've been collecting some stats to see what kind of benefits UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is running in High-performance mode http://wiki2.dovecot.org/LoginProcess with authentication caching http://wiki2.dovecot.org/Authentication/Caching During the weekend two servers (webmail3 and webmail4) has been running with local imapproxy and two servers without (webmail1 and webmail2). Each server has served about 1 million http requests, over 3 days. server avg. response time # requests webmail1.example.net 0.3704111092386 webmail2.example.net 0.3742271045141 webmail3.example.net 0.3780971043919 imapproxy webmail4.example.net 0.3785931028653 imapproxy ONLY requests that took more than 5 seconds to process: server avg. response time # requests webmail1.example.net 26.048 1125 webmail2.example.net 26.2997 1080 webmail3.example.net 28.5596 808 imapproxy webmail4.example.net 27.1004 964 imapproxy ONLY requests that took more than 10 seconds to process: server avg. response time # requests webmail1.example.net 49.1407 516 webmail2.example.net 53.0139 459 webmail3.example.net 59.7906 333 imapproxy webmail4.example.net 58.167 384 imapproxy The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot. Except you are most likely NOT leveraging the truly interesting part of imapproxy - the ability to restore the IMAP connection state via the XPROXYREUSE status response. This is a significant performance improvement since it also reduces processing load on the client side (everything before/including authentication needs to be done whether using imapproxy or not, so there is no client-side savings for these commands). For further information, see, e.g.: http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html These posts neglect the fact that you don't need to issue a CAPABILITY command if the connection is reused either, so that's an additional advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive user of the imapproxy instance for this feature to work correctly. Somewhat topical, since Timo was just mentioning support for some sort of IMAP state save/restore feature possibly making it into 2.2. michael
Re: [Dovecot] There is a way to know if a email has been expunged ?
On 13.2.2012, at 19.28, Alex Baule wrote:
> static void emexis_mailbox_allocated(struct mailbox *box)
> {
>union mailbox_module_context *zbox;
>
>zbox = p_new(box->pool, union mailbox_module_context, 1);
>zbox->super = box->v;
>box->v.open = Mplugin_mailbox_open;
>box->v.sync_notify = Mplugin_mailbox_sync_notify;
>
>MODULE_CONTEXT_SET_SELF(box, emexis_storage_module, zbox);
> }
The above is an "old way" to do it. It doesn't work well if you have any other
plugins loaded.
> struct mailbox_vfuncs *v = box->vlast;
> v->sync_notify = Mplugin_mailbox_sync_notify;
>
> But in this two cases, the Mplugin_mailbox_sync_notify is never
> called... i missing something ??
Never called at all? What storage backend are you using as the base? Maildir?
Anyway, the way you should be calling it is the exact same way quota plugin
does:
void Mplugin_mailbox_allocated(struct mailbox *box)
{
struct mailbox_vfuncs *v = box->vlast;
union mailbox_module_context *zbox;
zbox = p_new(box->pool, union mailbox_module_context, 1);
zbox->super = *v;
box->vlast = &zbox->super;
v->open = Mplugin_mailbox_open;
v->sync_notify = Mplugin_mailbox_sync_notify;
MODULE_CONTEXT_SET_SELF(box, Mplugin_storage_module, zbox);
}
[Dovecot] Logging to messages instead of secure
I need to log to /var/log/messages instead of /var/log/secure on a fedora system. Is this the correct entry for dovecot.conf? syslog_facility = info
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On 13.2.2012, at 21.36, Jan-Frode Myklebust wrote: >> Other things that would be interesting to try out (both from latency and >> disk IO usage point of view): >> >> - maildir_very_dirty_syncs > > We already have > > $ doveconf maildir_very_dirty_syncs > maildir_very_dirty_syncs = yes > > but I don't think this gave the advantage I was expecting.. Was > expecting this to move most iops to the index-luns, but the maildir > luns seems just as busy. This setting should get rid of almost all readdir() calls. If it doesn't, something's not working right. >> - mail_prefetch_count (Linux+maildir only, v2.1+) > > Will look into if this works with GPFS when we upgrade to v2.1. It has > it's own page cache, so I have no idea if it will respect > POSIX_FADV_WILLNEED or if one will need to use it's own API's for > hinting: > > > http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r4.gpfs300.doc%2Fbl1adm_mlacrge.html I guess if there's an easy way to lookup filename or fd -> blockNumber that wouldn't be difficult to implement with a plugin.
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On Mon, Feb 13, 2012 at 11:08:48AM -0800, Mark Moseley wrote: > > Out of curiosity, are you running dovecot locally on those webmail > servers as well, or is it talking to remote dovecot servers? The webmail servers are talking with dovecot director servers which in turn are talking with the backend dovecot servers. Each service running on different servers. Webmail-servers -> director-servers -> backend-servers > I ask because I'm looking at moving our webmail from an on-box setup to a > remote pool to support director and was going to look into whether > running imapproxyd would help there. We don't bother with it in the > local setup, since dovecot is so fast, but remote (but still on a LAN) > might be different. Doesn't seem so to us... > Though imapproxyd seems to make (wait for it...) > squirrelmail unhappy (complains about IMAP errors, when sniffing shows > none), though I've not bothered to debug it yet. :-) -jf
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On Mon, Feb 13, 2012 at 04:14:22PM +0200, Timo Sirainen wrote: > > The responstimes are not very fast, but they do seem to support > > the claim that an imapproxy isn't needed for dovecot. > > That's what I always suspected, but good to have someone actually test it. :) > This is with Maildir? Yes, this is maildirs (on GPFS). > > Other things that would be interesting to try out (both from latency and disk > IO usage point of view): > > - maildir_very_dirty_syncs We already have $ doveconf maildir_very_dirty_syncs maildir_very_dirty_syncs = yes but I don't think this gave the advantage I was expecting.. Was expecting this to move most iops to the index-luns, but the maildir luns seems just as busy. > - mail_prefetch_count (Linux+maildir only, v2.1+) Will look into if this works with GPFS when we upgrade to v2.1. It has it's own page cache, so I have no idea if it will respect POSIX_FADV_WILLNEED or if one will need to use it's own API's for hinting: http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r4.gpfs300.doc%2Fbl1adm_mlacrge.html -jf
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On Mon, Feb 13, 2012 at 5:54 AM, Jan-Frode Myklebust wrote: > We've been collecting some stats to see what kind of benefits > UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is > running in High-performance mode http://wiki2.dovecot.org/LoginProcess > with authentication caching http://wiki2.dovecot.org/Authentication/Caching > > During the weekend two servers (webmail3 and webmail4) has been running > with local imapproxy and two servers without (webmail1 and webmail2). Each > server has served about 1 million http requests, over 3 days. > > server avg. response time # requests > > webmail1.example.net 0.370411 1092386 > webmail2.example.net 0.374227 1045141 > webmail3.example.net 0.378097 1043919 imapproxy > webmail4.example.net 0.378593 1028653 imapproxy > > > ONLY requests that took more than 5 seconds to process: > > server avg. response time # requests > > webmail1.example.net 26.048 1125 > webmail2.example.net 26.2997 1080 > webmail3.example.net 28.5596 808 imapproxy > webmail4.example.net 27.1004 964 imapproxy > > ONLY requests that took more than 10 seconds to process: > > server avg. response time # requests > > webmail1.example.net 49.1407 516 > webmail2.example.net 53.0139 459 > webmail3.example.net 59.7906 333 imapproxy > webmail4.example.net 58.167 384 imapproxy > > The responstimes are not very fast, but they do seem to support > the claim that an imapproxy isn't needed for dovecot. Out of curiosity, are you running dovecot locally on those webmail servers as well, or is it talking to remote dovecot servers? I ask because I'm looking at moving our webmail from an on-box setup to a remote pool to support director and was going to look into whether running imapproxyd would help there. We don't bother with it in the local setup, since dovecot is so fast, but remote (but still on a LAN) might be different. Though imapproxyd seems to make (wait for it...) squirrelmail unhappy (complains about IMAP errors, when sniffing shows none), though I've not bothered to debug it yet.
Re: [Dovecot] dsync, hard-links and refcounts
On 02/13/2012 03:40 PM, Timo Sirainen wrote: It should deduplicate when GUIDs are the same.. I guess I'll have to look into why it's not working. I’d very much appreciate that. We will have to migrate many large mailboxes, and it would be a pity to needlessly waste space. I got the same result with some different configurations, using "mirror", "backup", or "-R backup" (though I have not tried 2.1 yet); so it should be easy to reproduce. However let me know if you need my configuration or anything. Cheers, Christoph
Re: [Dovecot] There is a way to know if a email has been expunged ?
Hi Again Timo !
I Trying to port quota plugin notify_sync to my plugin, but i found some issues.
I have the hooks already, to replace the mailbx_allocated, like quota
plugin does.
static struct mail_storage_hooks emexis_mail_storage_hooks = {
.mail_user_created = Mplugin_mail_user_created,
.mailbox_allocated = Mplugin_mailbox_allocated,
.mail_allocated = Mplugin_mail_allocated,
.mailbox_list_created = Mplugin_mailbox_list_created,
.mail_namespaces_created = Mplugin_antispam_mail_namespaces_created
};
inside the Mplugin_mailbox_allocated, i have:
static void emexis_mailbox_allocated(struct mailbox *box)
{
union mailbox_module_context *zbox;
zbox = p_new(box->pool, union mailbox_module_context, 1);
zbox->super = box->v;
box->v.open = Mplugin_mailbox_open;
box->v.sync_notify = Mplugin_mailbox_sync_notify;
MODULE_CONTEXT_SET_SELF(box, emexis_storage_module, zbox);
}
The v.open i rewrite because my plugin needs.
I try to call the sync_notify like this:
box->v.sync_notify = Mplugin_mailbox_sync_notify;
and follow quota plugin, i try to to this too:
struct mailbox_vfuncs *v = box->vlast;
v->sync_notify = Mplugin_mailbox_sync_notify;
But in this two cases, the Mplugin_mailbox_sync_notify is never
called... i missing something ??
Tks !
Em 12 de fevereiro de 2012 14:48, Alex Baule escreveu:
> Tks timo !
>
> I will see the plugin to do like it!
>
> Em 12/02/2012 13:46, "Timo Sirainen" escreveu:
>
>> Hi,
>>
>> Yeah, you shouldn't do the erasing directly in expunge(), because it may
>> still be aborted. Do it in sync_notify() like quota plugin does.
>>
>> On 12.2.2012, at 16.52, Alex Baule wrote:
>>
>> > I want to know because i have header and body splited. To erase the
>> > body, i must have shure, that header was expunged. Some tests that i
>> > made, the client call expunge, but i don't know why (there is notting
>> > in mail log) the header was not expunged, nut my body was, because i
>> > call it before expunge the header. something like this.
>> >
>> > if (found_body){
>> > erase_body();
>> > super.expunge(_mail);
>> > }
>> >
>> > Because that expunge that no happen, (the client call, but the email
>> > was not expunged) i think to do like this way:
>> >
>> > if (found_body){
>> > super.expunge(_mail);
>> > if(_mail was expunged){
>> > erase_body();
>> > }
>> > }
>> >
>> > TKs Timo !
>> >
>> > Em 12 de fevereiro de 2012 02:19, Timo Sirainen escreveu:
>> >> On 10.2.2012, at 19.39, Alex Baule wrote:
>> >>
>> >>> Hy Everyone...and Timo !
>> >>>
>> >>> There is a way to know if a email was expunged (deleted from hard
>> >>> disk) inside a plugin ? I rewrite the expunge function, but the real
>> >>> expunge function is void, i can't know if was really expunged.
>> >>>
>> >>> There is a way to know this ?
>> >>
>> >> It's not known until transaction_commit() what messages are expunged.
>> >> What do you need to know this for? There are a few different ways.
>> >>
>> >> If you simply need one of the sessions to definitely know that it
>> >> expunged a message, you can do it like quota plugin does. (So even if
>> >> there
>> >> are multiple clients doing EXPUNGE at the same time, quota is never
>> >> decreased more than once per mail.) See quota-storage.c and most
>> >> importantly
>> >> quota_mailbox_sync_notify().
>> >>
>> >> Quota code also shows the other method of doing it: keep track of what
>> >> mail_expunge()s have been called, and then in transaction commit check if
>> >> it
>> >> succeeds and if it does do what you want to do.
>> >>
>> >
>>
>
Re: [Dovecot] Managesieve doesn't listen on 2000/4190
On Mon, 13 Feb 2012, Timo Sirainen wrote: On 13.2.2012, at 18.26, Jacek Osiecki wrote: I have compiled pigeonhole, installed it, configured, everything seems to be done well - except the fact, that nothing is listening on port 2000 or 4190. Here is my doveconf -n. Any ideas? # 2.0.16: /etc/dovecot/dovecot.conf Try adding: protocols = imap pop3 sieve Great, it helped :) Thanks! Now time to see what surprises will show up while migrating there people from dovecot 1.0-stable :) Greetings, -- Jacek Osiecki jos...@ceti.pl GG:3828944 I don't want something I need. I want something I want.
Re: [Dovecot] Managesieve doesn't listen on 2000/4190
On 13.2.2012, at 18.26, Jacek Osiecki wrote: > I have compiled pigeonhole, installed it, configured, everything seems to be > done well - except the fact, that nothing is listening on port 2000 or 4190. > > Here is my doveconf -n. Any ideas? > > # 2.0.16: /etc/dovecot/dovecot.conf Try adding: protocols = imap pop3 sieve
[Dovecot] Managesieve doesn't listen on 2000/4190
Hi,
Finally I decided to give a try to dovecot 2.0, after spending few years
with 1.2. Unfortunately, it doesn't go smooth - finally I managed to cope
with LMTP transport and exim, but failed to run managesieve.
I have compiled pigeonhole, installed it, configured, everything seems to
be done well - except the fact, that nothing is listening on port 2000 or
4190.
Here is my doveconf -n. Any ideas?
# 2.0.16: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.39.4-vs2.3.1-pre9.2-nfs41-nfsd x86_64
auth_debug = yes
auth_debug_passwords = yes
debug_log_path = /var/log/dovecot/debug.log
dict {
quota = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
disable_plaintext_auth = no
first_valid_uid = 79
info_log_path = /var/log/dovecot/info.log
last_valid_uid = 79
log_path = /var/log/dovecot/main.log
mail_debug = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags
copy include variables body enotify environment mailbox date ihave
passdb {
driver = pam
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
sieve = /var/mail/sieve/%d/%u.sieve
sieve_dir = /var/mail/sieve/%d/%u
sieve_global_path = /var/mail/sieve/%d/global.sieve
}
postmaster_address = postmas...@hybrid.pl
service imap-login {
inet_listener imap {
address = 127.0.0.1
port = 143
}
inet_listener imaps {
address = 0.0.0.0
port = 993
ssl = yes
}
}
service lmtp {
inet_listener lmtp {
address = 127.0.0.1
port = 24
}
unix_listener lmtp {
mode = 0666
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
inet_listener sieve_deprecated {
port = 2000
}
}
service pop3-login {
inet_listener pop3 {
address = 127.0.0.1
port = 110
}
inet_listener pop3s {
address = 0.0.0.0
port = 995
ssl = yes
}
}
ssl = no
userdb {
driver = passwd
}
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocol lmtp {
mail_plugins = " sieve quota"
}
protocol lda {
mail_plugins = " sieve quota"
}
Best regards,
--
Jacek Osiecki jos...@ceti.pl GG:3828944
I don't want something I need. I want something I want.
Re: [Dovecot] Dovecot v2.2 plans
On 13.2.2012, at 17.35, foru...@smartmobili.com wrote: > Is there any plan to port dovecot to windows ? It probably works via Cygwin (although I think Cygwin has to do some kind of POSIX filesystem emulation, which slows things down and might cause some trouble if server crashes). Actually we have discussed this a little within my company.. I'm personally not interested in spending much time on it, but that's why we're hiring more coders so I won't have to do everything. :) If there is enough commercial interest, we might build something better than using Cygwin. Anyway, one thing is certain: No Dovecot for Windows questions in this mailing list. Perhaps I'll create another dovecot-windows@ mailing list. Perhaps I'll even give it a different name (dovedows? wincot? glassdove?)
Re: [Dovecot] Dovecot v2.2 plans
On 13.2.2012, at 17.51, Robert Schetterer wrote: > Am 13.02.2012 12:47, schrieb Timo Sirainen: >> Get rid of *.conf.ext files. Make everything part of dovecot.conf, so >> doveconf -n outputs ALL of the configuration > > might a question of taste, but i never liked the splitted up config > style, so i like this idea Note that I said *.conf.ext, not *.conf..
Re: [Dovecot] Dovecot v2.2 plans
Am 13.02.2012 12:47, schrieb Timo Sirainen: > Get rid of *.conf.ext files. Make everything part of dovecot.conf, so > doveconf -n outputs ALL of the configuration might a question of taste, but i never liked the splitted up config style, so i like this idea -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] Dovecot v2.2 plans
Am 13.02.2012 16:35, schrieb foru...@smartmobili.com: > Hi, > > Is there any plan to port dovecot to windows ? > cant wait Timos answer *g -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] Dovecot v2.2 plans
Hi, Is there any plan to port dovecot to windows ?
Re: [Dovecot] dsync, hard-links and refcounts
On 13.2.2012, at 16.16, Christoph Bußenius wrote: > when creating a copy of a mail, dovecot provides a feature that will store > its contents only once. In maildir, this is done by means of hard links, > while mdbox has some special refcounting mechanism. > > My question is, how can we convert mails from maildir to mdbox without > duplicating these copies? It seems that dsync does not detect the hard > links. Even if the hard-linked mails have the same GUID listed in > dovecot-uidlist, dsync creates multiple instances of the text. > - Is there some way to make dsync notice the hard links? (I used >dovecot 2.0.17) It should deduplicate when GUIDs are the same.. I guess I'll have to look into why it's not working.
[Dovecot] dsync, hard-links and refcounts
Hi, when creating a copy of a mail, dovecot provides a feature that will store its contents only once. In maildir, this is done by means of hard links, while mdbox has some special refcounting mechanism. My question is, how can we convert mails from maildir to mdbox without duplicating these copies? It seems that dsync does not detect the hard links. Even if the hard-linked mails have the same GUID listed in dovecot-uidlist, dsync creates multiple instances of the text. - Is there some way to make dsync notice the hard links? (I used dovecot 2.0.17) - Alternatively, is there some tool to deduplicate the mdbox after it has been converted from maildir? - NB: I am not talking about single-instance storage for attachments. Thank you very much, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 <> Raum 00.05.055 <> Boltzmannstr. 3 <> Garching
Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
On 13.2.2012, at 15.54, Jan-Frode Myklebust wrote: > The responstimes are not very fast, but they do seem to support > the claim that an imapproxy isn't needed for dovecot. That's what I always suspected, but good to have someone actually test it. :) This is with Maildir? Other things that would be interesting to try out (both from latency and disk IO usage point of view): - maildir_very_dirty_syncs - mail_prefetch_count (Linux+maildir only, v2.1+)
[Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend
We've been collecting some stats to see what kind of benefits UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is running in High-performance mode http://wiki2.dovecot.org/LoginProcess with authentication caching http://wiki2.dovecot.org/Authentication/Caching During the weekend two servers (webmail3 and webmail4) has been running with local imapproxy and two servers without (webmail1 and webmail2). Each server has served about 1 million http requests, over 3 days. server avg. response time # requests webmail1.example.net 0.3704111092386 webmail2.example.net 0.3742271045141 webmail3.example.net 0.3780971043919 imapproxy webmail4.example.net 0.3785931028653 imapproxy ONLY requests that took more than 5 seconds to process: server avg. response time # requests webmail1.example.net 26.048 1125 webmail2.example.net 26.2997 1080 webmail3.example.net 28.5596 808 imapproxy webmail4.example.net 27.1004 964 imapproxy ONLY requests that took more than 10 seconds to process: server avg. response time # requests webmail1.example.net 49.1407 516 webmail2.example.net 53.0139 459 webmail3.example.net 59.7906 333 imapproxy webmail4.example.net 58.167 384 imapproxy The responstimes are not very fast, but they do seem to support the claim that an imapproxy isn't needed for dovecot. -jf
[Dovecot] Dovecot v2.2 plans
Here's a list of things I've been thinking about implementing for Dovecot v2.2.
Probably not all of them will make it, but I'm at least interested in working
on these if I have time.
Previously I've mostly been working on things that different companies were
paying me to work on. This is the first time I have my own company, but the
prioritization still works pretty much the same way:
- 1. priority: If your company is highly interested in getting something
implemented, we can do it as a project via my company. This guarantees that
you'll get the feature implemented in a way that integrates well into your
system.
- 2. priority: Companies who have bought Dovecot support contract can let me
know what they're interested in getting implemented. It's not a guarantee that
it gets implemented, but it does affect my priorities. :)
- 3. priority: Things other people want to get implemented.
There are also a lot of other things I have to spend my time on, which are
before the 2. priority above. I guess we'll see how things work out.
dsync-based replication
---
I'll write a separate post about this later. Besides, it's coming for Dovecot
v2.1 so it's a bit off topic, but I thought I'd mention it anyway.
Shared mailbox improvements
---
Support for private flags for all mailbox formats:
namespace {
type = public
prefix = Public/
mail_location = mdbox:/var/vmail/public:PVTINDEX=~/mdbox/indexes-public
}
- dsync needs to be able to replicate the private flags as well as shared
flags.
- might as well add a common way for all mailbox formats to specify which
flags are shared and which aren't. $controldir/dovecot-flags would say which is
the default (private or shared) and what flags/keywords are the opposite.
- easy way to configure shared mailboxes to be accessed via imapc backend,
which would allow easy shared mailbox accesses across servers or simply between
two system users in same server. (this may be tricky to dsync.)
- global ACLs read from a single file supporting wildcards, instead of
multiple different files
- default ACLs for each namespace/storage root (maybe implemented using the
above..)
Metadata / annotations
--
Add support for server, mailbox and mail annotations. These need to be
dsyncable, so their changes need to be stored in various .log files:
1. Per-server metadata. This is similar to subscriptions: Add changes to
dovecot.mailbox.log file, with each entry name a hash of the metadata key that
was changed.
2. Per-mailbox metadata. Changes to this belong inside
mailbox_transaction_context, which write the changes to mailbox's
dovecot.index.log files. Each log record contains a list of changed annotation
keys. This gives each change a modseq, and also allows easily finding out what
changes other clients have done, so if a client has done ENABLE METADATA
Dovecot can easily push metadata changes to client by only reading the
dovecot.index.log file.
3. Per-mail metadata. This is pretty much equivalent to per-mailbox metadata,
except changes are associated to specific message UIDs.
The permanent storage is in dict. The dict keys have components:
- priv/ vs. shared/ for specifying private vs. shared metadata
- server/ vs mailbox// vs. mail//
- the metadata key name
This would be a good time to improve the dict configuration to allow things
like:
- mixed backends for different hierarchies (e.g. priv/mailbox/* goes to a
file, while the rest goes to sql)
- allow sql dict to be used in more relational way, so that mail annotations
could be stored with tables: mailbox (id, guid) and mail_annotation
(mailbox_id, key, value), i.e. avoid duplicating the guid everywhere.
Things to think through:
- How to handle quota? Probably needs to be different from regular mail quota.
Probably some per-user "metadata quota bytes" counter/limit.
- Dict lookups should be done asynchronously and prefetched as much as
possible. For per-mail annotation lookups mail_alloc() needs to include a list
of annotations that are wanted.
Configuration
-
Copy all mail settings to namespaces, so it'll be possible to use per-namespace
mailbox settings. Especially important for imapc_* settings, but can be useful
for others as well. Those settings that aren't explicitly defined in the
namespace will use the global defaults. (Should doveconf -a show all of these
values, or simply the explicitly set values?)
Get rid of *.conf.ext files. Make everything part of dovecot.conf, so doveconf
-n outputs ALL of the configuration. There are mainly 3 config files I'm
thinking about: dict-sql, passdb/userdb sql, passdb/userdb ldap. The dict-sql
is something I think needs a bigger redesign (mentioned above in "Metadata"
section), but the sql/ldap auth configs could be merged. One way could be:
sql_db sqlmails {
# most settings from dovecot-sql.conf.ext, except for queries
driver = mysql
connect = ...
Re: [Dovecot] Problems sending email direct into publich folders
On 13.2.2012, at 10.00, Kuran, Fabian wrote: >> Depends on how you want to do this.. For example all mails intended to be >> put to public namespace could be sent > to a "publicuser" named user, which >> has write permissions to the public namespace. Then you'll simply create a > >> sieve script for the publicuser which redirects the mails to the wanted >> folder (e.g. fileinto "public/hrztest"). > I'm also migrating from cyrus-imapd to dovecot. But this solution isn't very > good to me. With cyrus I could easily send an email to (for example) > bb+shared/arch...@mydomain.com. Then the mail will be putted to the public > folder. > > Is there a comparable way to do this with Dovecot? > FYI: I'm using dovecot in a clustered and multidomain setup. The Mails will > be deliviered over LMTP. You can set: lmtp_save_to_detail_mailbox = yes And I think then you should be able to send the mail to any user who has write access to it.
Re: [Dovecot] Issues with SIS and Backups - was Re: v2.1.0 status
On 13.2.2012, at 9.49, Jan-Frode Myklebust wrote: > On Sun, Feb 12, 2012 at 05:58:20PM +0200, Timo Sirainen wrote: >> >> doveadm backup -u user@domain backup: >> >> And it would output the user's messages to stdout (or to some file). So it >> would be similar to e.g. PostgreSQL's pg_dump. > > So only full backups, no incremental backups? I know people are doing incremental backups with PostgreSQL, so it should be possible with Dovecot too. I'm not sure about the details yet tough. > Then what's the benefit over just copying the files (of a snapshot)? If SIS is enabled, it's a little difficult to restore specific user's mails from a filesystem based backup.
Re: [Dovecot] Dovecot 2.0.18 build error with pigeonhole
Op 2/13/2012 11:11 AM, Morten Stevens schreef: On 13.02.2012 01:51, Bradley Giesbrecht wrote: On Feb 12, 2012, at 2:43 PM, Timo Sirainen wrote: On 13.2.2012, at 0.32, Morten Stevens wrote: It is probably a problem with pigeonhole 0.2.5. Yeah, unfortunately I had to break it to fix a bug.. Although I guess if I really wanted to I could have worked around it also. Anyway, http://hg.rename-it.nl/dovecot-2.0-pigeonhole/rev/58fc2f01c432 fixes it. Confirmed. Thanks. Okay, also confirmed. Release will follow later today. Regards, Stephan.
Re: [Dovecot] Dovecot 2.0.18 build error with pigeonhole
On 13.02.2012 01:51, Bradley Giesbrecht wrote: On Feb 12, 2012, at 2:43 PM, Timo Sirainen wrote: On 13.2.2012, at 0.32, Morten Stevens wrote: It is probably a problem with pigeonhole 0.2.5. Yeah, unfortunately I had to break it to fix a bug.. Although I guess if I really wanted to I could have worked around it also. Anyway, http://hg.rename-it.nl/dovecot-2.0-pigeonhole/rev/58fc2f01c432 fixes it. Confirmed. Thanks. Okay, also confirmed. Thank you. Best regards, Morten
Re: [Dovecot] Problems sending email direct into publich folders
Hi, > Depends on how you want to do this.. For example all mails intended to be put >to public namespace could be sent > to a "publicuser" named user, which has >write permissions to the public namespace. Then you'll simply create a > sieve >script for the publicuser which redirects the mails to the wanted folder (e.g. >fileinto "public/hrztest"). I'm also migrating from cyrus-imapd to dovecot. But this solution isn't very good to me. With cyrus I could easily send an email to (for example) bb+shared/arch...@mydomain.com. Then the mail will be putted to the public folder. Is there a comparable way to do this with Dovecot? FYI: I'm using dovecot in a clustered and multidomain setup. The Mails will be deliviered over LMTP. -- Bye, Fabian Kuran
