Re: [Dovecot] dovecot as gmail imap proxy
Asheesh Laroia wrote: On Tue, 17 Jun 2008, Farkas Levente wrote: hi, may be this is a bit of topic, but i hope not. it's always a big question which is the better to choose a hosting provider to keep and manage you mail or setup your own mail server, hosting, virus and spam filter etc.. now as more and more people and company move to gmail or google apps mail service it seems google can't handle the load and it's getting more and more slower. what's more not just the imap interface but nowadays the web interface used to be hang or even stop working. it's getting more and more annoying that the speed is worst than in a 10 years ago. i thing about how can this be solved without totally give up gmail as a mail service provider (and may be they can solve it in a few months/years:-). would it be possible to create a dovecot server as an imap proxy for gmail and google apps? What I have is: * rose, my primary server (but far away) * supercore, another server at my parents' house rose is the primary MX. I run offlineimap on supercore to sync a local Maildir on supercore with rose's IMAP - that way, my parents can use a server with <1ms ping, and as they use Dovecot to modify the local Maildir on supercore, a minute later offlineimap will wake up and create IMAP operations to synchronize rose with the state on supercore. i'm just look into offlineimap. half of the code for ui, another part to implement maildir and imap. so it seems for me there is only a little code which do the synchronization. Timo what do you think how much work would be add such a middleware imap server capabilities (ie. imap backend) to dovecot? -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] dovecot as gmail imap proxy
Timo Sirainen wrote: On Jun 17, 2008, at 6:14 PM, Timo Sirainen wrote: On Jun 17, 2008, at 6:03 PM, Farkas Levente wrote: would it be possible to create a dovecot server as an imap proxy for gmail and google apps? imho many people/company would be love to use such a setup where he can use gmail in vacation or any other time, but when the email is real critical apps (not acceptable to wait 1 minutes for an email to load or to open a folder/label) than he can use a proxy server eg. dovecot which act as a imap proxy to gmail and can be hosted on closer, faster server and this speed problem never happend. what's more it can be used as a mail backup server too. I've thought about a caching IMAP mail storage backend for Dovecot and even started implementing it once, but it didn't get very far. If someone wants to try to continue it, I can send what I've written so far. Although it's mostly a copy of cydir backend with a tiny bit of it replaced with IMAP code. Now that I think of it, the IMAP backend should do only direct IMAP backend accessing and if caching is wanted it could be a generic middle layer. Might be also useful for other things like caching mails from NFS server. Although Linux already has a cachefs for that. it seems other people also thing about this: http://community.livejournal.com/linux/1748995.html so it'd be useful thing for many of us:-) -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] dovecot as gmail imap proxy
Peter Hessler wrote: I do this exact thing, and it is faster than gmail itself. With IDLE, I get the msg alert on my dovecot server before the web interface shows a new message. which exact thing? and what does the "with idle" means? Also, far more reliable, with a superior UI. On 2008 Jun 17 (Tue) at 17:13:32 +0200 (+0200), Alexander Prinsier wrote: :Why don't you forward mail from your gmail account to your own server? :That's so much easier and more common. You get the benefit of good :filtering, and you have your own server to access your mail. If your own :server goes down, you still have gmail as backup. -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] dovecot as gmail imap proxy
Timo Sirainen wrote: On Jun 17, 2008, at 6:14 PM, Timo Sirainen wrote: On Jun 17, 2008, at 6:03 PM, Farkas Levente wrote: would it be possible to create a dovecot server as an imap proxy for gmail and google apps? imho many people/company would be love to use such a setup where he can use gmail in vacation or any other time, but when the email is real critical apps (not acceptable to wait 1 minutes for an email to load or to open a folder/label) than he can use a proxy server eg. dovecot which act as a imap proxy to gmail and can be hosted on closer, faster server and this speed problem never happend. what's more it can be used as a mail backup server too. I've thought about a caching IMAP mail storage backend for Dovecot and even started implementing it once, but it didn't get very far. If someone wants to try to continue it, I can send what I've written so far. Although it's mostly a copy of cydir backend with a tiny bit of it replaced with IMAP code. Now that I think of it, the IMAP backend should do only direct IMAP backend accessing and if caching is wanted it could be a generic middle layer. Might be also useful for other things like caching mails from NFS server. Although Linux already has a cachefs for that. yes, that's exactly what i'm thinking about. all imap command which issued by the client to the middle layer should have to propagate back to the backed (in this case gmail). but imho the hard part that we've to assume that the backend (in this case gmail) is slow, so the middle layer have to cache the imap commands to be able to propagate all command to the backend (and command queue for all mailbox). so if the connection between dovecot and gmail are fast then both has the same messages and flags, etc. if the connection is slow than gmail may have a small delay, but this delay probably won't be visible to the user (if he not use the web interface at the same time too). -- Levente "Si vis pacem para bellum!"
[Dovecot] dovecot as gmail imap proxy
hi, may be this is a bit of topic, but i hope not. it's always a big question which is the better to choose a hosting provider to keep and manage you mail or setup your own mail server, hosting, virus and spam filter etc.. now as more and more people and company move to gmail or google apps mail service it seems google can't handle the load and it's getting more and more slower. what's more not just the imap interface but nowadays the web interface used to be hang or even stop working. it's getting more and more annoying that the speed is worst than in a 10 years ago. i thing about how can this be solved without totally give up gmail as a mail service provider (and may be they can solve it in a few months/years:-). would it be possible to create a dovecot server as an imap proxy for gmail and google apps? imho many people/company would be love to use such a setup where he can use gmail in vacation or any other time, but when the email is real critical apps (not acceptable to wait 1 minutes for an email to load or to open a folder/label) than he can use a proxy server eg. dovecot which act as a imap proxy to gmail and can be hosted on closer, faster server and this speed problem never happend. what's more it can be used as a mail backup server too. anybody has any such setup, idea? would it be possible with dovecot. any other suggestion? yours. -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] Web-based shared calendar synchronizable with Outlook
gmail or google apps do everything fro you for free... On Fri, May 30, 2008 at 2:42 PM, Charles Marcus <[EMAIL PROTECTED]> wrote: > On 5/30/2008, Samuel HAMEAU ([EMAIL PROTECTED]) wrote: > >> Well, SW seems cool but misses the webcalendar part... >> > > No it doesn't... create an account and log into it. > > I don't care much for the interface, but it is there... > > -- > > Best regards, > > Charles > -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] Zimbra benchmarking
hi, nice test, but what would be also useful to compare cpu and memory usage! with zimbra it's more important then anything else. i'd like to see a comparison what kind of hardware (cpu and memory) required for: 10, 100, 500, 1000, 2000, 5000 mailbox server. this's where dovecot a big winner even for 10 mailbox. but to be fair zimbra is much more than dovecot it's integrate many things (ldap user and group management, samba domain controller, virus scanner, outlook connector (!) not just for email but also for calendar and address book, etc...) and to put together such a system with dovecot it's more than hard and may be impossible. Timo Sirainen wrote: > Now that I have a working kvm setup, I thought I'd finally try how > Zimbra works. This is mainly some microbenchmarking, so it may not have > much to do with actual performance in real life. > > Setup: > > - 1GB memory given to kvm (from host's 2GB) > - Intel Core 2 6600 (kvm uses only one CPU) > - CentOS 5 > - 15GB qcow2 image on XFS filesystem > - Zimbra 5.0 RC2 RHEL5 x86_64 > - Dovecot latest hg, dbox format > > Both Zimbra and Dovecot were tested running on the same kvm image. All > data could be cached in memory, so this didn't really test disk I/O. > > Dovecot was run with fsync_disable=no. Running with =yes would have made > its performance even better. > > Zimbra features/bugs > > > Zimbra's SEARCH command doesn't support substring searching as IMAP > requires: > > 1 search text performance > * SEARCH 136 183 227 231 232 233 245 > 1 OK SEARCH completed > 2 search text erformance > * SEARCH > 2 OK SEARCH completed > 3 search text performanc > * SEARCH > 3 OK SEARCH completed > > It's possible to search "performanc" in a non-standard way though: > > 4 search text "performanc*" > * SEARCH 136 183 227 231 232 233 245 > 4 OK SEARCH completed > > This means that the upcoming Dovecot v1.1 is probably the only IMAP > server that supports IMAP-compatible full text search indexes with > incremental updates. (Cyrus Squat requires rebuilding the whole index > from scratch just for adding one mail, which makes it kind of > impractical.) > > Zimbra doesn't support SORT or THREAD extensions. > > I had trouble benchmarking more than 5 simultaneous connections in one > mailbox. I'm not sure if this is a configurable setting somewhere, or if > there's just some bug. The error messages that my imaptest gave looked > like Zimbra was buggy, but I didn't look at them too closely. > > STORE command also seemed to be buggy, giving lots of "STORE failed" > errors, even with just one connection. > > Zimbra uses 500MB of memory just to start up, so it's not exactly for > hosting small installations. > > Login+Logout > > > Dovecot: > > ./imaptest - select=0 secs=10 seed=0 clients=1 > Logi Logo > 100% 100% > 3439 6878 > ./imaptest - select=0 secs=10 seed=0 clients=10 > Logi Logo > 100% 100% > 4415 8830 > > Zimbra: > > ./imaptest - select=0 secs=10 seed=0 clients=1 > Logi Logo > 100% 100% > 18032 36064 > ./imaptest - select=0 secs=10 seed=0 clients=10 > Logi Logo > 100% 100% > 25519 51056 > > Looks like Dovecot's imap process creation hurts it a lot compared to > Zimbra's thread creation. Luckily IMAP connections are usually > long-living, so this shouldn't matter much, except for webmails for > which you can use imapproxy in the middle. > > LIST "" * > - > > Dovecot: > > ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 > Logi List > 100% 100% >1 181411 > ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 > Logi List > 100% 100% > 10 133451 > > Zimbra: > > ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=1 > Logi List > 100% 100% >1 1021 > ./imaptest - select=0 logout=0 list=100 secs=10 seed=0 clients=10 > Logi List > 100% 100% > 10 972 > > Yes, Dovecot's LIST is over 100 times faster. > > STATUS INBOX (MESSAGES UNSEEN RECENT) > - > > 100 messages in INBOX > > dovecot: > > ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 > Logi Stat > 100% 100% >1 191003 > ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 > Logi Stat > 100% 100% > 10 171171 > > Zimbra: > > ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=1 > Logi Stat > 100% 100% >1 3697 > ./imaptest - logout=0 select=0 status=100 secs=10 seed=0 clients=10 > Logi Stat > 100% 100% > 10 7009 > > FETCH 1:* (UID FLAGS ENVELOPE INTERNALDATE BODYSTRUCTURE) > - > > 100 messages in INBOX. > > dovecot: > ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=1 > Logi Sele Fetc > 100% 100% 100% >11 2769 > ./imaptest - logout=0 select=100 fetch=100 secs=10 seed=0 clients=5 > Logi Sele Fetc > 100% 100% 100% >55 2902 > > Zimbra: > > ./imaptest - logout=0 select=100 fetch=100
Re: [Dovecot] receive date wrong in outlook
Timo Sirainen wrote: > On Sun, 2007-10-21 at 15:21 +0200, Farkas Levente wrote: >> probably that's the problem. what internaldate in dovecot, the file's >> modify date in the maildir? i'd be nice if i can somehow set >> internaldate to the date field with dovecot (actually i already wrote a >> shell script but it's not the beast solution). > > This is a problem only when migrating mails, and it should be done only > then, so I can't think of a reasonable way to implement this in Dovecot. > I think a shell script that's run once is the best way. but maybe a shell script which use dovecot internal header parsing would be much more better then my not so clever ones:-) -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] receive date wrong in outlook
Timo Sirainen wrote: > On Sun, 2007-10-14 at 11:53 +0200, Farkas Levente wrote: >> hi, >> we migrate from exchange to dovecot. we do it with imapsync. first we >> try to do it with --syncinternaldates, but this fails on many messages >> with invalid date, so we drop this option. unfortunately after this >> those users who use outlook see the received date as the date we do the >> migration in stead of the real received date. other clients eg webmail, >> thunderbird etc sees the right date. so my question what can i do? it's >> a dovecot bug or outlook and if outlook's bug is there any workaround >> fro this? > > The clients use either Date: header or IMAP INTERNALDATE. In some > clients it's configurable which one to use. I prefer INTERNALDATE > myself. > > So the only solution would be to sync the internaldates as well. What do > you mean messages have invalid dates? As long as imapsync uses a > properly formatted date for APPEND, it shouldn't fail. Although I did > some fixes related to this in v1.0.1. > > If exchange returns INTERNALDATEs with invalid format, imapsync could > figure this out itself and just not use it then. probably that's the problem. what internaldate in dovecot, the file's modify date in the maildir? i'd be nice if i can somehow set internaldate to the date field with dovecot (actually i already wrote a shell script but it's not the beast solution). -- Levente "Si vis pacem para bellum!"
[Dovecot] receive date wrong in outlook
hi, we migrate from exchange to dovecot. we do it with imapsync. first we try to do it with --syncinternaldates, but this fails on many messages with invalid date, so we drop this option. unfortunately after this those users who use outlook see the received date as the date we do the migration in stead of the real received date. other clients eg webmail, thunderbird etc sees the right date. so my question what can i do? it's a dovecot bug or outlook and if outlook's bug is there any workaround fro this? thanks in advance. -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] v1.1.alpha4 released / about dbox
On Vas, Szeptember 9, 2007 06:01, Timo Sirainen wrote: > On Sat, 2007-09-08 at 15:04 +0200, Farkas Levente wrote: >> Timo Sirainen wrote: >> > But since there's still a chance that index files could break >> (although >> > v1.1 tries harder than v1.0 to fix problems), it would be nice if the >> > flags/keywords were written to metadata block once in a while. So if >> the >> > index files are lost, flag changes wouldn't be completely lost. >> Metadata >> > updates of course use disk I/O so they shouldn't be done too often. >> > >> > I was thinking that the metadata could be updated: >> > >> > - if IMAP connection has been idling for 4 hours (not changing flags) >> > - when closing mailbox and there are changes older than 4h >> > - immediately if there are changes older than 24h (whenever mailbox is >> > being synced, e.g. SELECT/NOOP/STATUS) >> > >> > Or something like that. Those rules can of course be changed, but I'm >> > not sure if I should bother making them configurable from >> dovecot.conf. >> > There are already too many settings. >> >> what about a maintanance srcipt/daemon which can be run from cron and >> every sysadm can decided when and how often he'd like to update >> metadatas? > > That would require keeping the ": " information > in some central database. Otherwise if you had lots of mailboxes it > would waste a lot of disk I/O in such run. And I'm not really interested > in creating such a database at least yet. :) than may ber move this code to lda also helps. my main reason to suugest the above is the same why i like lda: try to distribute the system load in time. that's the main problem with indexing at reading time. every morning when most people start to read his mail dovecot start to index all mail which creates high load with lda we distribute this load from mail read time to mail arrival time which is much better place! the same should have to be do with metadata update. somehow distribute the load eg. move to some time which is not so important during the night or move to lda which happend during arrival time in stead of read time or any other place but not during read. --
Re: [Dovecot] v1.1.alpha4 released / about dbox
Timo Sirainen wrote: > But since there's still a chance that index files could break (although > v1.1 tries harder than v1.0 to fix problems), it would be nice if the > flags/keywords were written to metadata block once in a while. So if the > index files are lost, flag changes wouldn't be completely lost. Metadata > updates of course use disk I/O so they shouldn't be done too often. > > I was thinking that the metadata could be updated: > > - if IMAP connection has been idling for 4 hours (not changing flags) > - when closing mailbox and there are changes older than 4h > - immediately if there are changes older than 24h (whenever mailbox is > being synced, e.g. SELECT/NOOP/STATUS) > > Or something like that. Those rules can of course be changed, but I'm > not sure if I should bother making them configurable from dovecot.conf. > There are already too many settings. what about a maintanance srcipt/daemon which can be run from cron and every sysadm can decided when and how often he'd like to update metadatas? -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] quota: maildrop + dovecot. dovecot doesn't update maildirsize file
Timo Sirainen wrote: > On 25.7.2007, at 12.49, Frittella Laurento wrote: > >> Now it seems to work using ldap backend to sync maildrop and dovecot >> info ;) >> >> I followed wiki example and I'm using this simple convert script because >> I'm storing quota info in ldap using the courier format S to use >> it with maildrop (through authlib) too > > With v1.1 it's possible to do: > > user_attrs = ...,quotaCourier=quota_rule=*:backend=%$ would you update the quota wiki page since currently it's not too clear how to use (what's the global qouta setting, what can and how comes from userdb and what is the correct syntax). thanks. -- Levente "Si vis pacem para bellum!"
[Dovecot] RFE: please include quota waning patch
hi, it'd be very useful to include the quota warning patch in official release. without it the quota support is not really useful since mail simple dropped when quota is over. and most enduser never know what happend, they just recognize mails are not coming:-( thanks. -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] v1.1 status and benchmarks
Timo Sirainen wrote: > v1.1 plans have changed a bit. I'll release v1.1.alpha1 soon and hope to > have a stable v1.1 in a month or two. The rest of the features that > didn't make it into v1.1 will go to v1.2. I'll write more about this > when v1.1 alpha is released. > > I also did a bit of benchmarking. v1.1's performance improvements are > looking pretty great, it seems to be twice as fast as v1.0. > > v1.0: Maildir, dovecot-1.0 hg + inotify dotlock wait patch, mmap_disable=yes > v1.1: Maildir, dovecot hg, mmap_disable=no > cydir: Cydir, dovecot hg, mmap_disable=no > > 1 client > > > ./imaptest logout=0 seed=1 secs=10 clients=1 > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% >30% 5% > v1.0:1 1221 11791 2420 3458 323 1934 2419 2552 > v1.1:1 2606 25971 5221 7352 688 4181 5221 5489 > cydir: 1 4495 45341 8960 12786 1216 7224 8959 9434 > > 10 clients > -- > > ./imaptest logout=0 seed=1 secs=10 clients=10 > Logi List Stat Sele Fetc Fet2 Stor Dele Expu Appe > 100% 50% 50% 100% 100% 100% 50% 100% 100% 100% > 30% 5% > v1.0: 10 2058 2102 10 4157 6019 1426 3348 4157 4391 > v1.1: 10 5029 5072 10 10064 14352 3176 8092 10064 10624 > cydir: 10 6332 6393 10 12664 17965 3958 10154 12663 13336 what is cydir? dovecot suppurt cydir format or it's cyrus speed? if it's dovecot cydir support so fast will the v.x.y be ever so fast? or it'd be better to use/switch to cydir? is there any converter script? -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] Updated v1.1 and summer plans
Timo Sirainen wrote: > On Tue, 2007-05-22 at 00:52 +0800, M1 wrote: >> Dear Timo, >> >> How about managedsieve? > > I think it'll have to wait for v2.0. Especially because I want it to be > distributed in dovecot-sieve package, not in the main dovecot package. > This just isn't possible without the larger changes that v2.0 brings. > but it'd be a real need for everyone who use lda since we can't use procmail any longer and there is no other option:-( -- Levente "Si vis pacem para bellum!"
Re: [Dovecot] Updated v1.1 and summer plans
Timo Sirainen wrote: > Webmail.us will be sponsoring my Dovecot development for this summer. > Other companies are also welcome to participate in the costs. > Participation gets you: > > - listed in Credits in www.dovecot.org > - listed in AUTHORS file > - you can tell me how you want to use the feature and I'll make sure > that it supports it (within reasonable limits) > > For already implemented v1.1 features, see > http://www.dovecot.org/list/dovecot/2007-April/021974.html > > Some of those features require some more testing (new SORT and THREAD > code especially) and a bit of fixes to mailbox list indexes. Otherwise I > think it's working pretty well already, I've been using it for a long > time for my own mails. > > Upcoming v1.1 features, more or less implemented in this order: what's the plan for the 1.0 series? it'll be 1.0.x release soon and ofter or ...? yours. -- Levente "Si vis pacem para bellum!"
