Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Am 27.06.2012 07:44, schrieb Wojciech Puchar: Timo: I'm not sure if you are saying that all client-side caching is wrong. If so, I'm going to disagree with you, especially when dealing with more complex data structures. it is always good - on WAN links. Hi, i dont wanna flame into this thread, cause its heavy tec stuff which i dont really fit in but for some webmail you can use http://imapproxy.org/ its running here fine with squirrelmail and roundcube -- Best Regards MfG Robert Schetterer
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Hi, i dont wanna flame into this thread, cause its heavy tec stuff which i dont really fit in but for some webmail you can use http://imapproxy.org/ the discussion was about if running proxy at all make sense. Proxies are to reduce traffic or server load by avoiding repetitive requests. With dovecot it's unlikely proxy itself will be faster, so second reason doesn't exist. With same computer or fast lan or virtual lan (==normal way of running webmail) first reason doesn't exist.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Am 27.06.2012 09:32, schrieb Wojciech Puchar: Hi, i dont wanna flame into this thread, cause its heavy tec stuff which i dont really fit in but for some webmail you can use http://imapproxy.org/ the discussion was about if running proxy at all make sense. Proxies are to reduce traffic or server load by avoiding repetitive requests. With dovecot it's unlikely proxy itself will be faster, so second reason doesn't exist. With same computer or fast lan or virtual lan (==normal way of running webmail) first reason doesn't exist. Hi, sorry ,only my meaning, beside coding layout questions about dovecot etc which is clearly not my case for questions like: does a proxy make sense there will never be an uni right answer the answer may ever depend on what fits best at your side general setup/layout -- Best Regards MfG Robert Schetterer
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On Thu, Jun 21, 2012 at 11:44:33PM +0300, Timo Sirainen wrote: additionally you should install imapproxy on the webserver wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. That was me, there - http://dovecot.org/list/dovecot/2012-February/063666.html -jf
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. nothing strange. I really wonder if there are available FASTER implementations of imap service. Quite probably not. It's stupid how webmail works but dovecot doesn't have a problem to get new connections every now and then. just make sure you didn't set up SSL by accident.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Quoting Timo Sirainen t...@iki.fi: On 23.6.2012, at 13.21, Ed W wrote: But I don't know, whether this is the sort of caching you are referring to. what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server? Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago? Can be, but is it? :) Dovecot attempts to cache/index stuff as well. Normally there shouldn't be a need for extra caching layer except in cases of higher network latency. Timo: I'm not sure if you are saying that all client-side caching is wrong. If so, I'm going to disagree with you, especially when dealing with more complex data structures. Let me first say that I don't take IMAP response parsing to be a computationally easy action. So it's not just network latency you are worrying about; parsing a line can be the limiting factor in many cases. For example, a deeply threaded 400 message mailbox will return a THREAD response line that will take quite a bit of recursive parsing to decode. And various FETCH criteria most definitely benefit from local caching above/beyond what dovecot provides. An example: BODYSTRUCTURE. This may be cached on the dovecot side, but when received by the MUA you have to parse the IMAP BODYSTRUCTURE response (not trivial). You also have to potentially handle IMAP response codes in the server command completion line. And the bodystructure data is probably not all that useful until converted to a usable object on the MUA side, which may be another relatively expensive operation. So a locally cached bodystructure object is a substantial performance benefit over having to recreate this data from the cached data on the dovecot side. ENVELOPE is similar. Most likely this will be converted to an object representation in the MUA so you have the same benefits as BODYSTRUCTURE. Additionally, in IMP we do things like scan for broken charset headers (e.g. Subject headers that contain non-ASCII characters) and have some algorithms to fix these issues. This value-added code would be prohibitively expensive if we have to do it on every mailbox access. Message flags are another benefit to caching. The list of flags may be cached on dovecot, but not having to issue a flag FETCH every time you access a mailbox can be a substantial benefit. But I will heartily agree that nobody should be caching things like headertext or bodypart data. There is little/no benefit you receive from caching this locally. This is where you should be leveraging the storage on the IMAP server. As an MUA author you can't rely on the fact that you are connecting to a competent IMAP server. You just as likely are going to be connecting to a server that implements base RFC 3501, and most likely implements that incorrectly. Not all of us are lucky to connect to Dovecot (or Cyrus). So smart caching most definitely can and will increase performance of an MUA, regardless of caching performed by the IMAP server. michael
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Quoting Wojciech Puchar woj...@wojtek.tensor.gdynia.pl: It's stupid how webmail works but dovecot doesn't have a problem to get new connections every now and then. just make sure you didn't set up SSL by accident. Would you mind explaining why you think it is stupid the way webmail works? I assume you are angry because a webmail installation will normally need to create a new IMAP connection on every user interaction at the browser level. Unfortunately, HTTP is a stateless protocol which makes webmail a disconnected client. But it is no different than other disconnected clients, e.g. mail app on a smartphone. I am confused on why you think this is stupid. The existence of disconnected clients has been contemplated since the beginning of IMAP (see RFC 1733; RFC 4549), and much work has been done to the IMAP protocol (CONDSTORE, QRESYNC, to a lesser extent SORT/THREAD) to increase performance on these clients - especially since that's where MUA usage is exploding. michael
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Quoting Timo Sirainen t...@iki.fi: Well, I had completely forgotten about it :) Reading my old mail: There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works). Yeah .. Perhaps something like: 1. if client issues LOGOUT XSTATE 2. And server sees that it can actually save all of the state (some things are a bit tricky, and probably not worth the trouble in initial implementation) 3. Then the server server sends * OK XSTATE string * BYE This makes sense. Although wouldn't it be: * OK [XSTATE string] State saved. 4. The client can pipeline after LOGIN/AUTHENTICATE: a XSTATERESTORE string a OK Yeah! or a NO Not gonna work. Couple of suggestions here: 1) Maybe allow XSTATERESTORE to be sent BEFORE authentication also/instead? The way that Dovecot would restore state might be different from the way another IMAP server would restore state. It's possible that another server could optimize things if, at authentication time, it knew it was going to restore state. i.e.: a XSTATERESTORE string a OK Will attempt to restore state. b (LOGIN/AUTHENTICATE command) * OK [XSTATERESTOREOK] State restored. -- or -- * OK [XSTATERESTORENO] State NOT restored. b OK Logged in. 2) Could extend LOGIN/AUTHENTICATE to accept XSTATERESTORE parameter. Pros: saves round-trip. Cons: extending LOGIN/AUTHENTICATE at this stage of IMAP 4 development is probably overkill (Although this implementation already requires extending the LOGOUT command) Perhaps even a real RFC for this thing? .. If it's worth it.. Would save at least a few X bytes from network traffic :) It could potentially be a few more than X bytes. Here's an extreme example of the potential savings: Initial connection: --- 1 (LOGIN/AUTHENTICATE) 1 OK Logged in. 2 CAPABILITY * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE ACL RIGHTS=texk 2 OK Capability completed. 3 ID (name foo version 1.0) * ID (name foo2 version bar2 os linux) 3 OK ID completed 4 ENABLE QRESYNC * ENABLED QRESYNC 4 OK Enabled. 5 COMPARATOR de;* i;basic * COMPARATOR i;basic 5 OK Will use i;basic for collation 6 LANGUAGE DE * LANGUAGE (DE) * NAMESPACE (( /)) ((Other Users/ / TRANSLATION (Andere BenAPw-tzer/))) ((Public Folders/ / TRANSLATION (Gemeinsame PostfAM8-cher/))) 6 OK Sprachwechsel durch LANGUAGE-Befehl ausgefuehrt [IMAP session] 50 LOGOUT XSTATE * OK [XSTATE 123abc] * BYE Subsequent connection: -- 1 XSTATERESTORE 123abc 1 OK Will attempt to restore state. 2 (LOGIN/AUTHENTICATE) * OK [XSTATERESTOREOK] State restored. 2 OK Angemeldet. Given this (admittedly) extreme example, the savings are 689 bytes (+126 bytes for staterestore overhead, -815 bytes for state setup). Additionally, the server/client have to process 4 less IMAP commands. This is a significant savings IMHO. Whether or not this is appropriate for a real RFC, it would probably be useful to document in RFC fashion regardless. michael
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Timo: I'm not sure if you are saying that all client-side caching is wrong. If so, I'm going to disagree with you, especially when dealing with more complex data structures. it is always good - on WAN links.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
We are building a new system that will support a large number of users (high volume, high concurrent usage, etc). what is large? We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut? No idea how well courier IMAP performs. But have idea how well dovecot performs. I don't have large configs like thousands of users as i don't handle herd of random users style cases, but in every place i have dovecot IMAP takes unnoticable amount of server load. Just make a test. Definitely use maildir format, not mbox. dovecot heavily accesses it's index files. they are not large relative to e-mail sizes. With really large case if I/O will limit you i would recommend using SSD storage to keep just indexes.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Nearly all of them are non-caching. (I don't know of any caching ones.) which is definite adventage in spite of it's numerous security holes.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Nearly all of them are non-caching. (I don't know of any caching ones.) At least roundcube (v0.7.1 here) has some caching options: --[excerpt from roundcubes main.inc.php]- // Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'. $rcmail_config['imap_cache'] = null; // Enables messages cache. Only 'db' cache is supported. $rcmail_config['messages_cache'] = false; -[end] But I don't know, whether this is the sort of caching you are referring to. what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server?
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 23/06/2012 09:22, Wojciech Puchar wrote: Nearly all of them are non-caching. (I don't know of any caching ones.) At least roundcube (v0.7.1 here) has some caching options: --[excerpt from roundcubes main.inc.php]- // Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'. $rcmail_config['imap_cache'] = null; // Enables messages cache. Only 'db' cache is supported. $rcmail_config['messages_cache'] = false; -[end] But I don't know, whether this is the sort of caching you are referring to. what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server? Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago? Ed
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
But I don't know, whether this is the sort of caching you are referring to. what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server? Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago? fine. how about overhead of cache itself?
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 23.6.2012, at 13.21, Ed W wrote: But I don't know, whether this is the sort of caching you are referring to. what's a point of caching imap, except your webmail service is not locally connected (localhost or LAN) to imap server? Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago? Can be, but is it? :) Dovecot attempts to cache/index stuff as well. Normally there shouldn't be a need for extra caching layer except in cases of higher network latency.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Asking for items 600-615 from a threaded list, sorted by something, can be an expensive operation, especially if you just asked for items 585-600 a moment ago? Can be, but is it? :) Dovecot attempts to cache/index stuff as well. Normally there shouldn't be a need for extra caching layer except in cases of higher network latency. that is my point. and - esp. with webmail, i see no point to run such service in different place than dovecot server runs. Best - same server. Maybe - other server connected with fast LAN.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 22.6.2012, at 8.27, email builder wrote: So really, a new process is created under *two* circumstances? 1. when a process reaches client_limit number of *simultaneous* connections or 2. when a process has serviced service_count number of connections. Is this correct? Yes. So for service *-login, is it OK to do something like service_count=5000, client_limit=2000 It would work, but for login processes the service_count can be 0. I haven't seen them leaking any memory recently. One somewhat annoying thing with service_count1 is that the processes have to wait until all of the connections have disconnected before shutting down. For processes handling long running connections (especially IMAP) this can mean that you'll end up with a lot of processes that are ready to shutdown but a couple of connections prevent it from doing this.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 21.6.2012, at 21.05, email builder wrote: We are building a new system that will support a large number of users (high volume, high concurrent usage, etc). We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut? If you disable index index files in Dovecot, its performance should be slightly better than Courier. With index files the performance is typically much better in Dovecot, especially if you use a (non-caching) webmail. Any tips on making the migration (not migrating an existing system, I mean migrating our paradigm - things to consider, things to watch out for)? If you don't migrate any existing users, I guess this doesn't differ much from any other optimized Dovecot installation. Usually large installations (1M users) use NetApp NFS + Dovecot director. You might also want to enable full text searches. http://wiki2.dovecot.org/PerformanceTuning lists some other things.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Thank you very much for the fast reply. We are building a new system that will support a large number of users (high volume, high concurrent usage, etc). We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut? If you disable index index files in Dovecot, its performance should be slightly better than Courier. With index files the performance is typically much better in Dovecot, especially if you use a (non-caching) webmail. Interesting. What would be the motivations for disabling indexing? Indexing is by default enabled? Do you know what webmails are caching vs. non-caching? Am I correct that what you're pointing out is that with non-caching webmails you will notice IMAP performance differences more readily but that a caching webmail application might be better no matter which IMAP server because it reduces the need for webmail to make IMAP connections? Any tips on making the migration (not migrating an existing system, I mean migrating our paradigm - things to consider, things to watch out for)? If you don't migrate any existing users, I guess this doesn't differ much from any other optimized Dovecot installation. Usually large installations (1M users) use NetApp NFS + Dovecot director. You might also want to enable full text searches. http://wiki2.dovecot.org/PerformanceTuning lists some other things. Ah, I didn't know about Director. That looks very nice. I had in mind that we would have to use Perdition, but an integrated solution might be good. Anyone have any thoughts or opinions considering Perdition vs. Director? Full text searches don't hurt performance too bad? Thanks for the other links, I will certainly go read up on them.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On Thu, 2012-06-21 at 13:05 -0700, email builder wrote: Thank you very much for the fast reply. We are building a new system that will support a large number of users (high volume, high concurrent usage, etc). We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut? If you disable index index files in Dovecot, its performance should be slightly better than Courier. With index files the performance is typically much better in Dovecot, especially if you use a (non-caching) webmail. Interesting. What would be the motivations for disabling indexing? Indexing is by default enabled? Yes, enabled by default. There aren't many good reasons for disabling indexing. Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) Am I correct that what you're pointing out is that with non-caching webmails you will notice IMAP performance differences more readily but that a caching webmail application might be better no matter which IMAP server because it reduces the need for webmail to make IMAP connections? It's not about the IMAP connections themselves, but how often they fetch message (meta)data. http://www.imapwiki.org/Benchmarking should explain this better. Dovecot's indexing can lower the disk I/O usage perhaps by 10x compared to Courier. Any tips on making the migration (not migrating an existing system, I mean migrating our paradigm - things to consider, things to watch out for)? If you don't migrate any existing users, I guess this doesn't differ much from any other optimized Dovecot installation. Usually large installations (1M users) use NetApp NFS + Dovecot director. You might also want to enable full text searches. http://wiki2.dovecot.org/PerformanceTuning lists some other things. Ah, I didn't know about Director. That looks very nice. I had in mind that we would have to use Perdition, but an integrated solution might be good. Anyone have any thoughts or opinions considering Perdition vs. Director? Dovecot proxy has several Dovecot-specific features that make it work better than perdition (forwards client IP address to backend, handle CAPABILITY stuff better, maybe other things). Full text searches don't hurt performance too bad? They should improve the performance, at least from the user's point of view when doing a search on webmail. But yes, the indexing itself does cost CPU cycles, disk I/O and disk usage (perhaps 30% more disk space).
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Am 21.06.2012 22:22, schrieb Timo Sirainen: Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) roundcube can if configured additionally you should install imapproxy on the webserver wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Am 21.06.2012 22:22, schrieb Timo Sirainen: On Thu, 2012-06-21 at 13:05 -0700, email builder wrote: Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) At least roundcube (v0.7.1 here) has some caching options: --[excerpt from roundcubes main.inc.php]- // Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'. $rcmail_config['imap_cache'] = null; // Enables messages cache. Only 'db' cache is supported. $rcmail_config['messages_cache'] = false; -[end] But I don't know, whether this is the sort of caching you are referring to. - René signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On Jun 21, 2012, at 3:22 PM, Timo Sirainen wrote: On Thu, 2012-06-21 at 13:05 -0700, email builder wrote: Thank you very much for the fast reply. We are building a new system that will support a large number of users (high volume, high concurrent usage, etc). We have played with Dovecot, but in most serious applications we have traditionally used Courier IMAP. It's my (lay) understanding that with indexing and perhaps other things in Dovecot, it might perform better than Courier in larger environments like this. Am I correct or is it less clear-cut? If you disable index index files in Dovecot, its performance should be slightly better than Courier. With index files the performance is typically much better in Dovecot, especially if you use a (non-caching) webmail. Interesting. What would be the motivations for disabling indexing? Indexing is by default enabled? Yes, enabled by default. There aren't many good reasons for disabling indexing. Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) Prayer, from University of Cambridge, or Chickadee, a fork of it. It's essentially a proper IMAP client in C that runs on a server, and uses HTTPS (via an embedded server, no external dependency on apache or etc.) to the end user just to deliver the display. When I was on the email project for the University of Minnesota, I modified it heavily for interface and to add some features that admins are used to having in systems where apache is involved (virtual hosts, things like that). I have it available (GPL) as a vanilla, de-branded package--Chickadee. Website is currently offline as I've been switching hosts, anyone who's interested can feel free to drop me a line. -Brian
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 21.6.2012, at 23.34, Reindl Harald wrote: Am 21.06.2012 22:22, schrieb Timo Sirainen: Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) roundcube can if configured additionally you should install imapproxy on the webserver wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Am 21.06.2012 22:44, schrieb Timo Sirainen: On 21.6.2012, at 23.34, Reindl Harald wrote: Am 21.06.2012 22:22, schrieb Timo Sirainen: Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) roundcube can if configured additionally you should install imapproxy on the webserver wehre your webmail is running and configure the webmail for using 127.0.0.1 - so only one connection per user is persistent instead make a new one for each ajax-request Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. depends on network-latency, parallel users and last but not least count of folders - if you have 30 folders and roundcube refreshs every 20 seconds it will make in the worst case 180 connections for one user per minute maybe a bechmark with high load shows other values but felt performance in our setup is much better with imapproxy in front - roundcube feels like a desktop client signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 21.6.2012, at 23.48, Reindl Harald wrote: Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. depends on network-latency, parallel users and last but not least count of folders - if you have 30 folders and roundcube refreshs every 20 seconds it will make in the worst case 180 connections for one user per minute Really? Doesn't it simply connect once every 20 seconds and send 30 STATUS requests in one connection?
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Am 21.06.2012 22:52, schrieb Timo Sirainen: On 21.6.2012, at 23.48, Reindl Harald wrote: Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. depends on network-latency, parallel users and last but not least count of folders - if you have 30 folders and roundcube refreshs every 20 seconds it will make in the worst case 180 connections for one user per minute Really? Doesn't it simply connect once every 20 seconds and send 30 STATUS requests in one connection? not 100% sure i simply tried it with proxy, was happy that it feels faster and last but not least i have lesser entries in maillog which goes to a central mysql-server for self-developed web-interfaces signature.asc Description: OpenPGP digital signature
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 21.6.2012, at 23.48, Reindl Harald wrote: Someone benchmarked Dovecot a while ago in this list with and without imapproxy and the results showed that imapproxy simply slowed things down by adding extra latency. This probably isn't true for all installations, but I don't think there's much of a difference either way. depends on network-latency, parallel users and last but not least count of folders - if you have 30 folders and roundcube refreshs every 20 seconds it will make in the worst case 180 connections for one user per minute maybe a bechmark with high load shows other values but felt performance in our setup is much better with imapproxy in front - roundcube feels like a desktop client Oh, and of course it also depends on Dovecot configuration :) Authentication cache is needed and login processes must be in high performance mode. There is still the extra work of forking a new imap process (could also be avoided with yet another config option) and some other extra CPU usage, but those shouldn't cause much of a difference. The extra network latency during login is a good point though.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Quoting Timo Sirainen t...@iki.fi: On Thu, 2012-06-21 at 13:05 -0700, email builder wrote: Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) IMP is caching (message/mailbox/folder listing), with full QRESYNC/CONDSTORE support. michael
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 21/06/2012 21:54, Reindl Harald wrote: and last but not least i have lesser entries in maillog which goes to a central mysql-server for self-developed web-interfaces I recently added imapproxy to my Roundcube installation. Benchmarks showed a very slight slowdown, but as you point out it reduced the login count from dovecot and I use a login script to kind of report last login / length of session and this tallies better with an imap desktop user now I think the conclusion is that imapproxy is not necessary. There are some advantages (eg with high network latency between web and imap server, and reducing apparent login count), and some disadvantages (extra complexity, slowdown) On average I think few users should use it.. Or at least benchmark and add it reluctantly... Ed
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 21/06/2012 21:37, René Neumann wrote: Am 21.06.2012 22:22, schrieb Timo Sirainen: On Thu, 2012-06-21 at 13:05 -0700, email builder wrote: Do you know what webmails are caching vs. non-caching? Nearly all of them are non-caching. (I don't know of any caching ones.) At least roundcube (v0.7.1 here) has some caching options: --[excerpt from roundcubes main.inc.php]- // Type of IMAP indexes cache. Supported values: 'db', 'apc' and 'memcache'. $rcmail_config['imap_cache'] = null; // Enables messages cache. Only 'db' cache is supported. $rcmail_config['messages_cache'] = false; -[end] But I don't know, whether this is the sort of caching you are referring to. - René It is caching, but unless your mysql / memcache server is lower latency than your dovecot server, then the caching does very little. I tested it very briefly and it added a lot of latency to my results when adding a mysql cache. However, my setup has the mysql/dovecot/roundcube all on the same machine, so latency is minimal. Roughly I found that the amount of caching is absolutely massive, eg roughly subject headers, message ids and more for every message in every folder. This meant multiple seconds of latency on first login and then slight additional latency on every folder view. I guess this might breakeven in the situation of a roundcube installation in an office and dovecot on the far end of an ADSL line with 60-100ms+ of latency and bandwidth constraints, but it's really, really hard to see it's sensible for two machines in the same datacenter with an uncontended network connection between them This isn't to say that the caching isn't sensible for use with other mail servers, but I don't see it offers any benefit for most Dovecot installations? However, very clever and full featured webmail client! Ed W P.S. Sogo has a kind of caching in that it has a clientside javascript cache. Not what was meant, but for all practical purposes much more useful...
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Quoting Ed W li...@wildgooses.com: I think the conclusion is that imapproxy is not necessary. There are some advantages (eg with high network latency between web and imap server, and reducing apparent login count), and some disadvantages (extra complexity, slowdown) Not entirely true. See this thread: http://markmail.org/thread/z7ctwle2go6zafas Thread in short: imapproxy provides benefits for more MUAs that take advantage of the XIMAPPROXY feature (only IMP, AFAIK), and Timo is/was considering adding a similar state saving feature to Dovecot 2.2. michael
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 22.6.2012, at 0.58, Michael M Slusarz wrote: I think the conclusion is that imapproxy is not necessary. There are some advantages (eg with high network latency between web and imap server, and reducing apparent login count), and some disadvantages (extra complexity, slowdown) Not entirely true. See this thread: http://markmail.org/thread/z7ctwle2go6zafas Thread in short: imapproxy provides benefits for more MUAs that take advantage of the XIMAPPROXY feature (only IMP, AFAIK), and Timo is/was considering adding a similar state saving feature to Dovecot 2.2. Well, I had completely forgotten about it :) Reading my old mail: There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. Clients could restore their earlier state from days ago, as long as Dovecot still has the necessary .log records available (similar to how QRESYNC works). Yeah .. Perhaps something like: 1. if client issues LOGOUT XSTATE 2. And server sees that it can actually save all of the state (some things are a bit tricky, and probably not worth the trouble in initial implementation) 3. Then the server server sends * OK XSTATE string * BYE 4. The client can pipeline after LOGIN/AUTHENTICATE: a XSTATERESTORE string a OK Yeah! or a NO Not gonna work. Perhaps even a real RFC for this thing? .. If it's worth it.. Would save at least a few X bytes from network traffic :)
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Oh, and of course it also depends on Dovecot configuration :) Authentication cache is needed and login processes must be in high performance mode. I.e., I think: http://wiki2.dovecot.org/LoginProcess http://wiki2.dovecot.org/Authentication/Caching There is still the extra work of forking a new imap process (could also be avoided with yet another config option) Are you referring to client_limit or service_count or something else as yet undeveloped? Speaking of which, I cannot understand the different between those two. Hints in the configuration file (10-master.conf) and the wiki make them sound like they do the same thing -- ??
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
On 22.6.2012, at 5.28, email builder wrote:
Oh, and of course it also depends on Dovecot configuration :) Authentication
cache is needed and login processes must be in high performance mode.
I.e., I think:
http://wiki2.dovecot.org/LoginProcess
http://wiki2.dovecot.org/Authentication/Caching
Yes.
There is
still the extra work of forking a new imap process (could also be avoided
with
yet another config option)
Are you referring to client_limit or service_count or something else as yet
undeveloped?
service imap { service_count = 0 } (default=1) allows imap processes to be
reused for more than 1 connection. The downside is that if there are any bugs
in Dovecot, they might accidentally expose another user's email data to the
wrong user. That's very unlikely to happen but since this isn't a performance
problem in most (if any) systems I don't want to enable it by default. Dovecot
code is written so that write buffer overflows (= arbitrary code execution) is
minimized to be as zero possibility as I could think of, but read buffer
overflows (= exposing data within the process) isn't treated nearly as much
with paranoia.
Speaking of which, I cannot understand the different between those two.
Hints in the
configuration file (10-master.conf) and the wiki make them sound like they do
the same
thing -- ??
service_count limits the maximum of client_limit. One connection = one service.
Once a process has serviced service_count number of connections it
disconnects itself. There can never be more than client_limit number of
simultaneous connections. The important stuff to understand about these are:
* service_count=1: The most secure setting for a process. The process serves a
single connection and kills itself. No possibility of data leaking to
unintended connection.
* service_count=0, client_limit=1: The process does blocking operations (e.g.
blocking disk IO). You don't want one connection's blocking operation to affect
other connections. But you're not paranoid about security, since in case of
some bugs some data might leak to unintended connection.
* service_count0: Restart process ever N connections, just in case it leaks
some memory.
* client_limit1: Limit the amount of CPU/memory a single process takes. The
process should never be blocking on disk I/O or locks or anything else. This
means it shouldn't be used for imap/pop3/lmtp processes. For CPU bound
processes it's fine.
Maybe these could be copypasted to the wiki2/Services.
Re: [Dovecot] Dovecot performance under high load (vs. Courier)
Oh, and of course it also depends on Dovecot configuration :)
Authentication
cache is needed and login processes must be in high performance mode.
I.e., I think:
http://wiki2.dovecot.org/LoginProcess
http://wiki2.dovecot.org/Authentication/Caching
Yes.
There is
still the extra work of forking a new imap process (could also be
avoided with
yet another config option)
Are you referring to client_limit or service_count or something else as yet
undeveloped?
service imap { service_count = 0 } (default=1) allows imap processes to be
reused for more than 1 connection. The downside is that if there are any bugs
in
Dovecot, they might accidentally expose another user's email data to the
wrong user. That's very unlikely to happen but since this isn't a
performance problem in most (if any) systems I don't want to enable it by
default. Dovecot code is written so that write buffer overflows (= arbitrary
code execution) is minimized to be as zero possibility as I could think of,
but
read buffer overflows (= exposing data within the process) isn't treated
nearly as much with paranoia.
Speaking of which, I cannot understand the different between those two.
Hints in the
configuration file (10-master.conf) and the wiki make them sound like they
do the same
thing -- ??
service_count limits the maximum of client_limit. One connection = one
service.
Once a process has serviced service_count number of connections it
disconnects itself. There can never be more than client_limit number
of simultaneous connections. The important stuff to understand about these
are:
* service_count=1: The most secure setting for a process. The process serves
a
single connection and kills itself. No possibility of data leaking to
unintended
connection.
* service_count=0, client_limit=1: The process does blocking operations (e.g.
blocking disk IO). You don't want one connection's blocking operation to
affect other connections. But you're not paranoid about security, since in
case of some bugs some data might leak to unintended connection.
* service_count0: Restart process ever N connections, just in case it leaks
some memory.
* client_limit1: Limit the amount of CPU/memory a single process takes. The
process should never be blocking on disk I/O or locks or anything else. This
means it shouldn't be used for imap/pop3/lmtp processes. For CPU bound
processes it's fine.
So really, a new process is created under *two* circumstances? 1. when a
process reaches client_limit number of *simultaneous* connections or 2. when
a process has serviced service_count number of connections. Is this correct?
So for service *-login, is it OK to do something like service_count=5000,
client_limit=2000
Thanks for the help!
