Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Thu, Feb 09, 2012 at 01:48:09AM +0200, Timo Sirainen wrote: On 7.2.2012, at 10.25, Jan-Frode Myklebust wrote: Feb 6 16:13:10 loadbalancer2 dovecot: lmtp(6601): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy-data_input-eof) .. Should I try increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS, or do you have any other ideas for what might be causing it ? The backend server didn't reply within LMTP_PROXY_DEFAULT_TIMEOUT_MSECS (30 secs). It's actually 60 sec in v2.0 http://hg.dovecot.org/dovecot-2.0/file/750db4b4c7d3/src/lmtp/lmtp-proxy.c#l13 It still shouldn't have crashed of course, and that crash is already fixed in v2.1 (in the LMTP simplification change). Do you think we should rather run v2.1-rc* on our dovecot directors (for IMAP, POP3 and LMTP), even if we keep the backend servers on v2.0 ? Anyway, you can fix this without recompiling by returning e.g. proxy_timeout=60 passdb extra field for 60 secs timeout. Thanks, well consider that option if it crashes too often... Have only seen this problem once for the last week. -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 9.2.2012, at 14.56, Jan-Frode Myklebust wrote: Should I try increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS, or do you have any other ideas for what might be causing it ? The backend server didn't reply within LMTP_PROXY_DEFAULT_TIMEOUT_MSECS (30 secs). It's actually 60 sec in v2.0 http://hg.dovecot.org/dovecot-2.0/file/750db4b4c7d3/src/lmtp/lmtp-proxy.c#l13 30. LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS is not LMTP_PROXY_DEFAULT_TIMEOUT_MSECS It still shouldn't have crashed of course, and that crash is already fixed in v2.1 (in the LMTP simplification change). Do you think we should rather run v2.1-rc* on our dovecot directors (for IMAP, POP3 and LMTP), even if we keep the backend servers on v2.0 ? Yes, I've done a lot of improvements to proxying and error handling/logging in v2.1. Also I'm planning on finishing my email backlog soon and making the last v2.1-rc before renaming it to v2.1.0.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 7.2.2012, at 10.25, Jan-Frode Myklebust wrote: Feb 6 16:13:10 loadbalancer2 dovecot: lmtp(6601): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy-data_input-eof) .. Should I try increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS, or do you have any other ideas for what might be causing it ? The backend server didn't reply within LMTP_PROXY_DEFAULT_TIMEOUT_MSECS (30 secs). It still shouldn't have crashed of course, and that crash is already fixed in v2.1 (in the LMTP simplification change). Anyway, you can fix this without recompiling by returning e.g. proxy_timeout=60 passdb extra field for 60 secs timeout.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Mon, Feb 06, 2012 at 10:01:03PM +0100, Jan-Frode Myklebust wrote: Your fsyncs can run over 60 seconds? Hopefully not.. maybe just me being confused by the error message about lmtp_proxy_output_timeout. After adding http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c on friday, we haven't seen any problems so it looks like this problem is solved. Crap, saw 6 message might be sent more than once messages from postfix yesterday, all at the time of this crash on the director postfix/lmtp was talking with: Feb 6 16:13:10 loadbalancer2 dovecot: lmtp(6601): Panic: file lmtp-proxy.c: line 376 (lmtp_proxy_output_timeout): assertion failed: (proxy-data_input-eof) Feb 6 16:13:10 loadbalancer2 dovecot: lmtp(6601): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0 [0x2ab6f193d680] - /usr/lib64/dovecot/libdovecot.so.0 [0x2ab6f193d6d6] - /usr/lib64/dovecot/libdovecot.so.0 [0x2ab6f193cb93] - dovecot/lmtp [0x406d75] - /usr/lib64/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xcd) [0x2ab6f194859d] - /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x68) [0x2ab6f1949558] - /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x2d) [0x2ab6f194820d] - /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x2ab6f1936a83] - dovecot/lmtp(main+0x144) [0x403fa4] - /lib64/libc.so.6(__libc_start_main+0xf4) [0x35f8a1d994] - dovecot/lmtp [0x403da9] Feb 6 16:13:10 loadbalancer2 dovecot: master: Error: service(lmtp): child 6601 killed with signal 6 (core dumps disabled) Should I try increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS, or do you have any other ideas for what might be causing it ? -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 3.2.2012, at 14.25, Jan-Frode Myklebust wrote: I now implemented this patch on our directors, and pointed postfix at them. No problem seen so far, but I'm still a bit uncertain about the LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS. I know we're experienceing quite large delays when fsync'ing (slow IMAP APPEND). Do you think increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS is a sensible workaround if we start seeing lmtp_proxy_output_timeout problems again ? Your fsyncs can run over 60 seconds? I think even if you increase Dovecot's timeout you'll soon reach your MTA's LMTP timeout.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Mon, Feb 06, 2012 at 10:29:03PM +0200, Timo Sirainen wrote: On 3.2.2012, at 14.25, Jan-Frode Myklebust wrote: I now implemented this patch on our directors, and pointed postfix at them. No problem seen so far, but I'm still a bit uncertain about the LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS. I know we're experienceing quite large delays when fsync'ing (slow IMAP APPEND). Do you think increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS is a sensible workaround if we start seeing lmtp_proxy_output_timeout problems again ? Your fsyncs can run over 60 seconds? Hopefully not.. maybe just me being confused by the error message about lmtp_proxy_output_timeout. After adding http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c on friday, we haven't seen any problems so it looks like this problem is solved. But it doesn't seem unthinkable that ext3 users might see more than 60s for fsyncs... Some stalls on the order of minutes have been reported ref: https://lwn.net/Articles/328363/ I think even if you increase Dovecot's timeout you'll soon reach your MTA's LMTP timeout. My MTA's default is 10 minutes.. http://www.postfix.org/postconf.5.html#lmtp_data_done_timeout -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 09:03:18PM +0200, Timo Sirainen wrote: I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends: http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26 888-8-8-88-888-- unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c I now implemented this patch on our directors, and pointed postfix at them. No problem seen so far, but I'm still a bit uncertain about the LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS. I know we're experienceing quite large delays when fsync'ing (slow IMAP APPEND). Do you think increasing LMTP_PROXY_DATA_INPUT_TIMEOUT_MSECS is a sensible workaround if we start seeing lmtp_proxy_output_timeout problems again ? -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 20.1.2012, at 4.27, Stan Hoeppner wrote: I spent months looking into NFS related issues. I read through Linux and FreeBSD kernel source codes to figure out if there's something I could do to avoid the problems I see. I sent some patches to try to improve things, which of course didn't get accepted (some alternative ways might have been, but it would have required much more work from my part). The mail_nfs_* settings are the result of what I found out. They don't fully work, so I gave up. Yeah, I recall some of your posts from that time, and your frustration. If an NFS config option existed to simply turn off the NFS client caching, would that resolve most/all of the remaining issues? Or is the problem more complex than just the file caching? I ask as it would seem creating such a Boolean NFS config option should be simple to implement. If the devs could be convinced of the need for it. It would work, but the performance would suck. There are several huge Dovecot+NFS setups. They use director. It works well enough (and with the recent fixes, I'd hope perfectly). Are any of these huge setups using mdbox? Or does it make a difference? I think they're all Maildirs currently, but it shouldn't make a difference. The index files are the ones most easily corrupted, so if they work then everything else should work just as well. In those director setups there have been no index corruption errors. I.e. Indexes are indexes whether they be maildir or mdbox. Would Director alone allow the OP to avoid the cache corruption issues discussed in this thread? Or would there still be problems due to the split LDA setup? By using LMTP proxying with director there wouldn't be any problems. Or using director for IMAP/POP3 and not using dovecot-lda for mail deliveries would work too. In this case the OP has Netapp storage. Netapp units support both NFS exports as well as iSCSI LUNs. If the OP could utilize iSCSI instead of NFS, switching to GFS2 or OCFS, do you see these cluster filesystem as preferable for mdbox? I don't have personal experience with cluster filesystems in recent years (other than glusterfs, which had some problems, but most(/all?) were fixed already or are available from their commercial support..). Based on what I've heard, I'm guessing they work better than random-access-NFS, but even if there are no actual corruption problems, it sounds like their performance isn't very good. So would an ideal long term solution to indexes in a cluster (NFS or clusterFS) environment be something like Dovecot's own index metadata broker daemon/lock manager that controls access to the files/indexes? Either a distributed token based architecture, or maybe something 'simple' such as a master node which all others send index updates to with the master performing the actual writes to the files, similar to a database architecture? The former likely being more difficult to implement, the latter having potential scalability and SPOF issues. Or is the percentage of Dovecot cluster deployments so small that it's difficult to justify the development investment for such a thing? I'm not sure if such daemons would be of much help. For best performance the user's mail access should be redirected to the same server in any case, and doing that solves all the other problems as well. I've a few other clustering plans besides a regular NFS based setup, but all of them rely on user normally being redirected to the same server (exception: split brain operation when mails are replicated to multiple data centers).
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 20.1.2012, at 9.43, Robert Schetterer wrote: i.e i think it should be poosible to design partitioning with ldap or sql to i.e split up heavy and big mailboxes in seperate storage partitions etc am i right here Timo ? You can use per-user home or mail_location that points to different storages. If you want only some folders in separate storages, you could use symlinks, but deleting such a folder probably wouldn't delete the mails (or at least not all files).
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 8:39 PM, Stan Hoeppner s...@hardwarefreak.com wrote:
On 1/18/2012 7:54 AM, Timo Sirainen wrote:
On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote:
* All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames)
* Postfix will feed new email to Dovecot via LMTP
* Dovecot servers have been split based on their role
- Dovecot LDA Servers (running LMTP protocol)
- Dovecot POP/IMAP servers (running POP/IMAP protocols)
You're going to run into NFS caching troubles with the above split
setup. I don't recommend it. You will see error messages about index
corruption with it, and with dbox it can cause metadata loss.
http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director
Would it be possible to fix this NFS mdbox index corruption issue in
this split scenario by using a dual namespace and disabling indexing on
the INBOX? The goal being no index file collisions between LDA and imap
processes. Maybe something like:
namespace {
separator = /
prefix = #mbox/
location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY
inbox = yes
hidden = yes
list = no
}
namespace {
separator = /
prefix =
location = mdbox:~/mdbox
}
Client access to new mail might be a little slower, but if it eliminates
the index corruption issue and allows the split architecture, it may be
a viable option.
--
Stan
It could be that I botched my test up somehow, but when I tested
something similar yesterday (pointing the index at another location on
the LDA), it didn't work. I was sending from the LDA server and
confirmed that the messages made it to storage/m.# but without the
real indexes being updated. When I checked the mailbox via IMAP, it
never seemed to register that there was a message there, so I'm
guessing that dovecot never looks at the storage files but just relies
on the indexes to be correct. That sound right, Timo?
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 19.1.2012, at 6.39, Stan Hoeppner wrote:
You're going to run into NFS caching troubles with the above split
setup. I don't recommend it. You will see error messages about index
corruption with it, and with dbox it can cause metadata loss.
http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director
Would it be possible to fix this NFS mdbox index corruption issue in
this split scenario by using a dual namespace and disabling indexing on
the INBOX? The goal being no index file collisions between LDA and imap
processes. Maybe something like:
namespace {
separator = /
prefix = #mbox/
location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY
inbox = yes
hidden = yes
list = no
}
namespace {
separator = /
prefix =
location = mdbox:~/mdbox
}
Client access to new mail might be a little slower, but if it eliminates
the index corruption issue and allows the split architecture, it may be
a viable option.
That assumes that mails are only being delivered to INBOX (i.e. no Sieve or
+mailbox addressing). I suppose you could do that if you can live with that
limitation. Slightly better for performance would be to not actually keep INBOX
mails in mbox format but use snarf plugin to move them to mdbox.
And of course the above still requires that for imap/pop3 access the user is
redirected to the same server every time. I don't really see it helping much.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 19.1.2012, at 19.08, Mark Moseley wrote:
namespace {
separator = /
prefix = #mbox/
location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY
inbox = yes
hidden = yes
list = no
}
Client access to new mail might be a little slower, but if it eliminates
the index corruption issue and allows the split architecture, it may be
a viable option.
--
Stan
It could be that I botched my test up somehow, but when I tested
something similar yesterday (pointing the index at another location on
the LDA), it didn't work.
Note that Stan used mbox format for INBOX, not mdbox.
I was sending from the LDA server and
confirmed that the messages made it to storage/m.# but without the
real indexes being updated. When I checked the mailbox via IMAP, it
never seemed to register that there was a message there, so I'm
guessing that dovecot never looks at the storage files but just relies
on the indexes to be correct. That sound right, Timo?
Correct. dbox absolutely relies on index files always being up to date. In some
error situations it can figure out that it should do an index rebuild and then
it finds any missing mails, but in normal situations it doesn't even try,
because that would unnecessarily waste disk IO. (And there's of course doveadm
force-resync to force it.)
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 1/19/2012 1:18 PM, Timo Sirainen wrote:
On 19.1.2012, at 6.39, Stan Hoeppner wrote:
You're going to run into NFS caching troubles with the above split
setup. I don't recommend it. You will see error messages about index
corruption with it, and with dbox it can cause metadata loss.
http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director
Would it be possible to fix this NFS mdbox index corruption issue in
this split scenario by using a dual namespace and disabling indexing on
the INBOX? The goal being no index file collisions between LDA and imap
processes. Maybe something like:
namespace {
separator = /
prefix = #mbox/
location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY
inbox = yes
hidden = yes
list = no
}
namespace {
separator = /
prefix =
location = mdbox:~/mdbox
}
Client access to new mail might be a little slower, but if it eliminates
the index corruption issue and allows the split architecture, it may be
a viable option.
That assumes that mails are only being delivered to INBOX (i.e. no Sieve or
+mailbox addressing). I suppose you could do that if you can live with that
limitation. Slightly better for performance would be to not actually keep
INBOX mails in mbox format but use snarf plugin to move them to mdbox.
And of course the above still requires that for imap/pop3 access the user is
redirected to the same server every time. I don't really see it helping much.
I spent a decent amount of time last night researching the NFS cache
issue. It seems there is no way to completely disable NFS client
caching (in lie of rewriting the code oneself--a daunting tak), which
would seem to be the real solution to the mdbox index corruption problem.
So I went looking for alternatives and came up with the idea above.
Obviously it's far from an optimal solution and introduces some
limitations, but I thought it was worth tossing out for discussion.
Timo, it seems that when you designed mdbox you didn't have NFS based
clusters in mind. Do you consider mdbox simply not suitable for such an
NFS cluster deployment? If one has no choice but an NFS cluster
architecture, what Dovecot mailbox format do you recommend? Stick with
maildir?
In this case the OP has Netapp storage. Netapp units support both NFS
exports as well as iSCSI LUNs. If the OP could utilize iSCSI instead of
NFS, switching to GFS2 or OCFS, do you see these cluster filesystem as
preferable for mdbox?
--
Stan
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 20.1.2012, at 1.51, Stan Hoeppner wrote: I spent a decent amount of time last night researching the NFS cache issue. It seems there is no way to completely disable NFS client caching (in lie of rewriting the code oneself--a daunting tak), which would seem to be the real solution to the mdbox index corruption problem. So I went looking for alternatives and came up with the idea above. Obviously it's far from an optimal solution and introduces some limitations, but I thought it was worth tossing out for discussion. I spent months looking into NFS related issues. I read through Linux and FreeBSD kernel source codes to figure out if there's something I could do to avoid the problems I see. I sent some patches to try to improve things, which of course didn't get accepted (some alternative ways might have been, but it would have required much more work from my part). The mail_nfs_* settings are the result of what I found out. They don't fully work, so I gave up. Timo, it seems that when you designed mdbox you didn't have NFS based clusters in mind. Do you consider mdbox simply not suitable for such an NFS cluster deployment? If one has no choice but an NFS cluster architecture, what Dovecot mailbox format do you recommend? Stick with maildir? In the typical random-access NFS setup I don't consider any of Dovecot's formats suitable. Not maildir, not dbox. Perhaps in future I can redesign everything in a way that just happens to work well with all kinds of NFS setups, but I don't really hold a lot of hope for that. It seems that either you'll get bad performance (I'm not really interested in making Dovecot do that) or you'll use such a setup where you get good performance by avoiding the NFS problems. There are several huge Dovecot+NFS setups. They use director. It works well enough (and with the recent fixes, I'd hope perfectly). In this case the OP has Netapp storage. Netapp units support both NFS exports as well as iSCSI LUNs. If the OP could utilize iSCSI instead of NFS, switching to GFS2 or OCFS, do you see these cluster filesystem as preferable for mdbox? I don't have personal experience with cluster filesystems in recent years (other than glusterfs, which had some problems, but most(/all?) were fixed already or are available from their commercial support..). Based on what I've heard, I'm guessing they work better than random-access-NFS, but even if there are no actual corruption problems, it sounds like their performance isn't very good.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Fri, 2012-01-20 at 02:13 +0200, Timo Sirainen wrote: There are several huge Dovecot+NFS setups. They use director. It works well enough (and with the recent fixes, I'd hope perfectly). Not to mention other huge NFS setups that don't use director, and also have no problems. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 1/19/2012 6:13 PM, Timo Sirainen wrote: On 20.1.2012, at 1.51, Stan Hoeppner wrote: I spent a decent amount of time last night researching the NFS cache issue. It seems there is no way to completely disable NFS client caching (in lie of rewriting the code oneself--a daunting tak), which would seem to be the real solution to the mdbox index corruption problem. So I went looking for alternatives and came up with the idea above. Obviously it's far from an optimal solution and introduces some limitations, but I thought it was worth tossing out for discussion. I spent months looking into NFS related issues. I read through Linux and FreeBSD kernel source codes to figure out if there's something I could do to avoid the problems I see. I sent some patches to try to improve things, which of course didn't get accepted (some alternative ways might have been, but it would have required much more work from my part). The mail_nfs_* settings are the result of what I found out. They don't fully work, so I gave up. Yeah, I recall some of your posts from that time, and your frustration. If an NFS config option existed to simply turn off the NFS client caching, would that resolve most/all of the remaining issues? Or is the problem more complex than just the file caching? I ask as it would seem creating such a Boolean NFS config option should be simple to implement. If the devs could be convinced of the need for it. Timo, it seems that when you designed mdbox you didn't have NFS based clusters in mind. Do you consider mdbox simply not suitable for such an NFS cluster deployment? If one has no choice but an NFS cluster architecture, what Dovecot mailbox format do you recommend? Stick with maildir? In the typical random-access NFS setup I don't consider any of Dovecot's formats suitable. Not maildir, not dbox. Perhaps in future I can redesign everything in a way that just happens to work well with all kinds of NFS setups, but I don't really hold a lot of hope for that. It seems that either you'll get bad performance (I'm not really interested in making Dovecot do that) or you'll use such a setup where you get good performance by avoiding the NFS problems. There are several huge Dovecot+NFS setups. They use director. It works well enough (and with the recent fixes, I'd hope perfectly). Are any of these huge setups using mdbox? Or does it make a difference? I.e. Indexes are indexes whether they be maildir or mdbox. Would Director alone allow the OP to avoid the cache corruption issues discussed in this thread? Or would there still be problems due to the split LDA setup? In this case the OP has Netapp storage. Netapp units support both NFS exports as well as iSCSI LUNs. If the OP could utilize iSCSI instead of NFS, switching to GFS2 or OCFS, do you see these cluster filesystem as preferable for mdbox? I don't have personal experience with cluster filesystems in recent years (other than glusterfs, which had some problems, but most(/all?) were fixed already or are available from their commercial support..). Based on what I've heard, I'm guessing they work better than random-access-NFS, but even if there are no actual corruption problems, it sounds like their performance isn't very good. So would an ideal long term solution to indexes in a cluster (NFS or clusterFS) environment be something like Dovecot's own index metadata broker daemon/lock manager that controls access to the files/indexes? Either a distributed token based architecture, or maybe something 'simple' such as a master node which all others send index updates to with the master performing the actual writes to the files, similar to a database architecture? The former likely being more difficult to implement, the latter having potential scalability and SPOF issues. Or is the percentage of Dovecot cluster deployments so small that it's difficult to justify the development investment for such a thing? Thanks Timo. -- Stan
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Am 20.01.2012 01:13, schrieb Timo Sirainen: On 20.1.2012, at 1.51, Stan Hoeppner wrote: I spent a decent amount of time last night researching the NFS cache issue. It seems there is no way to completely disable NFS client caching (in lie of rewriting the code oneself--a daunting tak), which would seem to be the real solution to the mdbox index corruption problem. So I went looking for alternatives and came up with the idea above. Obviously it's far from an optimal solution and introduces some limitations, but I thought it was worth tossing out for discussion. I spent months looking into NFS related issues. I read through Linux and FreeBSD kernel source codes to figure out if there's something I could do to avoid the problems I see. I sent some patches to try to improve things, which of course didn't get accepted (some alternative ways might have been, but it would have required much more work from my part). The mail_nfs_* settings are the result of what I found out. They don't fully work, so I gave up. Timo, it seems that when you designed mdbox you didn't have NFS based clusters in mind. Do you consider mdbox simply not suitable for such an NFS cluster deployment? If one has no choice but an NFS cluster architecture, what Dovecot mailbox format do you recommend? Stick with maildir? In the typical random-access NFS setup I don't consider any of Dovecot's formats suitable. Not maildir, not dbox. Perhaps in future I can redesign everything in a way that just happens to work well with all kinds of NFS setups, but I don't really hold a lot of hope for that. It seems that either you'll get bad performance (I'm not really interested in making Dovecot do that) or you'll use such a setup where you get good performance by avoiding the NFS problems. There are several huge Dovecot+NFS setups. They use director. It works well enough (and with the recent fixes, I'd hope perfectly). In this case the OP has Netapp storage. Netapp units support both NFS exports as well as iSCSI LUNs. If the OP could utilize iSCSI instead of NFS, switching to GFS2 or OCFS, do you see these cluster filesystem as preferable for mdbox? I don't have personal experience with cluster filesystems in recent years (other than glusterfs, which had some problems, but most(/all?) were fixed already or are available from their commercial support..). Based on what I've heard, I'm guessing they work better than random-access-NFS, but even if there are no actual corruption problems, it sounds like their performance isn't very good. for info i have 3500 users behind keepalived loadbalancers with drbd ocfs2 on two lucid servers, they are heavy penetrated by pop3 with maildir on dove2 , in the begin i had some performance problem but they were mostly related to the raid controlers io, so imap was very slow. Fixing this raid problems gave good imap performance now ( beside some dovecot and kernel tuneups ), anyway i would overthink this whole setup again going up to more users i.e i guess mixing loadbalancers and directors is no problem, maildir seems to be slow by io in design , so mdbox might better, and after all i would more investigate about drbd and compare gfs ocfs and other cluster filesystems better, i.e switching to iSCSI i.e i think it should be poosible to design partitioning with ldap or sql to i.e split up heavy and big mailboxes in seperate storage partitions etc am i right here Timo ? anyway i would like to test some cross hostingplace setup with i.e glusterfs lustre etc to get more knowledge as base of a multi redundant mailsystem -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
[Dovecot] Performance of Maildir vs sdbox/mdbox
Hi Guys, I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. This is a brand new build, with customer mailboxes to be migrated in over the course of 3-4 months. Some details on our new environment: * Approximately 1.6M+ mailboxes once all legacy systems are combined * NetApp FAS6280 storage w/ 120TB usable for mail storage, 1TB of FlashCache in each controller * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers * Apache Lucene indexes will be used to accelerate IMAP search for users Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? * What kind of IOPS/User reduction could we expect to see under mdbox? * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. Thanks!
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Am 18.01.2012 13:44, schrieb Lee Standen: Hi Guys, I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. This is a brand new build, with customer mailboxes to be migrated in over the course of 3-4 months. Some details on our new environment: * Approximately 1.6M+ mailboxes once all legacy systems are combined * NetApp FAS6280 storage w/ 120TB usable for mail storage, 1TB of FlashCache in each controller * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) nfs may not be optimal clusterfilesystem might better, but this is an heavy seperate discussion * Postfix will feed new email to Dovecot via LMTP perfect * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers * Apache Lucene indexes will be used to accelerate IMAP search for users sounds ok Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak wow thats big Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? * What kind of IOPS/User reduction could we expect to see under mdbox? there should be people on the list , knowing this , by migration done * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? as far i remember mdbox takes 8 mails per file ( i am not using it currently, so i didnt investigate it ), better wait for more qualified answer, anyway mdbox seems recommended in your case from our last plans about 25k mailboxes we decide using mdbox, as far i remember I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. Thanks! from my personal knowledge io on storage has most influance of performance, if at last ,all other setup parts are solved optimal wait a little bit , i guess more matching answers will come up after all ,you can hire someone, perhaps Timo, if you stuck in something -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Spanish edu site here, 80k users, 4,5 TB of email, 6.000 iops (indexes) + 9.000 iops (mdboxes) in working hours here. We evaluated mdbox against Maildir and we found that with these setting dovecot 2 perfoms better than Maildir: mdbox_rotate_interval = 1d mdbox_rotate_size=60m zlib_save_level = 9 # 1..9 zlib_save = gz # or bz2 We detected 40% less iops with this setup *in working hours (more info below)*. Zlib saved some writes (15-30%). With mdbox, deletion of a message is written to indexes (use SSD for this), and a nightly cronjob deletes the real message from the mdbox, this saves us some iops in working hours. Also, backup software is MUCH happier handling hundreds of thousands files (mdbox) versus tens of millions (maildir) Mdbox has also drawbacks: you have to be VERY careful with your indexes, they contain data that can not be rebuilt from mdboxes. The nightly cronjob purging the mdboxes hammers the SAN. Full backup time is reduced, but incremental backup space time increases: if you delete a message, after purging it from the mdbox the mdbox file changes (size and date), so the incremental backup has to copy it again. Regards Javier
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote: I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. Unfortunately there aren't really any. Everyone who seems to switch to sdbox/mdbox usually also change their hardware at the same time, so there aren't really any before/after metrics. I've of course some unrealistic synthetic benchmarks, but I don't think they are very useful. So, I would also be very interested in seeing some before/after graphs of disk IO, CPU and memory usage of Maildir - dbox switch in same hardware. Maildir is anyway definitely worse performance then sdbox or mdbox. mdbox also uses less NFS operations, but I don't know how much faster (if any) it is with Netapps. * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers Can the same mailbox still be accessed via multiple geographic servers? I've had some plans for doing this kind of access/replication using dsync.. * Apache Lucene indexes will be used to accelerate IMAP search for users Dovecot's fts-solr or fts-lucene? Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? If you have webmail type of clients, definitely. For Outlook/Thunderbird you should still see improvement, but not necessarily as much. You didn't mention POP3. That isn't Dovecot's strong point. Its performance should be about the same as Courier-POP3, but could be less than QMail-POP3. Although if many of your POP3 users keep a lot of mails on server it * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? Maildir renames files a lot. From new/ - to cur/ and then every time message flag changes. That's why sdbox is faster. Why mdbox should be faster than sdbox is because mdbox puts (or should put) more mail data physically closer in disks to make reading it faster. I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. It's at least safer to first switch to Dovecot+Maildir to make sure that any problems you might find aren't related to the mailbox format..
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.01.2012 21:54, Timo Sirainen wrote: On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote: I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. Unfortunately there aren't really any. Everyone who seems to switch to sdbox/mdbox usually also change their hardware at the same time, so there aren't really any before/after metrics. I've of course some unrealistic synthetic benchmarks, but I don't think they are very useful. So, I would also be very interested in seeing some before/after graphs of disk IO, CPU and memory usage of Maildir - dbox switch in same hardware. Maildir is anyway definitely worse performance then sdbox or mdbox. mdbox also uses less NFS operations, but I don't know how much faster (if any) it is with Netapps. We have bought new hardware for this project too, so we might not be able to help out massively on that front... we do have NFS operations monitored though so we should at least be able to compare that metric since the underlying storage operating system is the same. All NetApp hardware runs their Data ONTAP operating system, so the metrics are assured to be the same :) How about this... are there any tools available (that you know of) to capture real live customer POP3/IMAP traffic and replay it against a separate system? That might be a feasible option for doing a like-for-like comparison in our environment? We could probably get something in place to simulate the load if we can do something like that... * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one. The biggest challenge I have at the moment if I try to sell the dbox format is providing some kind of data on the expected gains from this. If it's only a 10% reduction in NFS operations for the typical user, then it's probably not worth our while. - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers Can the same mailbox still be accessed via multiple geographic servers? I've had some plans for doing this kind of access/replication using dsync.. No, we're using the nginx proxy layer to ensure that if a user in Sydney (for example) tries to access a Perth mailbox, their connection is redirected (by nginx) to the Perth POP/IMAP servers. Postfix configuration is handling the same thing on the LMTP side. The requirement here is for all users to have the same settings regardless of location, but still be able to locate the email servers and data close to the customer. * Apache Lucene indexes will be used to accelerate IMAP search for users Dovecot's fts-solr or fts-lucene? fts-solr. I've been using Lucene/Solr interchangeably when discussing this project with my peers :) Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? If you have webmail type of clients, definitely. For Outlook/Thunderbird you should still see improvement, but not necessarily as much. You didn't mention POP3. That isn't Dovecot's strong point. Its performance should be about the same as Courier-POP3, but could be less than QMail-POP3. Although if many of your POP3 users keep a lot of mails on server it Our existing systems run with about 21K concurrent IMAP connections at any one point in time, not counting Webmail POP3 runs at about 3600 concurrent connections, but since those are not long lived it's not particularly indicative of customer numbers. Vague recollection is something like 25% IMAP, 55-60% POP3, rest 20% Webmail. I'd have to go back and check the breakdown again. * If someone can give some
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Out of interest, has the NFS issue been tested on NFS4? My understanding is that NFS4 has a lot of fixes for the locking/caching problems that plague NFS3, and we were planning to use NFS4 from day one. If this hasn't been tested, is there some kind of load simulator that we could run to see if the issue does occur in our environment? On 18.01.2012 21:54, Timo Sirainen wrote: On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote: I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. Unfortunately there aren't really any. Everyone who seems to switch to sdbox/mdbox usually also change their hardware at the same time, so there aren't really any before/after metrics. I've of course some unrealistic synthetic benchmarks, but I don't think they are very useful. So, I would also be very interested in seeing some before/after graphs of disk IO, CPU and memory usage of Maildir - dbox switch in same hardware. Maildir is anyway definitely worse performance then sdbox or mdbox. mdbox also uses less NFS operations, but I don't know how much faster (if any) it is with Netapps. * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers Can the same mailbox still be accessed via multiple geographic servers? I've had some plans for doing this kind of access/replication using dsync.. * Apache Lucene indexes will be used to accelerate IMAP search for users Dovecot's fts-solr or fts-lucene? Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? If you have webmail type of clients, definitely. For Outlook/Thunderbird you should still see improvement, but not necessarily as much. You didn't mention POP3. That isn't Dovecot's strong point. Its performance should be about the same as Courier-POP3, but could be less than QMail-POP3. Although if many of your POP3 users keep a lot of mails on server it * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? Maildir renames files a lot. From new/ - to cur/ and then every time message flag changes. That's why sdbox is faster. Why mdbox should be faster than sdbox is because mdbox puts (or should put) more mail data physically closer in disks to make reading it faster. I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. It's at least safer to first switch to Dovecot+Maildir to make sure that any problems you might find aren't related to the mailbox format..
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, 2012-01-18 at 22:36 +0800, Lee Standen wrote: How about this... are there any tools available (that you know of) to capture real live customer POP3/IMAP traffic and replay it against a separate system? That might be a feasible option for doing a like-for-like comparison in our environment? We could probably get something in place to simulate the load if we can do something like that... I've thought about that too before, but with IMAP traffic it doesn't work very well. Even if the storages were 100% synchronized at startup, the session states could easily become desynced. For example if client does a NOOP at the same time when two mails are being delivered to the mailbox, serverA might show only one of them while serverB would show two of them because it was executed a tiny bit later. All of the client's future commands could then be affected by this desync. (OK, I wrote the above thinking about a real-time system where you could redirect the client's traffic to two systems, but basically same problems exist for offline replays too. Although it would be easier to fix the replays to handle this.) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. Are you planning on also redirecting POP3/IMAP connections to somewhat randomly to the different servers? I really don't recommend that, even with Maildir.. Some of the errors will be user visible, even if no actual data loss happens. Users may get disconnected, and sometimes might have to clean their client's cache. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one. What is a single point of failure there? It's at least safer to first switch to Dovecot+Maildir to make sure that any problems you might find aren't related to the mailbox format.. Yep, I'm considering that. The flip side is that it's actually going to be difficult for us to change mail format once we've migrated into this system, but we have an opportunity for (literally) a month long testing phase beginning in Feb/March which will let us test as many possibilities as we can. The mailbox format switching can be done one user at a time with zero downtime with dsync.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, 2012-01-18 at 23:21 +0800, Lee Standen wrote: Out of interest, has the NFS issue been tested on NFS4? My understanding is that NFS4 has a lot of fixes for the locking/caching problems that plague NFS3, and we were planning to use NFS4 from day one. I've tried with Linux NFS4 server+client a few years ago. It seemed to have all the same caching problems as NFS3. If this hasn't been tested, is there some kind of load simulator that we could run to see if the issue does occur in our environment? http://imapwiki.org/ImapTest should easily trigger it. Just run it against two servers, both hammering the same mailbox.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
snip * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one. /snip I'm in the middle of working on a Maildir-mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting mail_nfs_index = yes and mmap_disable = yes and mail_fsync = always/optimized would mitigate most of the risks of index corruption, as well as probably turning indexing off on the LDA side of things--i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 19.54, Mark Moseley wrote: I'm in the middle of working on a Maildir-mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting mail_nfs_index = yes and mmap_disable = yes and mail_fsync = always/optimized would mitigate most of the risks of index corruption, They help, but aren't 100% effective and they also make the performance worse. as well as probably turning indexing off on the LDA side of things You can't turn off indexing with dbox. --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture?
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 07:58:31PM +0200, Timo Sirainen wrote: --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture? It hasn't been working reliably for lmtp in v2.0. To quote yourself: 888-8-8-88-888-- I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends: http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26 888-8-8-88-888-- unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 20.51, Jan-Frode Myklebust wrote: On Wed, Jan 18, 2012 at 07:58:31PM +0200, Timo Sirainen wrote: --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture? It hasn't been working reliably for lmtp in v2.0. Yes, besides that :) To quote yourself: 888-8-8-88-888-- I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends: http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26 888-8-8-88-888-- unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 9:58 AM, Timo Sirainen t...@iki.fi wrote: On 18.1.2012, at 19.54, Mark Moseley wrote: I'm in the middle of working on a Maildir-mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting mail_nfs_index = yes and mmap_disable = yes and mail_fsync = always/optimized would mitigate most of the risks of index corruption, They help, but aren't 100% effective and they also make the performance worse. In testing, it seemed very much like the benefits of reducing IOPS by up to a couple orders of magnitude outweighed having to use those settings. Both in scripted testing and just using a mail UI, with the NFS-ish settings, I didn't notice any lag and doing things like checking a good-sized mailbox were at least as quick as Maildir. And I'm hoping that reducing IOPS across the entire set of NFS servers will compound the benefits quite a bit. as well as probably turning indexing off on the LDA side of things You can't turn off indexing with dbox. Ah, too bad. I was hoping I could get away with the LDA not updating the index but just dropping the message into storage/m.# but it'd still be seen on the IMAP/POP side--but hadn't tested that. Guess that's not the case. --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture? Nothing, per se. It's just that migrating to mdbox *and* to a director architecture is quite a bit more added complexity than simply migrating to mdbox alone. Hopefully, I'm not hijacking this thread. This seems pretty pertinent as well to the OP.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 09:03:18PM +0200, Timo Sirainen wrote: On 18.1.2012, at 20.51, Jan-Frode Myklebust wrote: What's the problem with director-based architecture? It hasn't been working reliably for lmtp in v2.0. Yes, besides that :) Besides that it's great! unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c So with that oneliner on our directors, you expect lmtp proxying trough director to be better than lmtp to rr-dns towards backend servers? If so, I guess we should give it another try. -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 22.14, Jan-Frode Myklebust wrote: unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c So with that oneliner on our directors, you expect lmtp proxying trough director to be better than lmtp to rr-dns towards backend servers? If so, I guess we should give it another try. It should fix the hangs that were common. I'm not sure if it fixes everything without the complexity reduction patch.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 21.49, Mark Moseley wrote: What's the problem with director-based architecture? Nothing, per se. It's just that migrating to mdbox *and* to a director architecture is quite a bit more added complexity than simply migrating to mdbox alone. Yes, I agree it's safer to do one thing that a time. That's why I'd do a switch to director first. :)
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 1/18/2012 7:54 AM, Timo Sirainen wrote:
On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote:
* All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames)
* Postfix will feed new email to Dovecot via LMTP
* Dovecot servers have been split based on their role
- Dovecot LDA Servers (running LMTP protocol)
- Dovecot POP/IMAP servers (running POP/IMAP protocols)
You're going to run into NFS caching troubles with the above split
setup. I don't recommend it. You will see error messages about index
corruption with it, and with dbox it can cause metadata loss.
http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director
Would it be possible to fix this NFS mdbox index corruption issue in
this split scenario by using a dual namespace and disabling indexing on
the INBOX? The goal being no index file collisions between LDA and imap
processes. Maybe something like:
namespace {
separator = /
prefix = #mbox/
location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY
inbox = yes
hidden = yes
list = no
}
namespace {
separator = /
prefix =
location = mdbox:~/mdbox
}
Client access to new mail might be a little slower, but if it eliminates
the index corruption issue and allows the split architecture, it may be
a viable option.
--
Stan
