Re: [Dovecot] dbox vs. mdbox
On Thu, Mar 29, 2012 at 12:30 AM, Timo Sirainen wrote: > > The main problem is that it's difficult to do any "real world" tests with > IMAP, especially when users are using many different kinds of IMAP clients. > So I'm very interested in hearing some numbers (and disk IO graphs for a > few weeks would be great) before your migration and after your migration, > but the numbers for your tests might not mean all that much. I was considering using the imaptest tool to simulate IMAP activity. I would keep the same machine configuration, only varying the mailbox format while running imaptest against each setup for a few hours/days. I'm now converting the original Maildir format to both dbox formats and I'll give it a try. I'll share some graphs afterwards.
Re: [Dovecot] dbox vs. mdbox
On 28.3.2012, at 13.13, Hangas wrote: > Timo Sirainen iki.fi> writes: > >>> 4. Are there real-world benchmarks showing measurable differences between >>> maildir, sdbox mdbox? >> >> Not that I'm aware of. So far everyone I've tried to ask have replaced their >> whole mail system and their storage, so the before/after numbers can't be >> compared. I'm very interested in knowing myself too. > > I think I can give my contribution here. I'm planning to migrate from > dovecot > 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice > at > the time, but now I'm trying to decide the mailbox format for a 2.x > fresh install. .. > My ideia is to install a fresh server and replicate the production maildir on > it > to build a test "source disk" that I'll use then to experiment the > conversions > to sdbox and mdbox. > I then plan to test the performance of the dbox formats, but I can include > Maildir measurements just for the record. > > I'm open to suggestions on how to test this properly The main problem is that it's difficult to do any "real world" tests with IMAP, especially when users are using many different kinds of IMAP clients. So I'm very interested in hearing some numbers (and disk IO graphs for a few weeks would be great) before your migration and after your migration, but the numbers for your tests might not mean all that much.
Re: [Dovecot] dbox vs. mdbox
Timo Sirainen iki.fi> writes: > > 4. Are there real-world benchmarks showing measurable differences between > >maildir, sdbox mdbox? > > Not that I'm aware of. So far everyone I've tried to ask have replaced their > whole mail system and their storage, so the before/after numbers can't be > compared. I'm very interested in knowing myself too. I think I can give my contribution here. I'm planning to migrate from dovecot 1.x to 2.x. Currently, on 1.x I'm using Maildir as this was my best choice at the time, but now I'm trying to decide the mailbox format for a 2.x fresh install. The environment will be virtually the same as this is running in a virtualized environment. I'm keeping the same storage and storage network, same host hardware and about the same VM specs. Its the data from about 100 users, sizing about 300GB in size spread over about 2 million files in Maildir format. So I think this could provide statistically relevant information. My ideia is to install a fresh server and replicate the production maildir on it to build a test "source disk" that I'll use then to experiment the conversions to sdbox and mdbox. I then plan to test the performance of the dbox formats, but I can include Maildir measurements just for the record. I'm open to suggestions on how to test this properly
Re: [Dovecot] dbox vs. mdbox
The only solution I found was to use IMAP protocol to read from sdbox and write as mdbox. Dsync was NOT a solution at all. Sorry about that. On Sun, 06 Mar 2011 23:58:35 +0200, Timo Sirainen wrote: > On Sun, 2011-03-06 at 15:46 -0500, Joan Moreau wrote: > >> No, as you know very well: I sent plenty of emails about this matter at that time on this mailng list, but nobody found any solutions, so, I gave up. Dsync is too buggy to convert sdbox to mdbox. > I think I fixed all the crashes you reported. The only two problems left were 1) the "Too many open files" error, which I'm pretty sure is caused by 2) m.* file sizes are too small. http://hg.dovecot.org/dovecot-2.0/rev/d1f30e375f40 [1] should fix 1), but I don't understand the cause of 2), since your config looked ok. Did you solve it somehow or are your m.* files still too small? Links: -- [1] http://hg.dovecot.org/dovecot-2.0/rev/d1f30e375f40
Re: [Dovecot] dbox vs. mdbox
On Sun, 2011-03-06 at 15:46 -0500, Joan Moreau wrote: > No, as you know very well: I sent plenty of emails about this matter at > that time on this mailng list, but nobody found any solutions, so, I > gave up. > Dsync is too buggy to convert sdbox to mdbox. I think I fixed all the crashes you reported. The only two problems left were 1) the "Too many open files" error, which I'm pretty sure is caused by 2) m.* file sizes are too small. http://hg.dovecot.org/dovecot-2.0/rev/d1f30e375f40 should fix 1), but I don't understand the cause of 2), since your config looked ok. Did you solve it somehow or are your m.* files still too small?
Re: [Dovecot] dbox vs. mdbox
* Sdbox is using far too much I/O on a busy server, I had to switch to mdbox * Converting from Maildir to s/mdbox is easy Are you saying sdbox uses more disk I/O than maildir? That's unexpected. No. I am saying that sbbox is not sustainable when having very large mailbox, IO becomes too high (even with high-end storage devices) erting from sdbox to mdbox has been a complete nightmare. I have never managed to make it completely, finally made it through imap protocol between 2 instance of dovecot. You better choose before sd or md, but not try to convert between the 2 was difficult about it? You should have been able to do it really easily with dsync No, as you know very well: I sent plenty of emails about this matter at that time on this mailng list, but nobody found any solutions, so, I gave up. Dsync is too buggy to convert sdbox to mdbox.
Re: [Dovecot] dbox vs. mdbox
On 5.3.2011, at 4.14, Joan Moreau wrote: > * Sdbox > is using far too much I/O on a busy server, I had to switch to mdbox .. > * Converting from Maildir to s/mdbox is easy Are you saying sdbox uses more disk I/O than maildir? That's unexpected. > * Converting > from sdbox to mdbox has been a complete nightmare. I have never managed > to make it completely, finally made it through imap protocol between 2 > instance of dovecot. You better choose before sd or md, but not try to > convert between the 2 What was difficult about it? You should have been able to do it really easily with dsync.
Re: [Dovecot] dbox vs. mdbox
Hi, Just giving my own experience: * I am using dovecot 2.0.9 (well, now 2.0.10 since today) in production without problems * Sdbox is using far too much I/O on a busy server, I had to switch to mdbox * Mdbox is running well so far, and resources (IO or CPU) are not an issue anymore. * Converting from Maildir to s/mdbox is easy * Converting from sdbox to mdbox has been a complete nightmare. I have never managed to make it completely, finally made it through imap protocol between 2 instance of dovecot. You better choose before sd or md, but not try to convert between the 2 my 2 cents JM On Fri, 4 Mar 2011 23:19:21 +0200, Timo Sirainen wrote: > On 4.3.2011, at 23.05, Douglas Mortensen wrote: > >> I guess to get more specific, some of the questions I have regarding dbox vs. mdbox are: 1. What is the advantage to using multiple files? > mdbox in theory uses less disk I/O for "normal users". > >> 2. What is the advantage to using a single sdbox file for each user? > It's simpler. More difficult to get corrupted. Also if in future there exists a filesystem that supports smaller files better, it's then faster than mdbox. Probably unlikely that it will happen anytime soon. 3. > >> an of course be anything. > e="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">4. Are there real-world benchmarks showing measurable differences between maildir, sdbox, mdbox? torage, so the before/after numbers can't be compared. I'm very interested in knowing myself too. > e type="cite" style="padding-left:5px; border-left:#1010ff 2px solid; margin-left:5px; width:100%">5. Are sdbox & mdbox equally stable to Maildir? Are they recommended for production systems? sdbox is so simple that I doubt anyone will find any kind of corruption bugs. mdbox is more complex, but people are using > >> dles already corrupted files, v2.0.10 had several fixes related to that.
Re: [Dovecot] dbox vs. mdbox
On 4.3.2011, at 23.05, Douglas Mortensen wrote: > I guess to get more specific, some of the questions I have regarding dbox vs. > mdbox are: > > 1. What is the advantage to using multiple files? mdbox in theory uses less disk I/O for "normal users". > 2. What is the advantage to using a single sdbox file for each user? It's simpler. More difficult to get corrupted. Also if in future there exists a filesystem that supports smaller files better, it's then faster than mdbox. Probably unlikely that it will happen anytime soon. > 3. Is this a binary format, or txt (UTF?)? dbox headers/metadata is ASCII. The message bodies can of course be anything. > 4. Are there real-world benchmarks showing measurable differences between > maildir, sdbox, mdbox? Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too. > 5. Are sdbox & mdbox equally stable to Maildir? Are they recommended for > production systems? sdbox is so simple that I doubt anyone will find any kind of corruption bugs. mdbox is more complex, but people are using it in production and I haven't heard of any problems recently. Although there have been bugs in how mdbox handles already corrupted files, v2.0.10 had several fixes related to that.
Re: [Dovecot] dbox vs. mdbox
Thanks for the input Ed. The evening before you sent the message, I actually had decided to do just what you recommended (stick with what we know). We have since put the server into production, and things are working well. We built to a physical box, rather than a VM. I do like xen VMs myself. We didn't do the proxy, but I appreciate the tip. I actually read the wiki before posting. I guess to get more specific, some of the questions I have regarding dbox vs. mdbox are: 1. What is the advantage to using multiple files? 2. What is the advantage to using a single sdbox file for each user? 3. Is this a binary format, or txt (UTF?)? 4. Are there real-world benchmarks showing measurable differences between maildir, sdbox, mdbox? 5. Are sdbox & mdbox equally stable to Maildir? Are they recommended for production systems? This is the kind of info that I'm after. I didn't really see that in the wiki. In the wiki, I only saw that the different formats exist, and basics on how they work. Any feedback here would be appreciated. Thanks, - Doug Mortensen Network Consultant Impala Networks P: 505.327.7300
Re: [Dovecot] dbox vs. mdbox
Hi I really appreciate feedback. We're on a time-crunch to migrate from a debian 5 box w/ dovecot 1.1 to a debian 6 box w/ dovecot 2.0.9 (built from source). I would have thought if time is tight then stick with what you know and migrate later? If you aren't building your new box as a virtualised container of some sort then stop and consider that to be much more desirable... With a virtualised install you can easily spin up an alternative config and more easily migrate the format of the data separately to the config of the server? Oh, I'm also a huge fan of the "proxy" feature which allows you to spin up a new machine/config, then migrate users across slowly rather than one big bang. You can use it with your current migration to test the upgrade for example? According to the wiki, your answer to the dbox question is probably best given here: http://wiki2.dovecot.org/MailboxFormat/dbox Good luck Ed W
[Dovecot] dbox vs. mdbox
What are the pros and cons of both? Especially in regards to performance, stability, management & maintenance? I really appreciate feedback. We're on a time-crunch to migrate from a debian 5 box w/ dovecot 1.1 to a debian 6 box w/ dovecot 2.0.9 (built from source). Thanks, - Doug Mortensen Network Consultant Impala Networks Inc CCNA, MCSA, Security+, A+ Linux+, Network+, Server+ . www.impalanetworks.com P: (505) 327-7300 F: (505) 327-7545
