Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

Uh..

On 13.11.2012, at 1.02, Timo Sirainen wrote:

> On 13.11.2012, at 0.44, Robin wrote:
> 
>> On 11/11/2012 5:26 PM, Christoph Anton Mitterer wrote:
>>> Have you made systematic tests? I.e. compared times for all of these
>>> with those from the different dovecot backends.
>> 
>> The choice of Dovecot backends made no substantial difference.  I used 
>> maildir, sdbox, and mdbox.  I also added SiS (with mdbox).  Initial tests 
>> were on local multi-spindle RAID5 storage,
> 
> With local disks the tests often measure only the local RAM/CPU speed, unless 
> you're testing thousands of users.

..measuring disk I/O most importantly.

>> but to handicap Dovecot, I pushed it over NFS (also Linux 3.2 on a local 
>> GigE segment).  It wasn't slow enough to make dbmail competitive, even 
>> though you have to start turning off performance optimisation features in 
>> Dovecot to avoid NFS bugs.
> 
> NFS makes a better test case if you're measuring single user performance. 
> Much of it is probably due to the index file access latency, although not 
> all. In some cases Dovecot's prefetching mails can help (maildir, sdbox 
> backends with local disks currently, nothing preventing it from working in 
> other use cases though, even with Dovecot-SQL backend).


Prefetching is done only with mail_prefetch_count setting. Someone in 
blog.dovecot.org mentioned that it was bad for performance with local 
disk+maildir. Linux apparently doesn't do this with NFS. It would of course be 
possible to just have the prefetching create a new thread/process to download 
the mail locally and read it (similar to what the object storage plugin does).

Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

On 13.11.2012, at 0.44, Robin wrote:

> On 11/11/2012 5:26 PM, Christoph Anton Mitterer wrote:
>> Have you made systematic tests? I.e. compared times for all of these
>> with those from the different dovecot backends.
> 
> The choice of Dovecot backends made no substantial difference.  I used 
> maildir, sdbox, and mdbox.  I also added SiS (with mdbox).  Initial tests 
> were on local multi-spindle RAID5 storage,

With local disks the tests often measure only the local RAM/CPU speed, unless 
you're testing thousands of users.

> but to handicap Dovecot, I pushed it over NFS (also Linux 3.2 on a local GigE 
> segment).  It wasn't slow enough to make dbmail competitive, even though you 
> have to start turning off performance optimisation features in Dovecot to 
> avoid NFS bugs.

NFS makes a better test case if you're measuring single user performance. Much 
of it is probably due to the index file access latency, although not all. In 
some cases Dovecot's prefetching mails can help (maildir, sdbox backends with 
local disks currently, nothing preventing it from working in other use cases 
though, even with Dovecot-SQL backend).

>> I guess you’ve "only" tried dbmail?
> 
> I did try Manitou, but the lack of a proper IMAP service for it made 
> extensive "like for like" testing very difficult.  Manitou is still in the 
> very early days, alas.  It also relies on the SQL DB's underlying 
> authentication systems which is rather ... alarming.  It performs quite a bit 
> better than dbmail, but still it's not close to Dovecot.  At the time I 
> tested it, only custom-rolled clients could talk to it, i.e., no imap4/pop3 
> "gateways" to it.

Manitou seems to advertise itself as being email client .. although then also 
seems to say SQL is faster than IMAP (which doesn't make much sense itself).

> I think I was most alarmed to see that the widely assumed benefits of putting 
> mail on a SQL DB, i.e., fast searching/sorting, didn't actually happen in 
> reality.

SQL has nothing that makes any type of email access even potentially efficient. 
SQL indexes are mostly about binary trees, and there are about zero things in 
IMAP where I have thought of binary tree being even potentially useful. (Okay, 
potentially for expunging old mails when you have >1M mails in one folder. Not 
something you normally optimize for.)

With most of Dovecot's optimized lookups, latency is the most important thing. 
SQL is bad for latency. With remote systems it's usually much faster to just 
download 1 MB blob and parse it than fetch a couple of 100 byte blocks.

> As others have mentioned, I also shudder to think of backup/restore issues, 
> especially on a single user level.  The mechanisms of backing up and 
> restoring maildirs and even mdboxes, i.e., simple files, are not only well 
> understood, the failure modes are generally fully recoverable.  SQL-DB file 
> blobs, especially with MySQL, remind me too much of the "PST Hell" that 
> Exchange administrators face.  But maybe that's just my ignorance talking.

I'd think everyone would use the human-readable SQL dumps for database backups. 
At least with MySQL/PostgreSQL I wouldn't really trust anything else.

Re: [Dovecot] mbox vs. maildir storage block waste

[Robin]

On 11/11/2012 5:26 PM, Christoph Anton Mitterer wrote:
> Have you made systematic tests? I.e. compared times for all of these
> with those from the different dovecot backends.

The choice of Dovecot backends made no substantial difference.  I used maildir, 
sdbox, and mdbox.  I also added SiS (with mdbox).  Initial tests were on local 
multi-spindle RAID5 storage, but to handicap Dovecot, I pushed it over NFS 
(also Linux 3.2 on a local GigE segment).  It wasn't slow enough to make dbmail 
competitive, even though you have to start turning off performance optimisation 
features in Dovecot to avoid NFS bugs.

>> There wasn't a task that the dbmail setup performed faster than
>> Dovecot, in either low or high load situations.
> Which backend did you use?

Backend for dbmail?  Two MySQL versions (5.0 and 5.5) - InnoDB is required for 
dbmail, by the way.  Postgres 8.4 and 9.1 backends, using its default storage 
engine.  I tried the tests with both a separate DB machine, as well as a 
cohosted one with the dbmail connector using local sockets instead of TCP/IP, 
but that didn't significantly alter the performance.

I've found my first notes from the tests.  It was the second round of tests 
with the latest MySQL 5.0 server given some tuning to more aggressively use 
system memory.  You will note the puny size of the mail folder hive in this 
round.

> The mysqld process has consumed nearly an hour of CPU time during this 
> process.
> dbmail is configured to use local sockets rather than network I/O.
> 
> I'm using the PERL MailTools http://search.cpan.org/dist/MailTools/
> to import about 10 folders' worth of email, totaling about 560MB in raw size, 
> constituting about 23,000 emails.  The script basically creates the folders, 
> and does an APPEND for each email.  It's bog simple.
> 
> I DROP the database, recreated it, added the one user, verify DBMail 
> accepts authentication for the newly created mailbox, and then do the import.
> The MySQL files live on a freshly formatted ext4 filesystem.
> 
> The import takes Dovecot (MailDir or mdbox format), or Panda IMAP (mix) 
> about six minutes to complete.
> 
> DBMail 3 took 4h 23m.  Casual inspection of the system showed modestly 
> high CPU usage in mysqld and dbmail-imapd (as well as the import perl 
> command on occasion), but the Load Average didn't get too close to 1.0,
> let alone 2.0, which concerns me that I might have hit some kind of 
> "busy wait" pathology. 

To clarify the above:  To streamline iterative testing, I made a script to 
deactivate the currently running SQL server, unmount, re-format, re-mount, and 
re-populate the skeletal DB directories and restart the DB engine.  So between 
each test, no matter the imapd or DB back-end, the mailstore was presented with 
a freshly formatted volume on dedicated spindles.  The filesystem was ext4, 
formatted with:

lazy_itable_init=0,lazy_journal_init=0,dir_index=1,extents=1,uninit_bg=0,flex_bg=0,has_journal=0,inode_size=256,dir_index=1,

> Do you have detailed numbers?

Not really, but after it was clear that I wasn't going to get comparable 
performance even within the same magnitude, I stopped testing it.  I included 
the IMAP SEARCH performance comparison against fts_squat in my original mail to 
this list.  In addition to huge performance deficiencies, it also has/had fatal 
operational bugs.

> I guess you’ve "only" tried dbmail?

I did try Manitou, but the lack of a proper IMAP service for it made extensive 
"like for like" testing very difficult.  Manitou is still in the very early 
days, alas.  It also relies on the SQL DB's underlying authentication systems 
which is rather ... alarming.  It performs quite a bit better than dbmail, but 
still it's not close to Dovecot.  At the time I tested it, only custom-rolled 
clients could talk to it, i.e., no imap4/pop3 "gateways" to it.

I think I was most alarmed to see that the widely assumed benefits of putting 
mail on a SQL DB, i.e., fast searching/sorting, didn't actually happen in 
reality.

As others have mentioned, I also shudder to think of backup/restore issues, 
especially on a single user level.  The mechanisms of backing up and restoring 
maildirs and even mdboxes, i.e., simple files, are not only well understood, 
the failure modes are generally fully recoverable.  SQL-DB file blobs, 
especially with MySQL, remind me too much of the "PST Hell" that Exchange 
administrators face.  But maybe that's just my ignorance talking.

> All something I wouldn’t want to do on my production systems ;)

Neither would I.  But as I said, I was "desperate" to get this close to 
Dovecot's performance.  I had about 2-3 weeks to pre-qualify mail storage 
back-ends with an eye towards 4 or 5 digits of usercount, and maybe tens to 
hundreds of TBs' scale of mail storage.  Running across such poor performance 
with such relatively small loads disqualified the DB-based mail products very 
very quickly, for ME, anyway.

If you want to run your own tests, my s

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Thu, 2012-11-08 at 17:54 -0800, Robin wrote:
> The performance is surprisingly bad ... doing almost everything. 
>  Searches through IMAP, bulk importation of mail folders, large
> numbers of simultaneous mail deliveries, you name it.
Have you made systematic tests? I.e. compared times for all of these
with those from the different dovecot backends.


> There wasn't a task that the dbmail setup performed faster than
> Dovecot, in either low or high load situations.
Which backend did you use?



> When pressed on this lack of performance, I was instructed to "add
> more RAM" to the DB machine, and that for ideal performance I should
> have more RAM than my mailbox sizes.  *sigh*  This sounds great for a
> very small installation, but this clearly is not something that
> scales.
Yeah... that’s truly disappointing...

Do you have detailed numbers?

I guess you’ve "only" tried dbmail?




> The dbmail folk are earnest and hard-working, and I don't mean to cast
> the slightest bit of negativity on their project.  I think the
> assumptions about what SQL servers can do well often doesn't square
> with the reality of many applications that people try to fit them
> into.
hmm... 



> remove filesystem journaling, write barriers, etc on the mail db
> mountpoint.
All something I wouldn’t want to do on my production systems ;)




Thanks for your detailed information :)


Cheers,
Chris.


smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Daniel Parthey]

Christoph Anton Mitterer wrote:
> On Wed, 2012-11-07 at 17:30 +0200, Timo Sirainen wrote:
> > On 30.10.2012, at 2.16, Christoph Anton Mitterer wrote:
> > > Have you ever thought about adding a "real" DB backend? Nothing against
> > > dbox... :) ... and I have no performance comparison of dbox with what
> > > could be done with a DBMS... but the advantage of the later would be
> > > that you get all fancy features from database systems for free... like
> > > fast indexing, online replication, etc. p..
> > > One might even reuse something like AOX for this.
> 
> > SQL indexes aren't very helpful for IMAP-like data. It would be fun to
> > some day have SQL backend in Dovecot (there already is read-only
> > INBOX-only SQL backend), but I don't expect it to have very good
> > performance.
> I see... well I haven't tested AOX or dbmail so far (especially as
> they're not in Debian and I was too lazy till now to compile them)...

Bad performance experiences with dbmail 2.x were the main reason why
we migrated to dovecot. If you've got a MySQL database with 80 GB of
binary chunks then things are getting ugly, especially when it comes to
efficient backup and restore of whole mailboxes or single e-mails.
The SQL backend (and the IMAP user experience) becomes very slow
if the database does not fit completely into RAM.

There are many performance improvements and bug fixes in dbmail 3.x,
but instead of evaluating then, we decided to migrate to Dovecot.

One should think twice, or even three times about how to design an
efficient SQL backend for a good user experience.

Regards
Daniel
-- 
https://plus.google.com/103021802792276734820

Re: [Dovecot] mbox vs. maildir storage block waste

[Robert Schetterer]

Am 09.11.2012 02:54, schrieb Robin:
> I'll stay tuned, whether we ever see a fully usable SQL backend for
>> Dovecot :)

thats not a new idea, but there is still tons of stuff which has to
coded in more prime, as dovecot works nice with other existing storage
file backends, there isnt hard pressure for sql storage, but feel free
to code your own , youre welcome


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Re: [Dovecot] mbox vs. maildir storage block waste

[Robin]

Obvious caveats and qualifications apply here throughout this email.

Christoph Anton Mitterer  wrote:
> I see... well I haven't tested AOX or dbmail so far (especially as
> they're not in Debian and I was too lazy till now to compile them)...
> 
> At least I had the impression that performance (especially in searches)
> was one of the major things these people were proud of.
> 
> 
> I'll stay tuned, whether we ever see a fully usable SQL backend for
> Dovecot :)

I wouldn't hold your breath.

It's a recurringly seductive "meme" in email circles, but the reality is that 
email is mostly unstructured data with a few fields of reasonably structured 
data (dates, from, to, maybe attachment types + filenames).  The bulk of the 
emails, and the part of the emails that people really want to search quickly: 
the body, is unstructured, and doesn't perform quickly with the stock "full 
text search" modules in the main SQL engines.

I'd given dbmail2 a try with MySQL 5, 5.5, and Postgres 8.4 and 9.1 branches.  
I've dedicated 16GB of DDR3-1800/3.4GHz 6-core AMD 1090T with hardware RAID 
local storage (12 x Seagate ES 7200RPM spindles). (64 bit Slackware 13.37 
running Linux 3.2 kernels built for the platform.)

The performance is surprisingly bad ... doing almost everything.  Searches 
through IMAP, bulk importation of mail folders, large numbers of simultaneous 
mail deliveries, you name it.  There wasn't a task that the dbmail setup 
performed faster than Dovecot, in either low or high load situations.  When I 
tossed a test load that introduced lots of mail deliveries as well as searches 
and full folder pulls, things got really pear-shaped.  Even putting dovecot's 
mailstore on NFS (GigE) didn't really slow Dovecot down enough to make dbmail 
competitive.

When pressed on this lack of performance, I was instructed to "add more RAM" to 
the DB machine, and that for ideal performance I should have more RAM than my 
mailbox sizes.  *sigh*  This sounds great for a very small installation, but 
this clearly is not something that scales.

I think the final humiliation was comparing the body + header searching 
performance using Timo's practically obsolete fts_squat plugin against 
dbmail's.  Wow.  Squat was multiple orders of magnitude faster.  Lucene and 
Solr are even moreso when fed large datasets (mail folder hives of about 
100GB).  The SQL setups hit the obvious performance shelf once they were unable 
to maintain everything in RAM or cache.

The dbmail folk are earnest and hard-working, and I don't mean to cast the 
slightest bit of negativity on their project.  I think the assumptions about 
what SQL servers can do well often doesn't square with the reality of many 
applications that people try to fit them into.

On my first initial round of tests, I imported 24,000 emails comprising a mere 
560MB of space.  Just about all of the non-SQL imap servers handled the 
importation (basically IMAP APPENDs) within 6 minutes.  dbmail2 required hours 
(using MySQL), and a bit shorter time (but still hours') with Postgres.

>From an old email:

> Searching INBOX #msgs = 24714
>  [NOFIND] Time=2.072423, matches=24714 <--- this should be zero *BUG*
>  [date] Time=2.07519, matches=24714 <--- this is correct
>  [here] Time=2.072075, matches=24714 <--- this should be about 30% of total # 
> of msgs *BUG*
> 
> Does dbmail break IMAP SEARCH TEXT (i.e., search both body + headers)?  Is 
> this a result of relying on MySQL's search algorithms in text-like fields? 
> I'm still puzzled, because I can't believe that 'here' appears in EVERY 
> email.  It looks like dbmail's returning EVERY email on a SEARCH TEXT.  This 
> is not correct operation.
> 
> When I alter the search to use "FROM" as the key instead of "TEXT", the 
> results are more discriminating and meet expectations.
> 
> Searching INBOX #msgs = 24714
>  [NOFIND] Time=2.161049, matches=0
>  [james] Time=2.273255, matches=1049
>  [here] Time=2.165406, matches=2
> 
> Not that it matters, but it's much slower than Dovecot's fts_squat for 
> substring searches.
> 
> Dovecot's fts_squat IMAP SEARCH TEXT results are:
> 
> Searching INBOX #msgs = 55731
>  [Updating Index] Time=78.184637 (66% of the mailbox unindexed at start)
>  [NOFIND] Time=0.045654, matches=0
>  [date] Time=0.13364, matches=55731
>  [here] Time=0.069091, matches=24663

FWIW, I found Postgres to be faster than MySQL (5 and 5.5, though 5.5 with a 
hand-rolled config file using metrics supplied by a dbmail/MySQL guru helped a 
great deal for size(data_set) < size(PHYSICAL MEMORY) cases.

Where lots of write-commits were involved on the same exact setup.  MySQL "got 
close" to PSQL's performance when I did crazy things like remove filesystem 
journaling, write barriers, etc on the mail db mountpoint.  Obviously, this is 
desperation talking.

I concede that the motivations behind SQLising mail storage extends to 
administration/replication and other non-performance/scalability aspects.  I 
suspect what constitut

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Wed, 2012-11-07 at 17:30 +0200, Timo Sirainen wrote:
> On 30.10.2012, at 2.16, Christoph Anton Mitterer wrote:
> > Have you ever thought about adding a "real" DB backend? Nothing against
> > dbox... :) ... and I have no performance comparison of dbox with what
> > could be done with a DBMS... but the advantage of the later would be
> > that you get all fancy features from database systems for free... like
> > fast indexing, online replication, etc. p..
> > One might even reuse something like AOX for this.

> SQL indexes aren't very helpful for IMAP-like data. It would be fun to
> some day have SQL backend in Dovecot (there already is read-only
> INBOX-only SQL backend), but I don't expect it to have very good
> performance.

I see... well I haven't tested AOX or dbmail so far (especially as
they're not in Debian and I was too lazy till now to compile them)...

At least I had the impression that performance (especially in searches)
was one of the major things these people were proud of.


I'll stay tuned, whether we ever see a fully usable SQL backend for
Dovecot :)


Cheers,
Chris.



smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

On 30.10.2012, at 2.16, Christoph Anton Mitterer wrote:

> Have you ever thought about adding a "real" DB backend? Nothing against
> dbox... :) ... and I have no performance comparison of dbox with what
> could be done with a DBMS... but the advantage of the later would be
> that you get all fancy features from database systems for free... like
> fast indexing, online replication, etc. p..
> 
> One might even reuse something like AOX for this.

SQL indexes aren't very helpful for IMAP-like data. It would be fun to some day 
have SQL backend in Dovecot (there already is read-only INBOX-only SQL 
backend), but I don't expect it to have very good performance.

Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

On 30.10.2012, at 13.00, Charles Marcus wrote:

> On 2012-10-29 5:42 PM, Timo Sirainen  wrote:
>> On 29.10.2012, at 23.15, Christoph Anton Mitterer wrote:
>> 
>>> btw: What are the actual advantages of sdbox over maildir?
>>  * Not moving files from new/ to cur/ directory
>>  * Not renaming files when changing message flags
>>  * Not readdir()ing directories (although maildir_very_dirty_syncs=yes helps 
>> a lot with this)
>> 
>> Basically less disk I/O and making it possible to have mailboxes with a huge 
>> number of messages without everything slowing down horribly.
> 
> I had been wanting to ask about this too...
> 
> So... what are the disadvantages?

Message flags are stored only in dovecot.index files, and files get somewhat 
more easily corrupted than the whole filesystem. Having a separate 
dovecot.index.backup file helps with this though. Also there's the 
disadvantages if you can't easily switch away from Maildir because you're using 
some non-Dovecot tools to access it.

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Tue, 2012-10-30 at 07:03 -0400, Charles Marcus wrote:
> What makes the most sense for me is to use mbox (or mdbox) for longer 
> term storage that you may be offloading to slower storage systems, and 
> use maildir (or sdbox) for the new mails...
Was also something I thought about... still the more I think about it,
the more I hate, that with mbox meta-data is stored in the mails.


> Would work great as long as you have a reliable method for archiving 
> older mails out to your slower storage.
I still hope for some DB backend ;)


Chris.


smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Tue, 2012-10-30 at 07:00 -0400, Charles Marcus wrote:
> So... what are the disadvantages?
I (but I'm no expert) would guess that it's a dovecot-only format.
No support from most other tools,...

I'd guess you cannot use e.g. maildrop with it, or can you?


I personally was always a bit worried, when meta-data is put in the
mail... now AFAIU dbox does _not_ do this... and you can cleanly extract
each unmodified mail from the dbox fail (single or multi), right?


Cheers,
Chris.


smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Charles Marcus]

On 2012-10-29 4:54 PM, Christoph Anton Mitterer  
wrote:

In the end I probably changed my opinion.
~7GB of wasted block space for all my mails is actually quite a lot, but
in days of cheap disk space it's acceptable.
And with mbox one has IMHO the major disadvantage that mailservers
(including dovecot) store some meta-data_in_  it (i.e. in the mails
themselves) , which I don't like a lot.
I still think about reports that mbox is much faster with full text
search (which sounds reasonable)... but therefore one needs probably and
database backend anyway.


What makes the most sense for me is to use mbox (or mdbox) for longer 
term storage that you may be offloading to slower storage systems, and 
use maildir (or sdbox) for the new mails...


Would work great as long as you have a reliable method for archiving 
older mails out to your slower storage.


This is what I plan on doing someday...

--

Best regards,

Charles

Re: [Dovecot] mbox vs. maildir storage block waste

[Charles Marcus]

On 2012-10-29 5:42 PM, Timo Sirainen  wrote:

On 29.10.2012, at 23.15, Christoph Anton Mitterer wrote:


btw: What are the actual advantages of sdbox over maildir?

  * Not moving files from new/ to cur/ directory
  * Not renaming files when changing message flags
  * Not readdir()ing directories (although maildir_very_dirty_syncs=yes helps a 
lot with this)

Basically less disk I/O and making it possible to have mailboxes with a huge 
number of messages without everything slowing down horribly.



I had been wanting to ask about this too...

So... what are the disadvantages?

--

Best regards,

Charles

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Tue, 2012-10-30 at 00:05 +0200, Timo Sirainen wrote:
> > And I guess the interior of the files is the same? I.e. just the plain
> > mail without any changes or quoting?
> Yes, but it's in dbox format so it contains also some extra metadata (not in 
> the mail headers).
Yeah of course... but the important point here is the "not in the mail
headers" part :)

So I've added the following changes, please double check :)

http://master.wiki2.dovecot.org/MailboxFormat/dbox?action=diff&rev2=30&rev1=29



> > For sdbox, does that part with "loosing the indexes means game
> > over" ;) , too?
> You'll lost message flags then. Both sdbox and mdbox keep
> dovecot.index.backup files and repairing tries very hard to preserve
> everything from the indexes it sees, so I don't think it's a big
> concern as long as the system behaves properly.
Yeah... sounds not too bad... :)


Off topic:

Have you ever thought about adding a "real" DB backend? Nothing against
dbox... :) ... and I have no performance comparison of dbox with what
could be done with a DBMS... but the advantage of the later would be
that you get all fancy features from database systems for free... like
fast indexing, online replication, etc. p..

One might even reuse something like AOX for this.


Cheers,
Chris.


smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

On 29.10.2012, at 23.54, Christoph Anton Mitterer wrote:

> On Mon, 2012-10-29 at 23:42 +0200, Timo Sirainen wrote:
>>> btw: What are the actual advantages of sdbox over maildir?
>> 
>> * Not moving files from new/ to cur/ directory
>> * Not renaming files when changing message flags
>> * Not readdir()ing directories (although maildir_very_dirty_syncs=yes helps 
>> a lot with this)
>> 
>> Basically less disk I/O and making it possible to have mailboxes with a huge 
>> number of messages without everything slowing down horribly.
> 
> Oh that's quite some advantage...
> 
> And I guess the interior of the files is the same? I.e. just the plain
> mail without any changes or quoting?

Yes, but it's in dbox format so it contains also some extra metadata (not in 
the mail headers).

> For sdbox, does that part with "loosing the indexes means game
> over" ;) , too?

You'll lost message flags then. Both sdbox and mdbox keep dovecot.index.backup 
files and repairing tries very hard to preserve everything from the indexes it 
sees, so I don't think it's a big concern as long as the system behaves 
properly.

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Mon, 2012-10-29 at 23:42 +0200, Timo Sirainen wrote:
> > btw: What are the actual advantages of sdbox over maildir?
> 
>  * Not moving files from new/ to cur/ directory
>  * Not renaming files when changing message flags
>  * Not readdir()ing directories (although maildir_very_dirty_syncs=yes helps 
> a lot with this)
> 
> Basically less disk I/O and making it possible to have mailboxes with a huge 
> number of messages without everything slowing down horribly.

Oh that's quite some advantage...

And I guess the interior of the files is the same? I.e. just the plain
mail without any changes or quoting?


For sdbox, does that part with "loosing the indexes means game
over" ;) , too?


Thanks,
Chris


smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

On 29.10.2012, at 23.15, Christoph Anton Mitterer wrote:

> btw: What are the actual advantages of sdbox over maildir?

 * Not moving files from new/ to cur/ directory
 * Not renaming files when changing message flags
 * Not readdir()ing directories (although maildir_very_dirty_syncs=yes helps a 
lot with this)

Basically less disk I/O and making it possible to have mailboxes with a huge 
number of messages without everything slowing down horribly.

Re: [Dovecot] mbox vs. maildir storage block waste

[Christoph Anton Mitterer]

On Mon, 2012-10-29 at 23:06 +0200, Timo Sirainen wrote:
> There is of course mdbox also, which gives the best of both mbox and maildir 
> (and some of its own new annoyances).
Thanks, Timo,... I forgot to mention that.

For me _personally_ two things speak against using it:

a) To be honest, "you must not lose the dbox index files, they can't be
regenerated without data loss"[0] made me a bit scared ;-)

b) ext* has no integrity checking (by hash sums) so I used to create my
own that puts SHA512 hashes into the inodes of files (as USER_XATTRS).
This of course, works only when you have a storage format where files
don't change anymore once written,... which can't work with formats
having multiple mails per file.


Thanks,
Chris.

btw: What are the actual advantages of sdbox over maildir?


[0] http://wiki2.dovecot.org/MailboxFormat/dbox


smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] mbox vs. maildir storage block waste

[Timo Sirainen]

On 29.10.2012, at 22.54, Christoph Anton Mitterer wrote:

> I recently mentioned in several posts, that I'd tended to use mbox
> rather than maildir, because you don't loose so much space (due to
> always allocating full blocks per maildir file and thus per mail).
..
> In the end I probably changed my opinion.
> ~7GB of wasted block space for all my mails is actually quite a lot, but
> in days of cheap disk space it's acceptable.
> And with mbox one has IMHO the major disadvantage that mailservers
> (including dovecot) store some meta-data _in_ it (i.e. in the mails
> themselves) , which I don't like a lot.
> I still think about reports that mbox is much faster with full text
> search (which sounds reasonable)... but therefore one needs probably and
> database backend anyway.

There is of course mdbox also, which gives the best of both mbox and maildir 
(and some of its own new annoyances).

Re: [Dovecot] mbox vs maildir

[William Blunn]

On 18/09/2010 18:43, Jim Pazarena wrote:
I've had clients 'request' nested folders, and it would seem that 
maildir is designed with that ability while with mbox it is difficult 
and.or impossible to implement (nested can be achieved; but not nested 
AND populated in each nest level).


You can, under mbox, have mailboxes containing messages which are also 
the parents of other mailboxes, though it requires special configuration.


Two approaches are described at http://wiki2.dovecot.org/MboxChildFolders

My question is, is one format 'better' than the other? It would take a 
fair bit of time to convert my system to maildir and I would want to 
feel comfortable that this would be a true 'upgrade' in abilities, 
rather than simply a change to accommodate nested folders.


Maildir and mbox both have advantages and disadvantages.

If you are thinking about re-assessing your mail store strategy, you 
might also want to consider single-dbox and multi-dbox.


http://wiki2.dovecot.org/MailboxFormat/dbox

Bill

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Tue, 2007-07-10 at 13:57 -0400, FiL @ Kpoxa wrote:
> >>
> >> The last time I tried to convert from mbox to maildir, things got 
> >> pretty botched up, no data loss, but it wasn't pretty. :-)
> > just because you got it wrong doesn't make it's hard. you probably 
> > didn't take enough time to get it right.
> 
> Well, if you know the RIGHT way - just share it with the rest of us.
> I was asking Timo before and as far as I understood - there is no way of 
> converting mbox to Maildir without losing message UIDs.

With existing tools. It would be pretty easy to write such a tool that
creates dovecot-uidlist file with correct UIDs.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[FiL @ Kpoxa]

The last time I tried to convert from mbox to maildir, things got 
pretty botched up, no data loss, but it wasn't pretty. :-)
just because you got it wrong doesn't make it's hard. you probably 
didn't take enough time to get it right.


Well, if you know the RIGHT way - just share it with the rest of us.
I was asking Timo before and as far as I understood - there is no way of 
converting mbox to Maildir without losing message UIDs.
And given that we already have huge mailboxes, that means that every 
user, connecting to the server after the conversion will need to
redownload all the messages and delete all the messages in the local 
cache. Just imagine that user with ~5Gb of mail across 5-10 folders
connect from home not-so-fast DSL line to check email and has to wait 
until the client rechecks all the mail No, this is not right.
Exactly as Don Russell wrote - "no data loss, but it isn't pretty". And 
this is the only reason I don't migrate to Maildir. I really want to do it,

but this is not something I can do to my users.

FiL

Re: [Dovecot] mbox vs maildir

[Tim Tsai]

Can Dovecot handle mbox for some users and maildir for others? I'd 
like to try a conversion for one user... I'll probably create a new

user, then have procmail copy (via ! action code) all mail for one
user to that new user.
  


I recently inherited a sendmail + UW IMAP installation (where hundred 
megabytes mbox files were becoming issues) and have since migrated to 
Postfix + Dovecot and have been very pleased.  During the migration to 
the new server, I set up the old UW IMAP/POP3 on 127.0.0.1 and Perdition 
proxies on the original IP address.   From Perdition you can specify 
which destination server to use via regex so it was easy to transition 
on a per user (or domain or anything else that you can specify via 
regex) basis.  As a matter of fact we migrated the users over a one 
month period and had zero issues.  The final touch was changing the DNS 
to the new server.  Most users did not even know about the change and 
gave us plenty of opportunities for live testing.  The only thing to 
watch out for is that Perdition doesn't know what capabilities to 
advertise for IMAP (especially if the two servers are very different) so 
you have to use the least common eliminator initially.


If getting a new server isn't an option, Xen and VPS environments have 
made provisioning a new server very cheap nowadays...


Tim

Re: [Dovecot] mbox vs maildir

[Don Russell]

Don Russell wrote:

I'm using Dovecot 1.0.1-12 on Linux/Fedora 7
along with sendmail and procmail all running on the same box
mail is stored in mbox format


[snip]

Thanks to all who replied. This seems to have sparked quite a 
discussion, and given me quite a bit to read/look into.


Sounds like dbox is worth waiting for. I'd I'm going to convert from 
mbox, I'd rather convert once. :-)

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

I did the ext3 -> ext4 switch on two of our proxyservers a few months
ago. Then we forgot (!) about that test and the boxes just kept
running and running and running ...



Interesting... have you noticed any differences in performance?



No. But at least it didn't explode in my face :)


Heh.. yeah, thats always a GoodThing(TM)

--

Best regards,

Charle

Re: [Dovecot] mbox vs maildir

[Michal Soltys]

Ralf Hildebrandt wrote:


XFS is lousy for many small files. We tried XFS for our 9000 Users
(Maildir) and swithced back to ext3.




Properly tuned XFS is supposedly very nice. Check out:

http://everything2.com/index.pl?node_id=1479435

for more info.

Re: [Dovecot] Mbox vs maildir

[Timo Sirainen]

On 2.7.2007, at 1.33, Simon Gao wrote:


The upcoming dbox and cydir formats of course beat everything in
performance :)



Is dbox in Dovecot 1.1 tree now? Is it the redesigned dbox? How  
close is dbox ready for general use?


I am interested in testing it out. Where can I find instruction on  
how to configure dovecot to use dbox?


Not yet. Maybe in a month or so.



PGP.sig
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[albinootje]

mouss wrote:

>> Can Dovecot handle mbox for some users and maildir for others? I'd 
>> like to try a conversion for one user... I'll probably create a new
>> user, then have procmail copy (via ! action code) all mail for one
>> user to that new user.
> 
> Why not use one of the available mbox 2 maildir utilities.

in case you would have problems with this for whatever reason, another
option
is to simply start copying mail-folders inside a mail-client which
connects to both
old and new imap-server (or old mailboxes vs. new imap-server)

with that option you can even easily convert from mailboxes with "weird"
formats like native pegasus mail mailboxes to maildir-format, assuming that
you have set maildir-format as default on your new imap-server

just connect to your new imap-server with maildir as default inside your
email-client and start copying etc.

Re: [Dovecot] mbox vs maildir

[mouss]

mouss wrote:


mbox is broken by design. Look at the next line.
From what I can tell, mbox will convert the first word of this line to 
">From".
This means the message is modified, which is ok for raw text, but is 
not ok for structure text such as TeX or XML.


argh. the example doesn't even work since my mailer added a space before 
the From! anyway, the issue is real.

Re: [Dovecot] mbox vs maildir

[mouss]

Don Russell wrote:

I'm using Dovecot 1.0.1-12 on Linux/Fedora 7
along with sendmail and procmail all running on the same box
mail is stored in mbox format

It's a small system with a half dozen or so e-mail "accounts". Each 
with 40-60MB of messages in various folders.


I keep seeing messages about how mbox is antiquated and anybody with 
more than 100 messages etc should not use mbox, but use maildir instead.


I'm not entirely convinced there seem to be pros and cons for 
each. Is there a discussion somewhere that really highlights why one 
format is so much better than the other?


mbox is broken by design. Look at the next line.
From what I can tell, mbox will convert the first word of this line to 
">From".
This means the message is modified, which is ok for raw text, but is not 
ok for structure text such as TeX or XML.


I've also had to edit mbox files with via to remove garbage, probably 
caused by lock issues.


Regarding performances: while a single file should be faster to parse 
than loading N files, how about:

- moving messages between folders (including folders for different accounts)
- rsync-like backup is simple and fast with maildir
- using messages to retrain a spam filter ("mv" is all that is needed!).



The last time I tried to convert from mbox to maildir, things got 
pretty botched up, no data loss, but it wasn't pretty. :-)


just because you got it wrong doesn't make it's hard. you probably 
didn't take enough time to get it right.


Can Dovecot handle mbox for some users and maildir for others? I'd 
like to try a conversion for one user... I'll probably create a new 
user, then have procmail copy (via ! action code) all mail for one 
user to that new user.


Why not use one of the available mbox 2 maildir utilities.

Re: [Dovecot] mbox vs maildir

[Ralf Hildebrandt]

* Charles Marcus <[EMAIL PROTECTED]>:
> >I did the ext3 -> ext4 switch on two of our proxyservers a few months
> >ago. Then we forgot (!) about that test and the boxes just kept
> >running and running and running ...
> 
> Interesting... have you noticed any differences in performance?

No. But at least it didn't explode in my face :)

-- 
Ralf Hildebrandt ([EMAIL PROTECTED]) [EMAIL PROTECTED]
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.arschkrebs.de
Am I advocating HTML email? No. I think it sucks. I'm really not interested
in your artistic whims of yellow text on funky purple wallpaper.

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

Curious, though - why *not* make cydir a real usable format, if its 
performance is so good? 



What if dbox's performance will be even better? We'll see.


Heh... don't know why I even bothered asking - you are always about 357 
steps ahead of me... ;)


Is it only/because there is no good solution to 
the extra risk of data loss if indexes get lost/corrupted?



That, and dbox will have some other features such as single instance
attachment storage, which can't be implemented to cydir without making
the format more complex (and then it's practically the same as dbox).


Bingo - dbox wins... :)

Thanks for all you do Timo...

--

Best regards,

Charles

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Sat, 2007-06-30 at 13:01 -0400, Charles Marcus wrote:
> >> One advantage of cydir over dbox was mentioned by Mark above re 
> >> incremental backups - with dbox, you'd still have to backup the 
> >> entire mailbox file, while with cydir, you'd only have to copy
> >> newer messages.
> 
> > I was thinking about making dbox configurable. If it is run in
> > one-mail-per-file mode there's no need for locking either.
> 
> Ahhh... ! Ok, that would be 'a good thing'... :)
> 
> Curious, though - why *not* make cydir a real usable format, if its 
> performance is so good? 

What if dbox's performance will be even better? We'll see.

> Is it only/because there is no good solution to 
> the extra risk of data loss if indexes get lost/corrupted?

That, and dbox will have some other features such as single instance
attachment storage, which can't be implemented to cydir without making
the format more complex (and then it's practically the same as dbox).


signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

I did the ext3 -> ext4 switch on two of our proxyservers a few months
ago. Then we forgot (!) about that test and the boxes just kept
running and running and running ...


Interesting... have you noticed any differences in performance?

--

Best regards,

Charles

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

One advantage of cydir over dbox was mentioned by Mark above re 
incremental backups - with dbox, you'd still have to backup the 
entire mailbox file, while with cydir, you'd only have to copy

newer messages.



I was thinking about making dbox configurable. If it is run in
one-mail-per-file mode there's no need for locking either.


Ahhh... ! Ok, that would be 'a good thing'... :)

Curious, though - why *not* make cydir a real usable format, if its 
performance is so good? Is it only/because there is no good solution to 
the extra risk of data loss if indexes get lost/corrupted?


--

Best regards,

Charles

Re: [Dovecot] mbox vs maildir

[Ralf Hildebrandt]

* Charles Marcus <[EMAIL PROTECTED]>:

> Please, lets not start that war up again! ;)
> 
> Reiser has worked fine for me for many years, but I think the next time 
> I rebuild my servers I'll be using ext3, in anticipation of ext4 (since 
> it should be a fairly seamless switch)...

I did the ext3 -> ext4 switch on two of our proxyservers a few months
ago. Then we forgot (!) about that test and the boxes just kept
running and running and running ...

-- 
Ralf Hildebrandt ([EMAIL PROTECTED]) [EMAIL PROTECTED]
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.arschkrebs.de
A bus station is where buses stop. A train station is where trains stop.
On my desk, there is a workstation...

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Fri, 2007-06-29 at 16:37 -0400, Charles Marcus wrote:
> > Just remember that if you lose the index files there's no easy way to
> > recover the mailbox. Well, except by copying the files to maildir..
> > 
> > I'm not sure if I should try to make cydir anything else than a
> > benchmark format or a simple example for writing mail storage backends.
> > I'm hoping that dbox will be practically as fast in all situations.
> 
> One advantage of cydir over dbox was mentioned by Mark above re 
> incremental backups - with dbox, you'd still have to backup the entire 
> mailbox file, while with cydir, you'd only have to copy newer messages.

I was thinking about making dbox configurable. If it is run in
one-mail-per-file mode there's no need for locking either.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[Rick Romero]

On Fri, 2007-06-29 at 16:32 -0400, Charles Marcus wrote:
> On 6/29/2007, Rick Romero ([EMAIL PROTECTED]) wrote:
> > I know you can do the last with LVM on Linux, and I recall something
> > similar on FreeBSD - but I have no experience with either, and
> > they're both missing salvage and snapshot.
> 
> Eh? Guess I've just been dreaming then every time I do a snapshot on one 
> of my LVM volumes..

I see that now, Wikipedia has a decent comparison page:
http://en.wikipedia.org/wiki/Comparison_of_file_systems

Snapshot is a start, but still no salvage that I can find  :(

OTOH, Being 10 years old means NSS is 'only' limited to 8TB 

Re: [Dovecot] mbox vs maildir

[Nicolas KOWALSKI]

On 6/29/07, Charles Marcus <[EMAIL PROTECTED]> wrote:

> Just remember that if you lose the index files there's no easy way to
> recover the mailbox. Well, except by copying the files to maildir..
>
> I'm not sure if I should try to make cydir anything else than a
> benchmark format or a simple example for writing mail storage backends.
> I'm hoping that dbox will be practically as fast in all situations.

One advantage of cydir over dbox was mentioned by Mark above re
incremental backups - with dbox, you'd still have to backup the entire
mailbox file, while with cydir, you'd only have to copy newer messages.


True.  In my old job, this was the main reason I switched users
mailboxes from UW-imap MBX to Cyrus. Performance was also much better
after the switch.

--
Nicolas

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

Just remember that if you lose the index files there's no easy way to
recover the mailbox. Well, except by copying the files to maildir..

I'm not sure if I should try to make cydir anything else than a
benchmark format or a simple example for writing mail storage backends.
I'm hoping that dbox will be practically as fast in all situations.


One advantage of cydir over dbox was mentioned by Mark above re 
incremental backups - with dbox, you'd still have to backup the entire 
mailbox file, while with cydir, you'd only have to copy newer messages.


I think making cydir a real, viable replacement for standard maildir 
would be a good thing...


:)

--

Best regards,

Charles

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

On 6/29/2007, Rick Romero ([EMAIL PROTECTED]) wrote:

I know you can do the last with LVM on Linux, and I recall something
similar on FreeBSD - but I have no experience with either, and
they're both missing salvage and snapshot.


Eh? Guess I've just been dreaming then every time I do a snapshot on one 
of my LVM volumes..


;)

--

Best regards,

Charles

Re: [Dovecot] mbox vs maildir

[Charles Marcus]

All data I have right now tells me to stay ReiserFS though.  Even
Dovecot's own page says XFS may not be a wise choice.



My experience tells me to stay away from ReiserFS as well.


Please, lets not start that war up again! ;)

Reiser has worked fine for me for many years, but I think the next time 
I rebuild my servers I'll be using ext3, in anticipation of ext4 (since 
it should be a fairly seamless switch)...


--

Best regards,

Charles

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Fri, 2007-06-29 at 22:16 +0200, Nicolas KOWALSKI wrote:
> As I noticed almost no difference in performance (no real numbers
> here, just a usage feeling with 5-10k messages mailboxes) between
> dovecot 1.0 + maildir and cyrus 2.2.13/2.3.8, having dovecot use a
> storage format 10 times faster than Maildir is really attractive.

The 10 times was only when appending new messages. I haven't benchmarked
other operations. I doubt there is any noticeable performance gain for
single users, unless your mailbox has hundreds of thousands of messages.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[Nicolas KOWALSKI]

Timo Sirainen <[EMAIL PROTECTED]> writes:

> On Fri, 2007-06-29 at 21:49 +0200, Nicolas KOWALSKI wrote:
>> Timo Sirainen <[EMAIL PROTECTED]> writes:
>> 
>> > On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
>> >> Timo Sirainen <[EMAIL PROTECTED]> writes:
>> >> 
>> >> > The upcoming dbox and cydir formats of course beat everything in
>> >> > performance :)
>> >> 
>> >> cydir ? Does this mean there is a cyrus-like storage coming soon ?
>> >
>> > Already in v1.1 tree. It's what I'm using for my index file stress
>> > tests, because the format is practically just the index files and
>> > "." named files.
>> 
>> I just read http://www.dovecot.org/list/dovecot/2007-May/022772.html,
>> and the results look really impressive. 10 times faster than
>> Maildir... 
>> 
>> Time to upgrade and test ! :-)
>
> Just remember that if you lose the index files there's no easy way to
> recover the mailbox. Well, except by copying the files to maildir..

Good to know, Thanks. 

> I'm not sure if I should try to make cydir anything else than a
> benchmark format or a simple example for writing mail storage backends.
> I'm hoping that dbox will be practically as fast in all situations.

If performance of dbox is as fast as cydir, and if cydir is not easily
recoverable (a cyrus reconstruct-like tool would help a lot here),
well, cydir could stay as benchmark format. But...

As I noticed almost no difference in performance (no real numbers
here, just a usage feeling with 5-10k messages mailboxes) between
dovecot 1.0 + maildir and cyrus 2.2.13/2.3.8, having dovecot use a
storage format 10 times faster than Maildir is really attractive.

-- 
Nicolas

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Fri, 2007-06-29 at 21:49 +0200, Nicolas KOWALSKI wrote:
> Timo Sirainen <[EMAIL PROTECTED]> writes:
> 
> > On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
> >> Timo Sirainen <[EMAIL PROTECTED]> writes:
> >> 
> >> > The upcoming dbox and cydir formats of course beat everything in
> >> > performance :)
> >> 
> >> cydir ? Does this mean there is a cyrus-like storage coming soon ?
> >
> > Already in v1.1 tree. It's what I'm using for my index file stress
> > tests, because the format is practically just the index files and
> > "." named files.
> 
> I just read http://www.dovecot.org/list/dovecot/2007-May/022772.html,
> and the results look really impressive. 10 times faster than
> Maildir... 
> 
> Time to upgrade and test ! :-)

Just remember that if you lose the index files there's no easy way to
recover the mailbox. Well, except by copying the files to maildir..

I'm not sure if I should try to make cydir anything else than a
benchmark format or a simple example for writing mail storage backends.
I'm hoping that dbox will be practically as fast in all situations.




signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[Nicolas KOWALSKI]

Timo Sirainen <[EMAIL PROTECTED]> writes:

> On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
>> Timo Sirainen <[EMAIL PROTECTED]> writes:
>> 
>> > The upcoming dbox and cydir formats of course beat everything in
>> > performance :)
>> 
>> cydir ? Does this mean there is a cyrus-like storage coming soon ?
>
> Already in v1.1 tree. It's what I'm using for my index file stress
> tests, because the format is practically just the index files and
> "." named files.

I just read http://www.dovecot.org/list/dovecot/2007-May/022772.html,
and the results look really impressive. 10 times faster than
Maildir... 

Time to upgrade and test ! :-)

Thanks Timo,
-- 
Nicolas

Re: [Dovecot] mbox vs maildir

[Ben Winslow]

On Fri, 2007-06-29 at 13:56 -0400, Jesse C. Smillie wrote:
> Wow this is weird because I'm about to make this same jump next week!
> 
>  From what I'm reading so far the big draw back with mbox is the single 
> file with all the emails in it.  When you delete a message from that 
> file the whole file has to be rewritten without that email in it.  If 
> the box is big enough that can be a serious drag on the server.  We have 
> been using Dovecot here all school year for Imap & Pop3 with the Mbox 
> format and when two or more people delete at the same time the 
> utilization on my 3ware card shoots up.  We bought the BBU unit for the 
> 3ware so I could enable WRITE cache and that has helped tremendously. 
> 
> I thought this study in regards to speed was quite interesting:
>  http://www.courier-mta.org/mbox-vs-maildir/ 
> 
> 
> So far my testing conversion process has gone really well.  I am 
> surprised how easy it was to tell procmail to do MailDir instead and 
> even the conversion process was super easy.  For converting the old 
> inbox and folders I am using the tool  mb2md.pl from 
> http://batleth.sapienti-sat.org/projects/mb2md/
> 
> I was having a really hard time figuring all of this out until I ran 
> into this webpage:
> http://adam.rosi-kessel.org/weblog/2007/04/18/adams-super-simple-guide-to-mbox-maildir-conversion/
> 
> I know through namespaces you can do inbox in one type and other boxes 
> in another type.  I was initially thinking about doing all new stuff in 
> maildir and still support the old ~/mail format.  The setup seemed easy 
> enough, but I figured in the long run I am shutting down the server for 
> a few hours to do this so I mis well go all the way. 
> 
> The only thing I'm not sure of is what the best file system to keep this 
> on.  I have been keeping my home directories on ReiserFS for quite a 
> while, but one of our tech thinks XFS would be good.  All data I have 
> right now tells me to stay ReiserFS though.  Even Dovecot's own page 
> says XFS may not be a wise choice.

We've had good experiences with XFS, hosting ~40k mailboxes totalling
~400 GiB across 2 NFS fileservers (although we're not using dovecot for
those users, yet) on hardware that's not particularly beefy by today's
standards (and well below the specs you provided for your server.)

I've also personally had terrible experiences on a couple of other
systems with reiserfs -- especially when the FS became slightly corrupt
(due to failing hardware or the power going out at *just* the right
time), though performance wasn't that great either.

All the systems I mention were/are using Maildir, and this is somewhat
contrary to many benchmarks, but it's been working well for me.  YMMV
and all that rot.

-- 
Ben Winslow <[EMAIL PROTECTED]>


signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[Mark Nienberg]

Don Russell wrote:

I'm using Dovecot 1.0.1-12 on Linux/Fedora 7
along with sendmail and procmail all running on the same box
mail is stored in mbox format

It's a small system with a half dozen or so e-mail "accounts". Each with 
40-60MB of messages in various folders.


I keep seeing messages about how mbox is antiquated and anybody with 
more than 100 messages etc should not use mbox, but use maildir instead.


I'm not entirely convinced there seem to be pros and cons for each. 
Is there a discussion somewhere that really highlights why one format is 
so much better than the other?


One factor not often discussed: When I switched from mbox to maildir the size of 
incremental backups went down to a fraction of its previous size (only the new 
messages are backed up, not the entire mailbox).  It allowed me to do incremental 
backups to disk instead of tape, which I still use for full backups.


P.S.  consider maildrop instead of procmail if you switch to maildir.  See info in 
the dovecot wiki.


Mark

Re: [Dovecot] mbox vs maildir

[Rick Romero]

On Fri, 2007-06-29 at 20:53 +0200, Ralf Hildebrandt wrote:
> * Rick Romero <[EMAIL PROTECTED]>:
> 
> > > My experience tells me to stay away from ReiserFS as well.
> > 
> > How about NSS?   I was considering using Netware's NSS for my backend
> > server - either from Netware or Linux OES, but I'm not sure how OES
> > actually handles the filesystem.. Anyways, in addition to journaling,
> > you get deleted file salvage, open file backup/snapshot, and easily
> > expandable volumes...
> 
> Sounds like a job for ZFS :)

Yeah, but ZFS seems too new compared to 10 year old NSS ;)

Re: [Dovecot] mbox vs maildir

[Ralf Hildebrandt]

* Rick Romero <[EMAIL PROTECTED]>:

> > My experience tells me to stay away from ReiserFS as well.
> 
> How about NSS?   I was considering using Netware's NSS for my backend
> server - either from Netware or Linux OES, but I'm not sure how OES
> actually handles the filesystem.. Anyways, in addition to journaling,
> you get deleted file salvage, open file backup/snapshot, and easily
> expandable volumes...

Sounds like a job for ZFS :)

-- 
Ralf Hildebrandt ([EMAIL PROTECTED]) [EMAIL PROTECTED]
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.arschkrebs.de
Ballmer should step down in favour of Mr T, because he pity the fool
who don't got high-end video cards and 4GB RAM for Vista Aero!

Re: [Dovecot] mbox vs maildir

[Rick Romero]

On Fri, 2007-06-29 at 20:34 +0200, Ralf Hildebrandt wrote:
> * Jesse C. Smillie <[EMAIL PROTECTED]>:
> 
> > The only thing I'm not sure of is what the best file system to keep this 
> > on.  I have been keeping my home directories on ReiserFS for quite a 
> > while, but one of our tech thinks XFS would be good.
> 
> XFS is lousy for many small files. We tried XFS for our 9000 Users
> (Maildir) and swithced back to ext3.
> 
> > All data I have right now tells me to stay ReiserFS though.  Even
> > Dovecot's own page says XFS may not be a wise choice.
> 
> My experience tells me to stay away from ReiserFS as well.

How about NSS?   I was considering using Netware's NSS for my backend
server - either from Netware or Linux OES, but I'm not sure how OES
actually handles the filesystem.. Anyways, in addition to journaling,
you get deleted file salvage, open file backup/snapshot, and easily
expandable volumes...I know you can do the last with LVM on Linux,
and I recall something similar on FreeBSD - but I have no experience
with either, and they're both missing salvage and snapshot.

Ok ok, so I really just love salvage.  ;)

Re: [Dovecot] mbox vs maildir

[Jeff Kowalczyk]

FWIW, I used imapsync for the data migrations of a) an old sendmail server
with mbox format to a new server running postfix+dovecot, and b) and old
and busted Microsoft SBS2000 Exchange instance to a new server running
postfix+dovecot.

Worked well in both cases, and the process left the original
servers intact in case the migration hadn't gone as well.

Completely unscientific observation, but the Maildir format seems quite
responsive in the common email pattern of open the inbox, fetch the
message headers, let the filter process spam, finally read ham messages.
That has a lot to do with Dovecot's speed, of course.

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Fri, 2007-06-29 at 20:26 +0200, Nicolas KOWALSKI wrote:
> Timo Sirainen <[EMAIL PROTECTED]> writes:
> 
> > The upcoming dbox and cydir formats of course beat everything in
> > performance :)
> 
> cydir ? Does this mean there is a cyrus-like storage coming soon ?

Already in v1.1 tree. It's what I'm using for my index file stress
tests, because the format is practically just the index files and
"." named files.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[John Gateley]

On Fri, 29 Jun 2007 10:14:42 -0700
Don Russell <[EMAIL PROTECTED]> wrote:

> Can Dovecot handle mbox for some users and maildir for others?

Yes, if you don't have the mail_location variable set,
then Dovecot will look in
~/Maildir
/var/mail/username
~/mail
~/Mail
in that order.

See http://wiki.dovecot.org/MailLocation

j

Re: [Dovecot] mbox vs maildir

[Ralf Hildebrandt]

* Jesse C. Smillie <[EMAIL PROTECTED]>:

> The only thing I'm not sure of is what the best file system to keep this 
> on.  I have been keeping my home directories on ReiserFS for quite a 
> while, but one of our tech thinks XFS would be good.

XFS is lousy for many small files. We tried XFS for our 9000 Users
(Maildir) and swithced back to ext3.

> All data I have right now tells me to stay ReiserFS though.  Even
> Dovecot's own page says XFS may not be a wise choice.

My experience tells me to stay away from ReiserFS as well.

-- 
Ralf Hildebrandt ([EMAIL PROTECTED]) [EMAIL PROTECTED]
Postfix - Einrichtung, Betrieb und Wartung   Tel. +49 (0)30-450 570-155
http://www.arschkrebs.de
Unix is the answer, but only if you phrase the question very carefully.

Re: [Dovecot] mbox vs maildir

[Nicolas KOWALSKI]

Timo Sirainen <[EMAIL PROTECTED]> writes:

> The upcoming dbox and cydir formats of course beat everything in
> performance :)

cydir ? Does this mean there is a cyrus-like storage coming soon ?

-- 
Nicolas

Re: [Dovecot] mbox vs maildir

[Timo Sirainen]

On Fri, 2007-06-29 at 13:56 -0400, Jesse C. Smillie wrote:
> I thought this study in regards to speed was quite interesting:
>  http://www.courier-mta.org/mbox-vs-maildir/ 
> 

It probably doesn't have much relevance to Dovecot+mbox though. Maildir
is faster with expunges and with concurrent mailbox access, but I think
Dovecot+mbox (without external changes) is faster for pretty much
everything else.

Anyway I think the main reason to use maildir is that it's much more
difficult to corrupt mailboxes.

The upcoming dbox and cydir formats of course beat everything in
performance :)



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] mbox vs maildir

[Jesse C. Smillie]

Wow this is weird because I'm about to make this same jump next week!

From what I'm reading so far the big draw back with mbox is the single 
file with all the emails in it.  When you delete a message from that 
file the whole file has to be rewritten without that email in it.  If 
the box is big enough that can be a serious drag on the server.  We have 
been using Dovecot here all school year for Imap & Pop3 with the Mbox 
format and when two or more people delete at the same time the 
utilization on my 3ware card shoots up.  We bought the BBU unit for the 
3ware so I could enable WRITE cache and that has helped tremendously. 


I thought this study in regards to speed was quite interesting:
http://www.courier-mta.org/mbox-vs-maildir/ 



So far my testing conversion process has gone really well.  I am 
surprised how easy it was to tell procmail to do MailDir instead and 
even the conversion process was super easy.  For converting the old 
inbox and folders I am using the tool  mb2md.pl from 
http://batleth.sapienti-sat.org/projects/mb2md/


I was having a really hard time figuring all of this out until I ran 
into this webpage:

http://adam.rosi-kessel.org/weblog/2007/04/18/adams-super-simple-guide-to-mbox-maildir-conversion/

I know through namespaces you can do inbox in one type and other boxes 
in another type.  I was initially thinking about doing all new stuff in 
maildir and still support the old ~/mail format.  The setup seemed easy 
enough, but I figured in the long run I am shutting down the server for 
a few hours to do this so I mis well go all the way. 

The only thing I'm not sure of is what the best file system to keep this 
on.  I have been keeping my home directories on ReiserFS for quite a 
while, but one of our tech thinks XFS would be good.  All data I have 
right now tells me to stay ReiserFS though.  Even Dovecot's own page 
says XFS may not be a wise choice.


Hope some of this stuff helps you.  My server BTW is:
Slackware Slamd64 11 (Added Kerberos, Dovecot, etc after the fact)
Dual AMD Opteron 242s
4 Gigs RAM
800 Gig RAID 5 3G SATA array
ReiserFS on /home /var/spool/mail


-Jesse C. Smillie

"Insert inspirational or witty comment here"



Don Russell wrote:

I'm using Dovecot 1.0.1-12 on Linux/Fedora 7
along with sendmail and procmail all running on the same box
mail is stored in mbox format

It's a small system with a half dozen or so e-mail "accounts". Each 
with 40-60MB of messages in various folders.


I keep seeing messages about how mbox is antiquated and anybody with 
more than 100 messages etc should not use mbox, but use maildir instead.


I'm not entirely convinced there seem to be pros and cons for 
each. Is there a discussion somewhere that really highlights why one 
format is so much better than the other?


The last time I tried to convert from mbox to maildir, things got 
pretty botched up, no data loss, but it wasn't pretty. :-)


Can Dovecot handle mbox for some users and maildir for others? I'd 
like to try a conversion for one user... I'll probably create a new 
user, then have procmail copy (via ! action code) all mail for one 
user to that new user.


Thank you

-
This mail was scanned by BitDefender
For more informations please visit http://www.bitdefender.com


-
begin:vcard
fn:Jesse C.  Smillie
n:Smillie;Jesse C. 
org:Gateway School District;Technology Department
adr:;;;Monroeville;PA;15146;USA
email;internet:[EMAIL PROTECTED]
title:Mac  Tech / Linux Administrator / Mac Administrator
tel;work:412-858-0453
tel;cell:412-861-3423
x-mozilla-html:TRUE
version:2.1
end:vcard



smime.p7s
Description: S/MIME Cryptographic Signature