Re: maildir over mbox?
On Tue, 22 Jan 2002, David Clarke wrote: > Don't know why but for me there isn't much of a difference between them, > everyone else seems to be getting a big difference. I was however I just noticed the partition I was testing on was actually ext3, which probably explains my results. David. -- All generalizations are false, including this one. -- Mark Twain - David Clarke <[EMAIL PROTECTED]> | David Clarke Key Fingerprint : 869B 53DD 5E80 E1F0 93F6 9871 0508 0296 5957 F723
Re: maildir over mbox?
Matthew --
...and then Matthew D. Fuller said...
%
% On Mon, Jan 21, 2002 at 09:26:55PM -0500 I heard the voice of
% Derek D. Martin, and lo! it spake thus:
% >
% > In which case I would ask, dude, why? I thought my counterpart at
% > work was a pack rat... ;-)
%
% Hmmm
% Well, my current archives have... *does some quick scripting*
% (ttyp4):{920}% cat temp | dc
% 817181
%
% So not quite a million messages, but still far more inodes than I'd care
% to eat on /home.
Ouch. Yeah.
%
% I rotate my folders manually every few weeks; generally, once a mailbox
% (in Maildir, being an active mailbox) reached 4000-7000 messages, and
% starts taking more than 6 or 7 seconds to open, I tag-all and drop it
% into a mbox, then slot that mbox into my archives.
Makes sense.
%
% So, does that make ME a packrat? ;)
% For reference:
% (ttyp4):{921}% du -sh .
% 2.5G.
Well, ...
[zero] [9:05am] ~/Mail> cat `find . -type f -print | egrep -v /Z/ | \
egrep '/F\.'` | egrep '^From ' | wc -l
38124
[zero] [9:09am] ~/Mail> cat `find . -type f -print | egrep -v /Z/ | \
egrep -v '/F\.'` | egrep '^From ' | wc -l
31237
[zero] [9:13am] ~/Mail> zcat `find . -type f -print | egrep /Z/ | \
egrep '/F\.'` | egrep '^From ' | wc -l
168069
[zero] [9:16am] ~/Mail> zcat `find . -type f -print | egrep /Z/ | \
egrep -v '/F\.'` | egrep '^From ' | wc -l
28940
[zero] [9:16am] ~/Mail> echo "0 38124+ 31237+ 168069+ 28940+ pq" | dc
266370
[zero] [9:16am] ~/Mail> du -sh .
1.1G.
I suppose it does, since I figure I am. I honestly thought I'd at least
come close, but you're running nearly four times as many messages as I.
Of course, you're only taking up twice as much disk space; you must have
smaller messages overall ;-)
[zero] [9:18am] ~/Mail> ls -lFRt | tail
-rw--- 1 davidtg 829 Jan 12 1994 alan.z
How old is your oldest? :-)
%
% --
% Matthew Fuller (MF4839) |[EMAIL PROTECTED]
% Unix Systems Administrator |[EMAIL PROTECTED]
% Specializing in FreeBSD |http://www.over-yonder.net/
%
% "The only reason I'm burning my candle at both ends, is because I
% haven't figured out how to light the middle yet"
:-D
--
David T-G * It's easier to fight for one's principles
(play) [EMAIL PROTECTED] * than to live up to them. -- fortune cookie
(work) [EMAIL PROTECTED]
http://www.justpickone.org/davidtg/Shpx gur Pbzzhavpngvbaf Qrprapl Npg!
msg23553/pgp0.pgp
Description: PGP signature
Re: maildir over mbox?
On Mon, 21 Jan 2002, Alexander Skwar wrote: > FWIW, I just did some testing using mutt 1.3.25i on my Athlon 800 MHz, > 786 MB RAM, and a IBM DDRS-39130W SCSI UW hard drive running reiserfs. > The Maildir/mbox I tested, had 84.533 messages and about 321 MB. > Opening the mbox beast took 2:53 minutes, while opening the same > converted to Maildir (with mutt) took more than 25 minutes. Go > figure... A little different on my system, the mailbox I tested had 13028 emails, and was only fairly small, 36 meg. On a MAXTOR 6L040J2 running over ata66 on a reiser partition. Athlon 1000MHz, with 384MB ram. mbox 2.88s user 0.25s system 93% cpu 3.357 total maildir 3.10s user 0.82s system 94% cpu 4.014 total Don't know why but for me there isn't much of a difference between them, everyone else seems to be getting a big difference. I was however surprised with the cpu use. Looks like both are fairly heavy on my system. The times are fairly average, I ran it a few times and it made little difference, around 0.05s, and not much difference to the cpu usage. -- All generalizations are false, including this one. -- Mark Twain - David Clarke <[EMAIL PROTECTED]> | David Clarke Key Fingerprint : 869B 53DD 5E80 E1F0 93F6 9871 0508 0296 5957 F723
Re: maildir over mbox?
On Mon, Jan 21, 2002 at 09:26:55PM -0500 I heard the voice of
Derek D. Martin, and lo! it spake thus:
>
> Um... Oh, are you European? I seem to recall that Europeans switch
> the meaning of '.' and ',' in numbers, as compared to us US types...
> So perhaps you meant eighty-four thousand five hundred thirty-three
> messages?
>
> In which case I would ask, dude, why? I thought my counterpart at
> work was a pack rat... ;-)
Hmmm
Well, my current archives have... *does some quick scripting*
(ttyp4):{920}% cat temp | dc
817181
So not quite a million messages, but still far more inodes than I'd care
to eat on /home.
I rotate my folders manually every few weeks; generally, once a mailbox
(in Maildir, being an active mailbox) reached 4000-7000 messages, and
starts taking more than 6 or 7 seconds to open, I tag-all and drop it
into a mbox, then slot that mbox into my archives.
So, does that make ME a packrat? ;)
For reference:
(ttyp4):{921}% du -sh .
2.5G.
--
Matthew Fuller (MF4839) |[EMAIL PROTECTED]
Unix Systems Administrator |[EMAIL PROTECTED]
Specializing in FreeBSD |http://www.over-yonder.net/
"The only reason I'm burning my candle at both ends, is because I
haven't figured out how to light the middle yet"
Re: maildir over mbox?
* Alexander Skwar ([EMAIL PROTECTED]) wrote: > With mutt, I get the same kind of results. However, other MUAs behave > differently. For instance, when I did the testing, Evolution opened > the same Maildir way faster than it handled the mbox file (which > contained the same messages). mutt seems to be *extremely* optimized > for mbox. Evolution is aimed at maildir, and so does metadata caching. Probably worth a look by anyone interested in adding similar functionality to mutt (hello Mr Elkins :) And yes, mutt's optimized for mbox style stuff, in that it does minimal writes 'n' stuff which lesser clients probably don't. 'n' stuff. > reiserfs. I wonder what sort of interesting things you could do with mailfs.. having the underlying filesystem know something about the format of messages and how they're accessed would probably be good for those users who absolutely demand support for insanely huge mailspools. > The Maildir/mbox I tested, had 84.533 messages and about 321 MB. > Opening the mbox beast took 2:53 minutes, while opening the same > converted to Maildir (with mutt) took more than 25 minutes. Go > figure... I think the obvious answer here is "don't use mailboxes of any format that big and expect any sort of decent performance outside some very specific situations" :) Metadata caching would probably be good on really large mboxes too, but tbh I think the effort would be better spent on a decent mail archiving tool that handles all the different formats, compression etc, so us must-not-delete-anything obsessives can keep our working folders small :) -- Thomas 'Freaky' Hurst - [EMAIL PROTECTED] - http://www.aagh.net/ - Vax Vobiscum
Re: maildir over mbox?
At some point hitherto, Alexander Skwar hath spake thusly: > The Maildir/mbox I tested, had 84.533 messages and about 321 MB. Opening Eh? How does one have .533 messages in a mailbox? Perhaps fractional messages are some feature of Maildir that I was unaware of? Um... Oh, are you European? I seem to recall that Europeans switch the meaning of '.' and ',' in numbers, as compared to us US types... So perhaps you meant eighty-four thousand five hundred thirty-three messages? In which case I would ask, dude, why? I thought my counterpart at work was a pack rat... ;-) -- Derek Martin [EMAIL PROTECTED] - I prefer mail encrypted with PGP/GPG! GnuPG Key ID: 0x81CFE75D Retrieve my public key at http://pgp.mit.edu Learn more about it at http://www.gnupg.org msg23499/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
On Mon, Jan 21, 2002 at 10:37:01PM +0100, Alexander Skwar wrote: > running reiserfs. well ... as tests showed, ReiserFS seems to be a _really_ slow beast when it comes to read Maildir folders ... tried with Ext2/3? Should be really faster. -- Christian Ordig Germany
Re: maildir over mbox?
So sprach »Christian Ordig« am 2002-01-21 um 11:22:04 +0100 : > opening times might be ... but think about updating times and the Well, that's true, however, updating times aren't *that* important for me. When I receive new mail, I let procmail sort it into appropriate mailfiles; each list has got its own mailfile plus some other personal files. Now, when new mail arrives, mutt notices that there's new mail in one of the files, and I'll change to that file. So I find myself *very* often opening mailfiles. And since the difference is that tremendously bad for Maildir on reiserfs on my machine, I can't stand to wait that extremely long. Futher, updating times in mbox with mutt aren't that bad, after all. This may be due to the fact, that I much more often delete messages which are at the end of the file, which seems to be faster than deleteing huge/a lot of messages from the middle of the mbox. > "no locking needed" goodies :-) I don't use NFS, so I don't need this goody. Alexander Skwar -- How to quote: http://learn.to/quote (german) http://quote.6x.to (english) Homepage: http://www.iso-top.de | Jabber: [EMAIL PROTECTED] iso-top.de - Die günstige Art an Linux Distributionen zu kommen Uptime: 7 days 0 hours 12 minutes
Re: maildir over mbox?
So sprach »Benjamin Michotte« am 2002-01-20 um 19:33:10 +0100 : > > What are the benefits of using one type over the other? > opening a mbox with ± 7000 mails : less than 10 seconds. > opening the same in Maildir : 3 minutes... With mutt, I get the same kind of results. However, other MUAs behave differently. For instance, when I did the testing, Evolution opened the same Maildir way faster than it handled the mbox file (which contained the same messages). mutt seems to be *extremely* optimized for mbox. FWIW, I just did some testing using mutt 1.3.25i on my Athlon 800 MHz, 786 MB RAM, and a IBM DDRS-39130W SCSI UW hard drive running reiserfs. The Maildir/mbox I tested, had 84.533 messages and about 321 MB. Opening the mbox beast took 2:53 minutes, while opening the same converted to Maildir (with mutt) took more than 25 minutes. Go figure... Alexander Skwar -- How to quote: http://learn.to/quote (german) http://quote.6x.to (english) Homepage: http://www.iso-top.de | Jabber: [EMAIL PROTECTED] iso-top.de - Die günstige Art an Linux Distributionen zu kommen Uptime: 6 days 1 hour 14 minutes
Re: maildir over mbox?
I wrote: >% And it appears that the archive mailboxes *have* to be mboxes. >% If I try to save a message to an existing maildir folder, mutt objects. At 12:17 PM 1/21/02 -0500, David T-G wrote: >Um, that shouldn't be the case. mutt will happily read and write mbox, >Maildir, MMDF, and MH mailboxes with no complaints. Are you sure you're >using the trailing slash (dunno if it's really necessary)? What is the >objection? Can you give us more details? It actually didn't object; it simply created it as an mbox. But the answer was even simpler than the trailing slash, which I had. What I didn't have --- and thought I did --- was the set mbox_type="Maildir" in .muttrc Ooops! All is well now andy Andy Davidson --- Pheon Research If you know what you're doing, you're not learning anything.
Re: maildir over mbox?
Andy -- ...and then Andy Davidson said... % % At 04:50 AM 1/21/02 -0600, Matthew D. Fuller wrote: % > % >Personally, I use maildir for all my 'active' mailboxes (read: the ones ... % >I use mbox for my archive mailboxes, because it's simpler and more compact ... % % I am in the process of converting to using mutt, so I've been playing Welcome! % a bit. And it appears that the archive mailboxes *have* to be mboxes. % If I try to save a message to an existing maildir folder, mutt objects. Um, that shouldn't be the case. mutt will happily read and write mbox, Maildir, MMDF, and MH mailboxes with no complaints. Are you sure you're using the trailing slash (dunno if it's really necessary)? What is the objection? Can you give us more details? HTH & HAND :-D -- David T-G * It's easier to fight for one's principles (play) [EMAIL PROTECTED] * than to live up to them. -- fortune cookie (work) [EMAIL PROTECTED] http://www.justpickone.org/davidtg/Shpx gur Pbzzhavpngvbaf Qrprapl Npg! msg23462/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
Michael Elkins wrote: > > I'd be curious to get some feedback on my header caching patch for maildir > folders (can be found at http://www.sigpipe.org:8080/mutt/). ok a little more feedback. overall performance is a little zippier, but if i leave a folder open and it receives messages, i get the 'folder was externally modified' message, and sometimes the message indicator seems to have jumped up (although maybe i'm just hallucinating). this makes sense, but it might be nice if there were a way around it. w
Re: maildir over mbox?
At 04:50 AM 1/21/02 -0600, Matthew D. Fuller wrote: > >Personally, I use maildir for all my 'active' mailboxes (read: the ones >that mail gets delivered to and I read) because it's that much safer, >easier and more efficient to alter, and roughly similar in speed to open. >I use mbox for my archive mailboxes, because it's simpler and more compact >(I don't need to blow a few million inodes on mail archives, thank you >very much), and it's faster on the mailboxes with tens or hundreds of >thousands of messages. I am in the process of converting to using mutt, so I've been playing a bit. And it appears that the archive mailboxes *have* to be mboxes. If I try to save a message to an existing maildir folder, mutt objects. >And, having spent rather some time lately writing code to parse mbox's, >I'd like to make the following general comment: >Bah. Agreed. andy
Re: maildir over mbox?
At some point hitherto, Thomas Hurst hath spake thusly: > * Derek D. Martin ([EMAIL PROTECTED]) wrote: > > > At some point hitherto, Will Yardley hath spake thusly: > > > our office mail machine is (unfortunately) linux with ext2, and i > > > can attest to the fact that Maildir is pretty slow on ext2. > > > > And most other filesystems... Try it on FAT. =8^) > > I think the overhead of opening and closing tonnes of small files is > inherintly going to be slower than one big file on reading. Right... > -% mutt -Re 'push q' -f test-mbox > 6.80s user 0.75s system 85% cpu 8.810 total > > -% mutt -Re 'push q' -f test-maildir > 7.39s user 2.38s system 32% cpu 29.882 total Yeah, that's what I mean. And that's with your optimized filesystem and a pretty beefy system. Now don't get me wrong, I'm not saying that maildir doesn't have advantages. It does. For me though, this is a significant enough trade-off to make sticking with mbox worthwhile. I'm impatient! :) -- Derek Martin [EMAIL PROTECTED] - I prefer mail encrypted with PGP/GPG! GnuPG Key ID: 0x81CFE75D Retrieve my public key at http://pgp.mit.edu Learn more about it at http://www.gnupg.org msg23443/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
* Derek D. Martin ([EMAIL PROTECTED]) wrote: > At some point hitherto, Will Yardley hath spake thusly: > > our office mail machine is (unfortunately) linux with ext2, and i > > can attest to the fact that Maildir is pretty slow on ext2. > > And most other filesystems... Try it on FAT. =8^) I think the overhead of opening and closing tonnes of small files is inherintly going to be slower than one big file on reading. Maildir is a simpler format for the MUA; one of mutt's advantages is with mbox it does minimal writes, so for instance when it changes a status flag, it just overwrites the old one, where as a lesser client may well end up rewriting the entire mailbox from that point. With Maildir, even if the MUA rewrites the entire message each time the change is always limited to one small file. Maildir's advantages are in reliability (it's harder to corrupt 1000 small files than 1 big one), and modification cost (moving/deleting/editing a message is a constant time operation, where as mbox will potentially get slower as it gets bigger. This isn't so bad when the mailbox is small, or when you're modifying messages near the end, but things like message archiving is going to be faster (and easier) with maildir. > > i haven't tried copying one of my large Maildir folders over to one > > of my machines (FreeBSD, UFS filesystem with soft-updates) to see > > how much faster it is, but i've heard that the difference is pretty > > large. > > But how does it compare to mbox on the same FS? I'll bet it's still > significantly slower. Well, I'm running FreeBSD 4.5 + UFS+FFS+SU and dirhashing, let's see.. I did try this a few months ago, btw, Maildir added about 25% to opening time; good metadata caching could well make the difference here. Personally, though, I'd use a minimal mbox type file over a dbm for that; dbm's involve a lot of seeking, which is what we're trying to reduce :) Anyway; -% cat archive/2001/lists/cvs-all/11-2001.cvs-all archive/2001/lists/cvs-all/12-2001.cvs-all >test-mbox -% grep -c '^From ' test-mbox 7870 -% time mutt -Re 'push q' -f test-mbox 6.75s user 0.80s system 99% cpu 7.618 total It took about a minute to convert to a mailbox; -% mutt -Re 'push q' -f test-maildir 6.96s user 1.56s system 97% cpu 8.784 total Those are cached values, though, so they're not really measuring filesystem performance, s, let's malloc 300MB and fill it with zero's, and: -% mutt -Re 'push q' -f test-mbox 6.80s user 0.75s system 85% cpu 8.810 total -% mutt -Re 'push q' -f test-maildir 7.39s user 2.38s system 32% cpu 29.882 total Witness the overhead of all that seeking. This is a Dual 466MHz Celeron w/ 384MB and a 20GB UDMA/33 drive. -- Thomas 'Freaky' Hurst - [EMAIL PROTECTED] - http://www.aagh.net/ - You've been telling me to relax all the way here, and now you're telling me just to be myself? -- The Return of the Secaucus Seven
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 06:45:54PM -0800 I heard the voice of Michael Elkins, and lo! it spake thus: > > Mutt attempts to compensate for this by using quoted-printable encoding when > it detects things that might break a signature, thus escaping the problem. > But yes, mbox format is more susceptable to corruption of this form. Mutt also bypasses the problem by not doing From_-escaping when the message has its handy-dandy Content-Length: header (which makes it fun to parse the files if you don't think of it... "Why does this 8-message folder show 12 messages?"). Personally, I use maildir for all my 'active' mailboxes (read: the ones that mail gets delivered to and I read) because it's that much safer, easier and more efficient to alter, and roughly similar in speed to open. I use mbox for my archive mailboxes, because it's simpler and more compact (I don't need to blow a few million inodes on mail archives, thank you very much), and it's faster on the mailboxes with tens or hundreds of thousands of messages. And, having spent rather some time lately writing code to parse mbox's, I'd like to make the following general comment: Bah. -- Matthew Fuller (MF4839) |[EMAIL PROTECTED] Unix Systems Administrator |[EMAIL PROTECTED] Specializing in FreeBSD |http://www.over-yonder.net/ "The only reason I'm burning my candle at both ends, is because I haven't figured out how to light the middle yet"
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 09:39:52PM -0500, Derek D. Martin wrote: > But how does it compare to mbox on the same FS? I'll bet it's still > significantly slower. opening times might be ... but think about updating times and the "no locking needed" goodies :-) -- Christian Ordig Germany
Re: maildir over mbox?
one last thing - if you're using the patch by david champion to count attachments, it won't work with this patch. this is because mutt doesn't look at the message file at all... so all files show up as if they had one attachment. w
Re: maildir over mbox?
Michael Elkins wrote: > > I'd be curious to get some feedback on my header caching patch for > maildir folders (can be found at http://www.sigpipe.org:8080/mutt/). (thanks to michael for helping me to get this to compile)... anyway finally got this to work. note that you have to put '--enable-cache' in your configure args. also, the libdb2 include file on debian didn't have 'db_create' (which configure uses to test if the db stuff is working); s/db_create/db_open/g fixed that.. so if you have and older berkley db include file, you might need to do that. me figured it out so i think he is going to put something more generic in the patch perhaps. the header caching thing seems to speed things up by about double it does take a fair amount longer the first time (while it's building the db, i assume) zugzug% time /usr/bin/mutt -f mail/Trash-010202/ /usr/bin/mutt -f mail/Trash-010202/ 1.06s user 0.27s system 69% cpu 1.922 total zugzug% time ./src/build/mutt-1.3.26/mutt -f mail/Trash-010202/ ./src/build/mutt-1.3.26/mutt -f mail/Trash-010202/ 2.32s user 0.44s system 87% cpu 3.165 total if i remove the db file, time is: /usr/bin/mutt -f mail/Trash-010202/ 3.15s user 1.06s system 81% cpu 5.163 total this is for a folder with about 6k messages. the subjective time it takes to open is a bit longer than the time reported here. in any event, there's definitely a lot of improvement here. personally i think it would be great if this made it into the main source tree before 1.4, but i realize some people might object to the caching thing. w
Re: maildir over mbox?
On Mon, Jan 21, 2002 at 02:29:22AM, Christian Ordig wrote: > well ... promises of ReiserFS should even tell us "it's optimized for > filesystems holding thousands of small files" ... well, I think it's "it's optimized for... hum, nothing" ;) > > Number of messages: 9089 (mutt-users archive of 2001) 7939 (mutt-users & mutt-dev) > Hardware: P100, 128MB RAM, Socket7 ASUS Mainboard, 2.5GB Western Digital > IDE Harddrive P2-350, 390MB RAM, ASUS Mainboard (P2-B), 20GB Quantum IDE HDD > System: OpenBSD 2.9 Linux 2.4.17 on Slackware 8.0 (the best linux distribution ;p (yep, it's a troll ;p) > Mutt-Version: 1.2.5 compile options: > -DOMAIN > -DEBUG > -HOMESPOOL -USE_SETGID -USE_DOTLOCK -USE_FCNTL +USE_FLOCK > -USE_IMAP -USE_GSS -USE_SSL -USE_POP +HAVE_REGCOMP -USE_GNU_REGEX > +HAVE_COLOR +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS +ENABLE_NLS Mutt 1.3.26i (2002-01-18) Compile options: -DOMAIN -DEBUG -HOMESPOOL -USE_SETGID +USE_DOTLOCK -DL_STANDALONE +USE_FCNTL -USE_FLOCK -USE_POP +USE_IMAP -USE_GSS -USE_SSL -USE_SASL +HAVE_REGCOMP -USE_GNU_REGEX +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT +ENABLE_NLS -LOCALES_HACK +HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR +HAVE_ICONV -ICONV_NONTRANS +HAVE_GETSID +HAVE_GETADDRINFO > Filesystem: UFS, mounted sync ext3 since yesteray ;) > ... strange ... Maybe Linux is a really poor kernel ;) > Christian Ordig ---end quoted text--- cu, binny -- question = ( to ) ? be : ! be; -- Wm. Shakespeare °v° Benjamin Michotte<[EMAIL PROTECTED]> _o_ web : http://www.baby-linux.net msg23403/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
On Mon, Jan 21, 2002 at 02:31:56AM, Christian Ordig wrote: > > little question: cached or first opening ? the first opening takes about 20 seconds (for this 7900 mails) and when it's cached, it takes about 10-15 seconds, which is really more acceptable ;) > Christian Ordig ---end quoted text--- binny -- question = ( to ) ? be : ! be; -- Wm. Shakespeare °v° Benjamin Michotte<[EMAIL PROTECTED]> _o_ web : http://www.baby-linux.net msg23400/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
Derek D. Martin <[EMAIL PROTECTED]> writes: > No manager I've ever worked for would tolerate waiting 3 minutes to > open their inbox... That's funny because where I work, we use Lotus Notes, and I'm sure many managers routinely wait this long for Notes to open their inboxes (particularly if they don't replicate). Remarkable what you can get used to, eh? It's always a relief to come home a use a *real* mail client ;-) Sam "OT--Sorry" Padgett
Re: maildir over mbox?
Derek D. Martin wrote: > > But how does it compare to mbox on the same FS? I'll bet it's still > significantly slower. but with mbox, the entire file has to be stated every time the file is read or modified. with a large file, this can be pretty resource intensive, and can also be time consuming. this is a big problem with POP3 - and this is a big advantage for machines where most users are connecting remotely. Maildir keeps the load lower, and in many cases is faster to boot. your assumption may also be faulty because the problem ext2 has is with opening all those little files; so it doesn't necessarily follow that mbox will also be faster on other filesystems. in other words, Maildir is (i'm pretty sure) inherently faster, but is limited by file system performance. there was an article somewhere (i don't have the link) on this (specifically on the fact that ext2 and Maildir could be slow); i'll see if i can dig up the link. i'm excited to check out me's header caching patch - i didn't realize there was a patch available - so thanks for letting us know! w
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 09:21:44PM -0500, Derek D. Martin wrote: >It's not any safer if you do it RIGHT. In computer science, you want >to tend to optimize for the common case, and the common case when >reading e-mail is wanting quick access. :) Absolutely, I mean in my experience, I'll choise maildir format to separate message from someone, because it's more difficult if your type mailbox is mbox (1 file for many message) compare with maildir (1 message is 1 file). hey it's my opinion OK, I don't know about the other, IMHO..:) -- budsz msg23397/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
Derek D. Martin wrote: > I'm no expert, but it strikes me that OPENING maildir mailboxes on ANY > filesystem will ALWAYS be slower than mbox, because of what you need > to do. An mbox mailbox will generally have little fragmentation on I'd be curious to get some feedback on my header caching patch for maildir folders (can be found at http://www.sigpipe.org:8080/mutt/). Initial reports are that it greatly speeds things up since after the first time you load a mailbox, you only have to traverse the list of files in a directory without actually opening them up for reading. I'm not familiar with the structure of inodes and how fast it is to traverse the list of files in a directory, but I suspect it will be significantly faster than the current method.
Re: maildir over mbox?
At some point hitherto, Will Yardley hath spake thusly: > Christian Ordig wrote: > > > > Filesystem: UFS, mounted sync > [...] > > Are there others having such poor performance with Maildir as Benjamin > > has? And with which filesystem OS combinations? > > our office mail machine is (unfortunately) linux with ext2, and i can > attest to the fact that Maildir is pretty slow on ext2. And most other filesystems... Try it on FAT. =8^) > i haven't tried copying one of my large Maildir folders over to one of > my machines (FreeBSD, UFS filesystem with soft-updates) to see how much > faster it is, but i've heard that the difference is pretty large. But how does it compare to mbox on the same FS? I'll bet it's still significantly slower. -- Derek Martin [EMAIL PROTECTED] - I prefer mail encrypted with PGP/GPG! GnuPG Key ID: 0x81CFE75D Retrieve my public key at http://pgp.mit.edu Learn more about it at http://www.gnupg.org msg23394/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
Andy Davidson wrote: > As a slight aside on this discussion, I had read somewhere --- citation > lost --- that the munging of mboxes to escape lines beginning "From " in a > message to ">From " messed up PGP signing. Is this valid? [I suspect not, > because I see lots of signed messages and you can't *all* be using > maildirs, can you? :-)] Mutt attempts to compensate for this by using quoted-printable encoding when it detects things that might break a signature, thus escaping the problem. But yes, mbox format is more susceptable to corruption of this form.
Re: maildir over mbox?
At some point hitherto, Roman Neuhauser hath spake thusly: > This format can get _very_ slow with large mailboxes on filesystems that do > not handle directoris with many files in them. This should include the > Linux ext2fs. > > FreeBSD post-4.4 FFS with softupdates and dirhash should shine with this > format. I'm no expert, but it strikes me that OPENING maildir mailboxes on ANY filesystem will ALWAYS be slower than mbox, because of what you need to do. An mbox mailbox will generally have little fragmentation on most Unix-like filesystems, so when you open it (in the mailbox sense, as opposed to the filesystem sense, i.e. to get all the headers), it will be something like this: open read read read read close Whereas a maildir format mailbox has many small files, which could be anywhere on the disk partition. So when you open one, you have to go through these contortions: open read close seek (the disk heads, not the syscall) open read close seek open read close seek I can't see how this could ever be fast... -- Derek Martin [EMAIL PROTECTED] - I prefer mail encrypted with PGP/GPG! GnuPG Key ID: 0x81CFE75D Retrieve my public key at http://pgp.mit.edu Learn more about it at http://www.gnupg.org msg23392/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
Christian Ordig wrote: > > Filesystem: UFS, mounted sync [...] > Are there others having such poor performance with Maildir as Benjamin > has? And with which filesystem OS combinations? our office mail machine is (unfortunately) linux with ext2, and i can attest to the fact that Maildir is pretty slow on ext2. i haven't tried copying one of my large Maildir folders over to one of my machines (FreeBSD, UFS filesystem with soft-updates) to see how much faster it is, but i've heard that the difference is pretty large. if you look at past threads, you'll see a lot of mention of Maildir being slow; this is usually from linux-centric people who don't realize the problem is with ext2 :> w
Re: maildir over mbox?
At some point hitherto, budsz hath spake thusly: > >yes, I know. I tried to convert my mbox to Maildirs, but about 3 minutes > >to open a folder is really awfull, so I keep mbox > > If we look in speed to read right..? how about savety...? let's say I want > to copy paste 1000 email to some place, I think maildir format is better > (could separate each other message). It's not any safer if you do it RIGHT. In computer science, you want to tend to optimize for the common case, and the common case when reading e-mail is wanting quick access. :) No manager I've ever worked for would tolerate waiting 3 minutes to open their inbox... -- Derek Martin [EMAIL PROTECTED] - I prefer mail encrypted with PGP/GPG! GnuPG Key ID: 0x81CFE75D Retrieve my public key at http://pgp.mit.edu Learn more about it at http://www.gnupg.org msg23390/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
As a slight aside on this discussion, I had read somewhere --- citation lost --- that the munging of mboxes to escape lines beginning "From " in a message to ">From " messed up PGP signing. Is this valid? [I suspect not, because I see lots of signed messages and you can't *all* be using maildirs, can you? :-)] andy Andy Davidson --- Pheon Research If you know what you're doing, you're not learning anything.
Re: maildir over mbox?
> Oh my good... convert my reiserfs partition to ext3... > about 10-15 seconds to open my mutt Maildir now !!! ooops ... I should have already read this mail before answering the last subthread ,-) little question: cached or first opening ? > Absolutly... reiserfs sucks. *g* -- Christian Ordig Germany
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 09:39:05PM +0100, Benjamin Michotte wrote: > On Sun, Jan 20, 2002 at 09:01:39PM, Christian Ordig wrote: > > uhhh ... what kind of system did you use for measurement?? > P2-350 with a 20Gb HDD running Linux 2.4.17 on a Slackware 8.0. > My ~/mail is on a 600 Mb reiserfs partition. I think I will convert my > /home dir to ext3 and then try Maildir to compare. well ... promises of ReiserFS should even tell us "it's optimized for filesystems holding thousands of small files" ... > > on my P100 with a quite old HDD running OpenBSD 2.9 it takes > > about one minute to open the whole mutt archive of last year > > (about 9900 messages)... > It takes about 7 seconds to open my mutt archive with 7914 messages. ok this system should be quite faster per default than the stuff I use, but should UFS be really _that_ faster? I took a watch and measured time. First opening the folder (sorting by threads and displaying) takes 57seconds. (in words: fifty-seven :-) Number of messages: 9089 (mutt-users archive of 2001) Hardware: P100, 128MB RAM, Socket7 ASUS Mainboard, 2.5GB Western Digital IDE Harddrive System: OpenBSD 2.9 Mutt-Version: 1.2.5 compile options: -DOMAIN -DEBUG -HOMESPOOL -USE_SETGID -USE_DOTLOCK -USE_FCNTL +USE_FLOCK -USE_IMAP -USE_GSS -USE_SSL -USE_POP +HAVE_REGCOMP -USE_GNU_REGEX +HAVE_COLOR +HAVE_PGP -BUFFY_SIZE -EXACT_ADDRESS +ENABLE_NLS Filesystem: UFS, mounted sync > yes, I know. I tried to convert my mbox to Maildirs, but about 3 minutes > to open a folder is really awfull, so I keep mbox sure it is... but I think it should really be faster with your hardware, shouldn't it? (I just guess so, since the HDD alone should already be faster than my old IDE drive ...) ... strange ... maybe I'll copy a big folder over to my Dual PIII system with a UDMA100 drive having Linux and Ext2 and verify the poor results. Are there others having such poor performance with Maildir as Benjamin has? And with which filesystem OS combinations? confused greetings ... :-) -- Christian Ordig Germany
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 09:39:05PM +0100, Benjamin Michotte wrote: >On Sun, Jan 20, 2002 at 09:01:39PM, Christian Ordig wrote: >> uhhh ... what kind of system did you use for measurement?? >P2-350 with a 20Gb HDD running Linux 2.4.17 on a Slackware 8.0. >My ~/mail is on a 600 Mb reiserfs partition. I think I will convert my >/home dir to ext3 and then try Maildir to compare. > >> on my P100 with a quite old HDD running OpenBSD 2.9 it takes >> about one minute to open the whole mutt archive of last year >> (about 9900 messages)... >It takes about 7 seconds to open my mutt archive with 7914 messages. > >> the advantage of Maildir over mbox I see: deleting a single message >> in a really big mailbox is nothing more than simply deleting one >> file with mbox it means writing the whole folder again leaving out >> this one message . >yes, I know. I tried to convert my mbox to Maildirs, but about 3 minutes >to open a folder is really awfull, so I keep mbox If we look in speed to read right..? how about savety...? let's say I want to copy paste 1000 email to some place, I think maildir format is better (could separate each other message). -- budsz msg23384/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 07:33:10PM, Benjamin Michotte wrote: > > What are the benefits of using one type over the other? > opening a mbox with ± 7000 mails : less than 10 seconds. > opening the same in Maildir : 3 minutes... Oh my good... convert my reiserfs partition to ext3... about 10-15 seconds to open my mutt Maildir now !!! Absolutly... reiserfs sucks. > ---end quoted text--- cu, binny which is going to convert all his HDD to ext3 now ;) -- question = ( to ) ? be : ! be; -- Wm. Shakespeare °v° Benjamin Michotte<[EMAIL PROTECTED]> _o_ web : http://www.baby-linux.net msg23380/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
> Date: Sun, 20 Jan 2002 12:08:44 -0600 > From: David Rock <[EMAIL PROTECTED]> > To: Mutt Users List <[EMAIL PROTECTED]> > Subject: maildir over mbox? > > I was just wondering what the real differences were between maildir and > mbox formats? I know mbox is an appended file while maildir is a > separate directory for each mail (each what, exactly)? > > What are the benefits of using one type over the other? mbox This is the traditional mailbox format. It is a simple plain text file with the messages in it. Each message has this structure: From [EMAIL PROTECTED] date Headers Body Since MUAs recognize the beginning of a message in an mbox by a line starting with "From " (this is commonly called "the From_ line"), any lines in message bodies starting with this string will get rewritten as ">From " by your MDA. It is not a good idea to use this mailbox format if you have your mail on a NFS share. Many NFS implementations have buggy locking, and you can easily have your mailboxes stomped. See mbox(5) for more info (different man pages installed at least with mutt and qmail). maildir This is a newer format, introduced by the qmail MTA. Each mailbox is a directory containing three subdirectories: tmp/, new/, and cur/. When your MDA delivers a message to a maildir mailbox, it writes it to the tmp/ directory, and when the message is succesfully written, it moves it to the new/ directory. Since this is atomical operation, the mailbox is safe from curruption. It is quite safe to use over NFS. Also, it can be easier when you want to process your mail using regular tools provided in the base system: you can use a small shell script to move old mail away from your "active" mailboxes to an "archive" using find(1), xargs(1), and alike. This format can get _very_ slow with large mailboxes on filesystems that do not handle directoris with many files in them. This should include the Linux ext2fs. FreeBSD post-4.4 FFS with softupdates and dirhash should shine with this format. See maildir(5) (installed at least with qmail) for more info. -- FreeBSD 4.4-STABLE 9:47PM up 4:11, 8 users, load averages: 2.25, 2.44, 2.60
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 09:01:39PM, Christian Ordig wrote: > uhhh ... what kind of system did you use for measurement?? P2-350 with a 20Gb HDD running Linux 2.4.17 on a Slackware 8.0. My ~/mail is on a 600 Mb reiserfs partition. I think I will convert my /home dir to ext3 and then try Maildir to compare. > on my P100 with a quite old HDD running OpenBSD 2.9 it takes > about one minute to open the whole mutt archive of last year > (about 9900 messages)... It takes about 7 seconds to open my mutt archive with 7914 messages. > the advantage of Maildir over mbox I see: deleting a single message > in a really big mailbox is nothing more than simply deleting one > file with mbox it means writing the whole folder again leaving out > this one message . yes, I know. I tried to convert my mbox to Maildirs, but about 3 minutes to open a folder is really awfull, so I keep mbox > Christian Ordig ---end quoted text--- binny -- question = ( to ) ? be : ! be; -- Wm. Shakespeare °v° Benjamin Michotte<[EMAIL PROTECTED]> _o_ web : http://www.baby-linux.net msg23376/pgp0.pgp Description: PGP signature
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 07:33:10PM +0100, Benjamin Michotte wrote: > opening a mbox with ± 7000 mails : less than 10 seconds. > opening the same in Maildir : 3 minutes... uhhh ... what kind of system did you use for measurement?? on my P100 with a quite old HDD running OpenBSD 2.9 it takes about one minute to open the whole mutt archive of last year (about 9900 messages)... the advantage of Maildir over mbox I see: deleting a single message in a really big mailbox is nothing more than simply deleting one file with mbox it means writing the whole folder again leaving out this one message . Not to tell the locking which is obsolete with Maildir, which makes it easier with Mailfolders over NFS as I also use it ;-) -- Christian Ordig Germany
Re: maildir over mbox?
On Sun, Jan 20, 2002 at 12:08:44PM -0600, David Rock <[EMAIL PROTECTED]> wrote: > I was just wondering what the real differences were between maildir and > mbox formats? I know mbox is an appended file while maildir is a > separate directory for each mail (each what, exactly)? > > What are the benefits of using one type over the other? > > Thanks. Maildir has the significant advantage that it requires no locking, making it safer to use in environments without good locking primitives, and that your whole mailbox can never get corrupted by one delivery going awry, since each message lives in its own file. Having had my inbox mysteriously corrupted once years ago, I think that that is a significant advantage. Maildir has the disadvantages that it is nonstandard (not really a problem -- you can very easily save everything in a maildir mailbox to an mbox mailbox if you want to give it to someone whose mailer doesn't understand maildir) and that it takes up more space on disk, since even a tiny message will use up a whole disk block. Speed can go either way, depending on the filesystem. Traditional unix filesystems are very slow at reading huge directories. Perhaps mutt should support maildirs split up into subdirectories for this reason? But it doesn't now. -Daniel -- Daniel E. Eisenbud [EMAIL PROTECTED] "We should go forth on the shortest walk perchance, in the spirit of undying adventure, never to return,--prepared to send back our embalmed hearts only as relics to our desolate kingdoms." --Henry David Thoreau, "Walking"
Re: maildir over mbox?
Hi, On Sun, Jan 20, 2002 at 12:08:44PM, David Rock wrote: > I was just wondering what the real differences were between maildir and > mbox formats? I know mbox is an appended file while maildir is a > separate directory for each mail (each what, exactly)? one folder for a box, each mail in a separate file. > What are the benefits of using one type over the other? opening a mbox with ± 7000 mails : less than 10 seconds. opening the same in Maildir : 3 minutes... > David Rock ---end quoted text--- cu, binny -- question = ( to ) ? be : ! be; -- Wm. Shakespeare °v° Benjamin Michotte<[EMAIL PROTECTED]> _o_ web : http://www.baby-linux.net msg23372/pgp0.pgp Description: PGP signature
maildir over mbox?
I was just wondering what the real differences were between maildir and mbox formats? I know mbox is an appended file while maildir is a separate directory for each mail (each what, exactly)? What are the benefits of using one type over the other? Thanks. -- David Rock [EMAIL PROTECTED] msg23371/pgp0.pgp Description: PGP signature
