Re: [OT] maildir (was Re: procmail and Large File Support)
On Monday 28 February 2005 14:26, sean finney [EMAIL PROTECTED] wrote: i came up with the number by totalling the mailbox sizes of a 3000 user mail system, and then dividing by the total number of messages in these mailboxes. this generated a number around 13k average message size. i had to do this as part of assessing the feasability of migrating to maildir without reformatting the filesystem. A couple of years ago I did the same thing on a system with over a million users and got much the same result. I thought it was illegal to modify a message. marking a message as read is one example. moving a message from one mailbox to another is another example. although it's not modifying the message itself, it's moving its location, which with a crappy imap server can mean re-writing the contents of two mailboxes. In most jurisdictions it's legal to do almost anything as long as the users are informed in advance. Anyway this list is about solving technical problems not being bothered with laws in some strange part of the world. On Monday 28 February 2005 19:18, Michelle Konzack [EMAIL PROTECTED] wrote: Mailbox is MUCH slower as Maildir, because it must be scaned entierly, but with maildir, most you can spead up the searches while scaning only the Headers. Of course this only applies if there are a significant number of messages larger than the file system block size (usually 4K). If you have a maildir in which every message is less than 4K in size you may find that scanning it is slower than scanning an mbox with the same data. The kernel can do read-ahead for a large file to improve performance. Also an application can do read-ahead (calling setbuf() with a large buffer would be one way to do it). Usually there are a significant number of messages 4K so this is the case. Also Maildir wins on all modifications to the mail store other than adding new messages. Another noteworthy thing about Maildir is that when an application messes up it will probably only trash one message. I use Maildir for my Kmail local storage for this reason, I've had problems in the past with Kmail crashing and corrupting mbox storage. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
Am 2005-02-27 18:19:45, schrieb sean finney: can't help but chime in here :) On Mon, Feb 28, 2005 at 09:22:30AM +1100, Brian May wrote: Not every situation warrants using maildir, it uses a large number of inodes, is slow to scan (yes, mbox isn't very good either), Mailbox is MUCH slower as Maildir, because it must be scaned entierly, but with maildir, most you can spead up the searches while scaning only the Headers. inefficient at storing large number of very small files (due to block size limitations of file system), and more complicated to transfer/move/share. What is complicate ? You need only the right programs... it does use a large number of inodes, but i've found that even on large filesystems with many users, there's not a real risk of starving the fs of inodes. ymmv. i'm not sure about the transferring/moving/sharing though. figuring the average email is about 13-15k, i believe an ext2/ext3 filesystem created with default options would fill up before running out of inodes. I have striped the Messages by Received: Headers and the most Messages are under 4 kByte now. I have a Mailarchive from around 130 Mailinglists with 5,3 Million Messages and my ext3 Filesystem has up to 18.000.000 Inodes and a blocksize of 1 kByte. I had never problems with it. Also I have only one Mailfolder per Mailinglist (linux-kernel has for example more then 190.000 Messges in it.) Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many of these disadvantages. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the individual file, as opposed to the entire mailbox. in some of the sloppier imap servers (*cough* uw-imap *cough* *cough*), this can cause huge, grind-your-server-to-a-halt performance hits as deleting, or merely reading a new message necessates a huge amount of i/o. Right. I use courier and it works perfectly with Maildir... No blocking or high load, even if I open m linux-kernel Mailbox sean Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
Am 2005-02-27 20:19:09, schrieb Ron Johnson: Sure, for those *20* GB mbox files. Who has 20 GByte mailboxes ? - It is realy braindamaged... Even on xfs, open a 20 GByte Mailbox will eat up all resources on the System Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Mon, 2005-02-28 at 09:25 +0100, Michelle Konzack wrote: Am 2005-02-27 20:19:09, schrieb Ron Johnson: Sure, for those *20* GB mbox files. Who has 20 GByte mailboxes ? - It is realy braindamaged... The same person with the 2GB mbox that started this thread, after s/he neglected it for a few more months. Even on xfs, open a 20 GByte Mailbox will eat up all resources on the System Guess you'd better use Maildir, then, huh? ;) -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive. LUKE: But how will I know why Python is better than Perl? YODA: You will know. When your code you try to read six months from now. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
Am 2005-02-28 02:43:45, schrieb Ron Johnson: On Mon, 2005-02-28 at 09:25 +0100, Michelle Konzack wrote: Who has 20 GByte mailboxes ? - It is realy braindamaged... The same person with the 2GB mbox that started this thread, after s/he neglected it for a few more months. :-/ Oh yes, the SPAM/Virus folder. Then try to check it for false-positives... :-) Even on xfs, open a 20 GByte Mailbox will eat up all resources on the System Guess you'd better use Maildir, then, huh? ;) I have no problems with my huge Maildir :-) 'mutt' open the 187.000 Messages in around 47 seconds local (/home mounted via nfs v3) and via courier-imap-ssl in around 30 seconds. I know the avantages of Maildir. Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Monday 28 February 2005 01:51, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: [snip] figuring the average email is about 13-15k, i believe an ext2/ext3 That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. I ran statistics on maildirs of the university (of arts) mailserver I administer: ~90k per mail. Regards, David -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir ber ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: [OT] maildir (was Re: procmail and Large File Support)
On Mon, 2005-02-28 at 22:55 +0100, David Schmitt wrote: On Monday 28 February 2005 01:51, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: [snip] figuring the average email is about 13-15k, i believe an ext2/ext3 That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. I ran statistics on maildirs of the university (of arts) mailserver I administer: ~90k per mail. Art majors passing around scanned pictures of croissants? ;) -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. When asked to name the chief qualification a politician should have. It's the ability to foretell what will happen tomorrow, next month, and next year --- and to explain afterward why it didn't happen. Sir Winston Churchill -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
[OT] maildir (was Re: procmail and Large File Support)
can't help but chime in here :) On Mon, Feb 28, 2005 at 09:22:30AM +1100, Brian May wrote: Not every situation warrants using maildir, it uses a large number of inodes, is slow to scan (yes, mbox isn't very good either), inefficient at storing large number of very small files (due to block size limitations of file system), and more complicated to transfer/move/share. it does use a large number of inodes, but i've found that even on large filesystems with many users, there's not a real risk of starving the fs of inodes. ymmv. i'm not sure about the transferring/moving/sharing though. figuring the average email is about 13-15k, i believe an ext2/ext3 filesystem created with default options would fill up before running out of inodes. Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many of these disadvantages. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the individual file, as opposed to the entire mailbox. in some of the sloppier imap servers (*cough* uw-imap *cough* *cough*), this can cause huge, grind-your-server-to-a-halt performance hits as deleting, or merely reading a new message necessates a huge amount of i/o. sean -- signature.asc Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: can't help but chime in here :) On Mon, Feb 28, 2005 at 09:22:30AM +1100, Brian May wrote: [snip] figuring the average email is about 13-15k, i believe an ext2/ext3 That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. filesystem created with default options would fill up before running out of inodes. Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many Reiserfs, of course. of these disadvantages. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. He was about as useful in a crisis as a sheep. Dorothy Eden -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. Status: O? -- --- Paul TBBle Hampson, MCSE 8th year CompSci/Asian Studies student, ANU The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361) [EMAIL PROTECTED] No survivors? Then where do the stories come from I wonder? -- Capt. Jack Sparrow, Pirates of the Caribbean This email is licensed to the recipient for non-commercial use, duplication and distribution. --- signature.asc Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Mon, 2005-02-28 at 11:54 +1100, Paul Hampson wrote: On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. Status: O? I don't know what that means. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. I take my children everywhere, but they always find their way back home. Robert Orben -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many Reiserfs, of course. You meant XFS, right? (Sorry, couldn't be helped. :) -- Glenn Maynard -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, 2005-02-27 at 20:54 -0500, Glenn Maynard wrote: On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many Reiserfs, of course. You meant XFS, right? (Sorry, couldn't be helped. :) Sure, for those *20* GB mbox files. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. Rightly hating violence, [pacifists] do not wish to recognise that it is integral to modern society and that their own fine feelings and noble attitudes are all the fruit of injustice backed up by force. They do not want to learn where their incomes come from. George Orwell -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. i came up with the number by totalling the mailbox sizes of a 3000 user mail system, and then dividing by the total number of messages in these mailboxes. this generated a number around 13k average message size. i had to do this as part of assessing the feasability of migrating to maildir without reformatting the filesystem. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. marking a message as read is one example. moving a message from one mailbox to another is another example. although it's not modifying the message itself, it's moving its location, which with a crappy imap server can mean re-writing the contents of two mailboxes. sean -- signature.asc Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, 2005-02-27 at 22:26 -0500, sean finney wrote: On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. i came up with the number by totalling the mailbox sizes of a 3000 user mail system, and then dividing by the total number of messages in these mailboxes. this generated a number around 13k average message size. i had to do this as part of assessing the feasability of migrating to maildir without reformatting the filesystem. Wow. Lot's of html and lots of attachments. It might also be useful to calculate the mode and standard deviation. Why? Really big attachments *might* be skewing the average. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. marking a message as read is one example. moving a message from one mailbox to another is another example. although it's not modifying the message itself, it's moving its location, which with a crappy imap server can mean re-writing the contents of two mailboxes. *cough* wu- *cough* ;) -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. Lead the people with governmental measures and regulate them by law and punishments, and they will avoid wrongdoing, but will have no sense of honor and shame. Lead them by virtue and regulate them by the rules of propriety and they will have a sense of shame and, moreover, set themselves right. Confucius -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
