Re: [OT] maildir (was Re: procmail and Large File Support)
On Monday 28 February 2005 14:26, sean finney [EMAIL PROTECTED] wrote: i came up with the number by totalling the mailbox sizes of a 3000 user mail system, and then dividing by the total number of messages in these mailboxes. this generated a number around 13k average message size. i had to do this as part of assessing the feasability of migrating to maildir without reformatting the filesystem. A couple of years ago I did the same thing on a system with over a million users and got much the same result. I thought it was illegal to modify a message. marking a message as read is one example. moving a message from one mailbox to another is another example. although it's not modifying the message itself, it's moving its location, which with a crappy imap server can mean re-writing the contents of two mailboxes. In most jurisdictions it's legal to do almost anything as long as the users are informed in advance. Anyway this list is about solving technical problems not being bothered with laws in some strange part of the world. On Monday 28 February 2005 19:18, Michelle Konzack [EMAIL PROTECTED] wrote: Mailbox is MUCH slower as Maildir, because it must be scaned entierly, but with maildir, most you can spead up the searches while scaning only the Headers. Of course this only applies if there are a significant number of messages larger than the file system block size (usually 4K). If you have a maildir in which every message is less than 4K in size you may find that scanning it is slower than scanning an mbox with the same data. The kernel can do read-ahead for a large file to improve performance. Also an application can do read-ahead (calling setbuf() with a large buffer would be one way to do it). Usually there are a significant number of messages 4K so this is the case. Also Maildir wins on all modifications to the mail store other than adding new messages. Another noteworthy thing about Maildir is that when an application messes up it will probably only trash one message. I use Maildir for my Kmail local storage for this reason, I've had problems in the past with Kmail crashing and corrupting mbox storage. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Sat, Mar 19, 2005 at 09:54:09AM -0600, Gunnar Wolf wrote: Ola Lundqvist dijo [Wed, Mar 16, 2005 at 09:18:33PM +0100]: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. It will grow quite fast. Ummm... And wouldn't it make more sense for them to switch to maildir instead of mbox? I wouldn't like to search for new mails in there. Of course. We use maildir for all accounts. I was just commenting on the fact that 2 GB mailboxes is needed, and quite often. Regards, // Ola -- Gunnar Wolf - [EMAIL PROTECTED] - (+52-55)1451-2244 / 5554-9450 PGP key 1024D/8BB527AF 2001-10-23 Fingerprint: 0C79 D2D1 2C4E 9CE4 5973 F800 D80E F35A 8BB5 27AF -- --- Ola Lundqvist systemkonsult --- M Sc in IT Engineering / [EMAIL PROTECTED] Annebergsslingan 37\ | [EMAIL PROTECTED] 654 65 KARLSTAD| | http://www.opal.dhs.org Mobile: +46 (0)70-332 1551 | \ gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9 / --- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
Ola Lundqvist dijo [Wed, Mar 16, 2005 at 09:18:33PM +0100]: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. It will grow quite fast. Ummm... And wouldn't it make more sense for them to switch to maildir instead of mbox? I wouldn't like to search for new mails in there. -- Gunnar Wolf - [EMAIL PROTECTED] - (+52-55)1451-2244 / 5554-9450 PGP key 1024D/8BB527AF 2001-10-23 Fingerprint: 0C79 D2D1 2C4E 9CE4 5973 F800 D80E F35A 8BB5 27AF -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
Scripsit Gunnar Wolf [EMAIL PROTECTED] Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. It will grow quite fast. Ummm... And wouldn't it make more sense for them to switch to maildir instead of mbox? I wouldn't like to search for new mails in there. Woah... deja vu. -- Henning MakholmI ... I have to return some videos. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Sat, 2005-03-19 at 09:54 -0600, Gunnar Wolf wrote: Ola Lundqvist dijo [Wed, Mar 16, 2005 at 09:18:33PM +0100]: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. It will grow quite fast. Ummm... And wouldn't it make more sense for them to switch to maildir instead of mbox? I wouldn't like to search for new mails in there. But, but, but, we *love* using 6GB mbox files ! -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. No drug, not even alcohol, causes the fundamental ills of society. If we're looking for the sources of our troubles, we shouldn't test people for drugs, we should test them for stupidity, ignorance, greed and love of power. P. J. O'Rourke signature.asc Description: This is a digitally signed message part
Re: procmail and Large File Support
Am 2005-03-16 21:18:33, schrieb Ola Lundqvist: Hello Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. It will grow quite fast. You mean some of this Mailnglist like [EMAIL PROTECTED] ? Oh yes, there are some realy friendly people which had me subscribed to a couple of this Mailinglists... goten more then 18000 Messages in two weeks vacancy... around 500 kByte/message... Fortunatly I am using Maildir :-) Regards, // Ola Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: procmail and Large File Support
On Wed, 2005-03-16 at 21:45 +0100, Pierre Habouzit wrote: Le Mer 16 Mars 2005 21:36, Ron Johnson a écrit : On Wed, 2005-03-16 at 21:18 +0100, Ola Lundqvist wrote: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: [snip] File quotas will fix that in a hurry. that's definetely a (sorry) stupid answer. Because anyway, it feels not very sensible to use mbox flat file and not maildir for such big mailboxes ;) We agree. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. If trees could scream, would we be so cavalier about cutting them down? We might, if they screamed all the time, for no good reason. Jack Handey signature.asc Description: This is a digitally signed message part
Re: procmail and Large File Support
Hello On Wed, Mar 16, 2005 at 02:36:21PM -0600, Ron Johnson wrote: On Wed, 2005-03-16 at 21:18 +0100, Ola Lundqvist wrote: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. File quotas will fix that in a hurry. Not necessary if they pay for their usage. :) Regards, // Ola It will grow quite fast. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. Give a man a fish, you feed him for a day, teach him to fish, he gets mad at you for making him have to work so hard. -- - Ola Lundqvist --- / [EMAIL PROTECTED] Annebergsslingan 37 \ | [EMAIL PROTECTED] 654 65 KARLSTAD | | +46 (0)54-10 14 30 +46 (0)70-332 1551 | | http://www.opal.dhs.org UIN/icq: 4912500 | \ gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9 / --- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Thu, 2005-03-17 at 20:12 +0100, Ola Lundqvist wrote: Hello On Wed, Mar 16, 2005 at 02:36:21PM -0600, Ron Johnson wrote: On Wed, 2005-03-16 at 21:18 +0100, Ola Lundqvist wrote: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. File quotas will fix that in a hurry. Not necessary if they pay for their usage. :) Well, I gotta give you that... -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. I haven't committed a crime. What I did was fail to comply with the law. David Dinkins signature.asc Description: This is a digitally signed message part
Re: procmail and Large File Support
Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. It will grow quite fast. Regards, // Ola -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive. LUKE: But how will I know why Python is better than Perl? YODA: You will know. When your code you try to read six months from now. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- - Ola Lundqvist --- / [EMAIL PROTECTED] Annebergsslingan 37 \ | [EMAIL PROTECTED] 654 65 KARLSTAD | | +46 (0)54-10 14 30 +46 (0)70-332 1551 | | http://www.opal.dhs.org UIN/icq: 4912500 | \ gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9 / --- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Wed, 2005-03-16 at 21:18 +0100, Ola Lundqvist wrote: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. File quotas will fix that in a hurry. It will grow quite fast. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. Give a man a fish, you feed him for a day, teach him to fish, he gets mad at you for making him have to work so hard. signature.asc Description: This is a digitally signed message part
Re: procmail and Large File Support
Le Mer 16 Mars 2005 21:36, Ron Johnson a crit : On Wed, 2005-03-16 at 21:18 +0100, Ola Lundqvist wrote: Hello On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Some people tend to have really large inboxes. I have had a number of customers that have several GB inbox. They tend to get quite a lot of attachments (reports etc) and do not have the time to delete mail. File quotas will fix that in a hurry. that's definetely a (sorry) stupid answer. anyway, it feels not very sensible to use mbox flat file and not maildir for such big mailboxes ;) -- O Pierre Habouzit O OOOhttp://www.madism.org pgpDscTu7I9Fr.pgp Description: PGP signature
Re: [OT] maildir (was Re: procmail and Large File Support)
Am 2005-02-27 18:19:45, schrieb sean finney: can't help but chime in here :) On Mon, Feb 28, 2005 at 09:22:30AM +1100, Brian May wrote: Not every situation warrants using maildir, it uses a large number of inodes, is slow to scan (yes, mbox isn't very good either), Mailbox is MUCH slower as Maildir, because it must be scaned entierly, but with maildir, most you can spead up the searches while scaning only the Headers. inefficient at storing large number of very small files (due to block size limitations of file system), and more complicated to transfer/move/share. What is complicate ? You need only the right programs... it does use a large number of inodes, but i've found that even on large filesystems with many users, there's not a real risk of starving the fs of inodes. ymmv. i'm not sure about the transferring/moving/sharing though. figuring the average email is about 13-15k, i believe an ext2/ext3 filesystem created with default options would fill up before running out of inodes. I have striped the Messages by Received: Headers and the most Messages are under 4 kByte now. I have a Mailarchive from around 130 Mailinglists with 5,3 Million Messages and my ext3 Filesystem has up to 18.000.000 Inodes and a blocksize of 1 kByte. I had never problems with it. Also I have only one Mailfolder per Mailinglist (linux-kernel has for example more then 190.000 Messges in it.) Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many of these disadvantages. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the individual file, as opposed to the entire mailbox. in some of the sloppier imap servers (*cough* uw-imap *cough* *cough*), this can cause huge, grind-your-server-to-a-halt performance hits as deleting, or merely reading a new message necessates a huge amount of i/o. Right. I use courier and it works perfectly with Maildir... No blocking or high load, even if I open m linux-kernel Mailbox sean Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
Am 2005-02-27 20:19:09, schrieb Ron Johnson: Sure, for those *20* GB mbox files. Who has 20 GByte mailboxes ? - It is realy braindamaged... Even on xfs, open a 20 GByte Mailbox will eat up all resources on the System Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Mon, 2005-02-28 at 09:25 +0100, Michelle Konzack wrote: Am 2005-02-27 20:19:09, schrieb Ron Johnson: Sure, for those *20* GB mbox files. Who has 20 GByte mailboxes ? - It is realy braindamaged... The same person with the 2GB mbox that started this thread, after s/he neglected it for a few more months. Even on xfs, open a 20 GByte Mailbox will eat up all resources on the System Guess you'd better use Maildir, then, huh? ;) -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive. LUKE: But how will I know why Python is better than Perl? YODA: You will know. When your code you try to read six months from now. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
Am 2005-02-28 02:43:45, schrieb Ron Johnson: On Mon, 2005-02-28 at 09:25 +0100, Michelle Konzack wrote: Who has 20 GByte mailboxes ? - It is realy braindamaged... The same person with the 2GB mbox that started this thread, after s/he neglected it for a few more months. :-/ Oh yes, the SPAM/Virus folder. Then try to check it for false-positives... :-) Even on xfs, open a 20 GByte Mailbox will eat up all resources on the System Guess you'd better use Maildir, then, huh? ;) I have no problems with my huge Maildir :-) 'mutt' open the 187.000 Messages in around 47 seconds local (/home mounted via nfs v3) and via courier-imap-ssl in around 30 seconds. I know the avantages of Maildir. Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/8845235667100 Strasbourg/France IRC #Debian (irc.icq.com) signature.pgp Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Monday 28 February 2005 01:51, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: [snip] figuring the average email is about 13-15k, i believe an ext2/ext3 That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. I ran statistics on maildirs of the university (of arts) mailserver I administer: ~90k per mail. Regards, David -- - hallo... wie gehts heute? - *hust* gut *rotz* *keuch* - gott sei dank kommunizieren wir ber ein septisches medium ;) -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Re: [OT] maildir (was Re: procmail and Large File Support)
On Mon, 2005-02-28 at 22:55 +0100, David Schmitt wrote: On Monday 28 February 2005 01:51, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: [snip] figuring the average email is about 13-15k, i believe an ext2/ext3 That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. I ran statistics on maildirs of the university (of arts) mailserver I administer: ~90k per mail. Art majors passing around scanned pictures of croissants? ;) -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. When asked to name the chief qualification a politician should have. It's the ability to foretell what will happen tomorrow, next month, and next year --- and to explain afterward why it didn't happen. Sir Winston Churchill -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
Colin == Colin Watson [EMAIL PROTECTED] writes: Colin Not everyone likes maildir; I gave up on it after Colin experimenting with it and realising that making it harder Colin for myself to use standard Unix text-processing tools on my Colin mailboxes was just too annoying. This is true of spam-bin Colin folders more than other folders, if anything. Not every situation warrants using maildir, it uses a large number of inodes, is slow to scan (yes, mbox isn't very good either), inefficient at storing large number of very small files (due to block size limitations of file system), and more complicated to transfer/move/share. Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many of these disadvantages. The major benefits of Maildir, IIRC, seem to be: * No locking required for mail delivery. Not all folders require this. * More robust separation of messages (only an issue because mbox. standards are so pathetic and everybody has there own ideas what constitutes a reliable method to detect the end of a message). So what appears to be to separate messages contained in mbox format in mutt can be interpreted as only one message by formail (see bug #295604 for example). -- Brian May [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
[OT] maildir (was Re: procmail and Large File Support)
can't help but chime in here :) On Mon, Feb 28, 2005 at 09:22:30AM +1100, Brian May wrote: Not every situation warrants using maildir, it uses a large number of inodes, is slow to scan (yes, mbox isn't very good either), inefficient at storing large number of very small files (due to block size limitations of file system), and more complicated to transfer/move/share. it does use a large number of inodes, but i've found that even on large filesystems with many users, there's not a real risk of starving the fs of inodes. ymmv. i'm not sure about the transferring/moving/sharing though. figuring the average email is about 13-15k, i believe an ext2/ext3 filesystem created with default options would fill up before running out of inodes. Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many of these disadvantages. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the individual file, as opposed to the entire mailbox. in some of the sloppier imap servers (*cough* uw-imap *cough* *cough*), this can cause huge, grind-your-server-to-a-halt performance hits as deleting, or merely reading a new message necessates a huge amount of i/o. sean -- signature.asc Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: can't help but chime in here :) On Mon, Feb 28, 2005 at 09:22:30AM +1100, Brian May wrote: [snip] figuring the average email is about 13-15k, i believe an ext2/ext3 That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. filesystem created with default options would fill up before running out of inodes. Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many Reiserfs, of course. of these disadvantages. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. He was about as useful in a crisis as a sheep. Dorothy Eden -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. Status: O? -- --- Paul TBBle Hampson, MCSE 8th year CompSci/Asian Studies student, ANU The Boss, Bubblesworth Pty Ltd (ABN: 51 095 284 361) [EMAIL PROTECTED] No survivors? Then where do the stories come from I wonder? -- Capt. Jack Sparrow, Pirates of the Caribbean This email is licensed to the recipient for non-commercial use, duplication and distribution. --- signature.asc Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Mon, 2005-02-28 at 11:54 +1100, Paul Hampson wrote: On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: On Sun, 2005-02-27 at 18:19 -0500, sean finney wrote: recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. Status: O? I don't know what that means. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. I take my children everywhere, but they always find their way back home. Robert Orben -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many Reiserfs, of course. You meant XFS, right? (Sorry, couldn't be helped. :) -- Glenn Maynard -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, 2005-02-27 at 20:54 -0500, Glenn Maynard wrote: On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: Of course, all of these factors depend on the file system used. I am confident somebody could point out a file-system that eliminates many Reiserfs, of course. You meant XFS, right? (Sorry, couldn't be helped. :) Sure, for those *20* GB mbox files. -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. Rightly hating violence, [pacifists] do not wish to recognise that it is integral to modern society and that their own fine feelings and noble attitudes are all the fruit of injustice backed up by force. They do not want to learn where their incomes come from. George Orwell -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. i came up with the number by totalling the mailbox sizes of a 3000 user mail system, and then dividing by the total number of messages in these mailboxes. this generated a number around 13k average message size. i had to do this as part of assessing the feasability of migrating to maildir without reformatting the filesystem. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. marking a message as read is one example. moving a message from one mailbox to another is another example. although it's not modifying the message itself, it's moving its location, which with a crappy imap server can mean re-writing the contents of two mailboxes. sean -- signature.asc Description: Digital signature
Re: [OT] maildir (was Re: procmail and Large File Support)
On Sun, 2005-02-27 at 22:26 -0500, sean finney wrote: On Sun, Feb 27, 2005 at 06:51:32PM -0600, Ron Johnson wrote: That seems awfully huge. In my (Maildir) archive of d-u, the average size is 4,959 bytes. Of course, there are no html mails. Though, even in my Evolution list archive, where there are many more html-mails, the average size is only 6,097. i came up with the number by totalling the mailbox sizes of a 3000 user mail system, and then dividing by the total number of messages in these mailboxes. this generated a number around 13k average message size. i had to do this as part of assessing the feasability of migrating to maildir without reformatting the filesystem. Wow. Lot's of html and lots of attachments. It might also be useful to calculate the mode and standard deviation. Why? Really big attachments *might* be skewing the average. recent versions of kernel/ext2/ext3 have built-in dirent hashing, which cuts heavily on the many-files penalty. another benefit of maildir is that when you modify a single message, you only need to modify the I thought it was illegal to modify a message. marking a message as read is one example. moving a message from one mailbox to another is another example. although it's not modifying the message itself, it's moving its location, which with a crappy imap server can mean re-writing the contents of two mailboxes. *cough* wu- *cough* ;) -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. Lead the people with governmental measures and regulate them by law and punishments, and they will avoid wrongdoing, but will have no sense of honor and shame. Lead them by virtue and regulate them by the rules of propriety and they will have a sense of shame and, moreover, set themselves right. Confucius -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Consider a spam-bin folder that you don't split by month or whatever and don't check very often. -- Colin Watson [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Sat, 2005-02-26 at 10:23 +, Colin Watson wrote: On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Consider a spam-bin folder that you don't split by month or whatever and don't check very often. Maildir? -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. I don't pretend we have all the answers. But the questions are certainly worth thinking about. Arthur C. Clarke -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Sat, Feb 26, 2005 at 04:27:48AM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 10:23 +, Colin Watson wrote: On Fri, Feb 25, 2005 at 07:45:47PM -0600, Ron Johnson wrote: On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? Consider a spam-bin folder that you don't split by month or whatever and don't check very often. Maildir? Not everyone likes maildir; I gave up on it after experimenting with it and realising that making it harder for myself to use standard Unix text-processing tools on my mailboxes was just too annoying. This is true of spam-bin folders more than other folders, if anything. -- Colin Watson [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Fri, 25 Feb 2005, Steve Langasek wrote: The version in experimental has -D_FILE_OFFSET_BITS=64, and it works on files larger than 2GB, but I have only tested it on the i386 architecture. Please use the value of $(getconf LFS_CFLAGS) instead; it appears (based on past exim4 bug reports) that using -D_FILE_OFFSET_BITS=64 on natively 64-bit systems such as alpha causes surprising breakage of some glibc APIs. Ok, will do that. * Would the release managers approve this change for sarge? If this is the only change, yes. Hmm, while we are at it, I think it would be good to fix #295604 as well. This is the (trivial) patch: diff -ru procmail-3.22.orig/src/header.h procmail-3.22/src/header.h --- procmail-3.22.orig/src/header.h 1999-07-06 08:12:22.0 +0200 +++ procmail-3.22/src/header.h 2005-02-17 00:34:47.0 +0100 @@ -168,3 +168,5 @@ X(readreceiptto, Read-Receipt-To:) /* miscellaneous extension */ X(fakesender, Fake-Sender:) X(envelopeto, Envelope-To:)/* exim extension */ +X(useragent, User-Agent:) +X(nntppostingdate, NNTP-Posting-Date:) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: procmail and Large File Support
On Sat, Feb 26, 2005 at 12:59:49PM +0100, Santiago Vila wrote: On Fri, 25 Feb 2005, Steve Langasek wrote: The version in experimental has -D_FILE_OFFSET_BITS=64, and it works on files larger than 2GB, but I have only tested it on the i386 architecture. Please use the value of $(getconf LFS_CFLAGS) instead; it appears (based on past exim4 bug reports) that using -D_FILE_OFFSET_BITS=64 on natively 64-bit systems such as alpha causes surprising breakage of some glibc APIs. Ok, will do that. * Would the release managers approve this change for sarge? If this is the only change, yes. Hmm, while we are at it, I think it would be good to fix #295604 as well. This is the (trivial) patch: diff -ru procmail-3.22.orig/src/header.h procmail-3.22/src/header.h --- procmail-3.22.orig/src/header.h 1999-07-06 08:12:22.0 +0200 +++ procmail-3.22/src/header.h 2005-02-17 00:34:47.0 +0100 @@ -168,3 +168,5 @@ X(readreceiptto, Read-Receipt-To:) /* miscellaneous extension */ X(fakesender, Fake-Sender:) X(envelopeto, Envelope-To:)/* exim extension */ +X(useragent, User-Agent:) +X(nntppostingdate, NNTP-Posting-Date:) Looks straightforward, yes. -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: procmail and Large File Support
Hi Santiago, On Sat, Feb 26, 2005 at 12:53:40AM +0100, Santiago Vila wrote: I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: * Am I right to think that adding -D_FILE_OFFSET_BITS=64 to CFLAGS should be enough to fix this, as explained by this URL?: http://www.suse.de/~aj/linux_lfs.html The version in experimental has -D_FILE_OFFSET_BITS=64, and it works on files larger than 2GB, but I have only tested it on the i386 architecture. Please use the value of $(getconf LFS_CFLAGS) instead; it appears (based on past exim4 bug reports) that using -D_FILE_OFFSET_BITS=64 on natively 64-bit systems such as alpha causes surprising breakage of some glibc APIs. * Would the release managers approve this change for sarge? If this is the only change, yes. Cheers, -- Steve Langasek postmodern programmer signature.asc Description: Digital signature
Re: procmail and Large File Support
On Sat, 2005-02-26 at 00:53 +0100, Santiago Vila wrote: Hello. I have several reports saying procmail does not support mbox folders larger than 2GB. Questions: OT here, but WTF are people smoking, to have 2GB mbox files? -- - Ron Johnson, Jr. Jefferson, LA USA PGP Key ID 8834C06B I prefer encrypted mail. LUKE: Is Perl better than Python? YODA: No... no... no. Quicker, easier, more seductive. LUKE: But how will I know why Python is better than Perl? YODA: You will know. When your code you try to read six months from now. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
