Re: ext2/3 files per directory limits
Ron Peterson <[EMAIL PROTECTED]> wrote: > > I would like to better understand ext2/3's performance characteristics. > > I'm specifically interested in how ext2/3 will handle a /var/spool/mail > directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as > 75,000 messages daily. Virtually all access is via imap, w/ approx > ~1000 imapd processes running during peak load. Local delivery is via > procmail, which by default uses both kernel-supported locking calls and > .lock files. > > I understand that various tuning parameters will have an impact, > e.g. putting the journal on a separate device, setting the noatime mount > option, etc. I also understand that there are other mailbox formats and > other strategies for locating mail spools (e.g. in user's home > directories). > > I'm interested in people's thoughts on these issues, but I'm mostly > interested in whether or not the scenario I described falls within > ext2/3's designed capabilities. > noatime will help. increasing the journal size _may_ help. With 6k files per directory you'll benefit from indexed directories (htree). Use `tune2fs -O dir_index'. dir_index isn't available for ext2. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: ext2/3 files per directory limits
On Wed, 2005-02-23 at 22:11 -0500, Ron Peterson wrote: > I would like to better understand ext2/3's performance characteristics. > > I'm specifically interested in how ext2/3 will handle a /var/spool/mail > directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as > 75,000 messages daily. Virtually all access is via imap, w/ approx > ~1000 imapd processes running during peak load. Local delivery is via > procmail, which by default uses both kernel-supported locking calls and > .lock files. > > I understand that various tuning parameters will have an impact, > e.g. putting the journal on a separate device, setting the noatime mount > option, etc. I also understand that there are other mailbox formats and > other strategies for locating mail spools (e.g. in user's home > directories). > > I'm interested in people's thoughts on these issues, but I'm mostly > interested in whether or not the scenario I described falls within > ext2/3's designed capabilities. Yes, ext2 and ext3 can handle that load easily. You should not have to do any special tuning. The real question is why in the world you would want to use mbox format for this. It simply does not scale. Use maildir. Lee - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: ext2/3 files per directory limits
On Wed, 23 Feb 2005, Ron Peterson wrote: I would like to better understand ext2/3's performance characteristics. I'm specifically interested in how ext2/3 will handle a /var/spool/mail directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as 75,000 messages daily. Virtually all access is via imap, w/ approx ~1000 imapd processes running during peak load. Local delivery is via procmail, which by default uses both kernel-supported locking calls and .lock files. At some point it makes sense to subdivide you mail load because serialization of i/o on that one filesystem becomes a bigger issue than the performance of your filesystem... We deliver into mbox formatted mailboxes inside users homedirs, some folks do a similar thing with maildir. In the end you can on make one filesystem so fast. beyond that you need more filesystems to acheive any kind of reasonable scaling... I understand that various tuning parameters will have an impact, e.g. putting the journal on a separate device, setting the noatime mount option, etc. I also understand that there are other mailbox formats and other strategies for locating mail spools (e.g. in user's home directories). I'm interested in people's thoughts on these issues, but I'm mostly interested in whether or not the scenario I described falls within ext2/3's designed capabilities. Best. -- -- Joel Jaeggli Unix Consulting [EMAIL PROTECTED] GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
ext2/3 files per directory limits
I would like to better understand ext2/3's performance characteristics. I'm specifically interested in how ext2/3 will handle a /var/spool/mail directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as 75,000 messages daily. Virtually all access is via imap, w/ approx ~1000 imapd processes running during peak load. Local delivery is via procmail, which by default uses both kernel-supported locking calls and .lock files. I understand that various tuning parameters will have an impact, e.g. putting the journal on a separate device, setting the noatime mount option, etc. I also understand that there are other mailbox formats and other strategies for locating mail spools (e.g. in user's home directories). I'm interested in people's thoughts on these issues, but I'm mostly interested in whether or not the scenario I described falls within ext2/3's designed capabilities. Best. -- Ron Peterson Network & Systems Manager Mount Holyoke College http://www.mtholyoke.edu/~rpeterso - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
ext2/3 files per directory limits
I would like to better understand ext2/3's performance characteristics. I'm specifically interested in how ext2/3 will handle a /var/spool/mail directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as 75,000 messages daily. Virtually all access is via imap, w/ approx ~1000 imapd processes running during peak load. Local delivery is via procmail, which by default uses both kernel-supported locking calls and .lock files. I understand that various tuning parameters will have an impact, e.g. putting the journal on a separate device, setting the noatime mount option, etc. I also understand that there are other mailbox formats and other strategies for locating mail spools (e.g. in user's home directories). I'm interested in people's thoughts on these issues, but I'm mostly interested in whether or not the scenario I described falls within ext2/3's designed capabilities. Best. -- Ron Peterson Network Systems Manager Mount Holyoke College http://www.mtholyoke.edu/~rpeterso - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: ext2/3 files per directory limits
On Wed, 23 Feb 2005, Ron Peterson wrote: I would like to better understand ext2/3's performance characteristics. I'm specifically interested in how ext2/3 will handle a /var/spool/mail directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as 75,000 messages daily. Virtually all access is via imap, w/ approx ~1000 imapd processes running during peak load. Local delivery is via procmail, which by default uses both kernel-supported locking calls and .lock files. At some point it makes sense to subdivide you mail load because serialization of i/o on that one filesystem becomes a bigger issue than the performance of your filesystem... We deliver into mbox formatted mailboxes inside users homedirs, some folks do a similar thing with maildir. In the end you can on make one filesystem so fast. beyond that you need more filesystems to acheive any kind of reasonable scaling... I understand that various tuning parameters will have an impact, e.g. putting the journal on a separate device, setting the noatime mount option, etc. I also understand that there are other mailbox formats and other strategies for locating mail spools (e.g. in user's home directories). I'm interested in people's thoughts on these issues, but I'm mostly interested in whether or not the scenario I described falls within ext2/3's designed capabilities. Best. -- -- Joel Jaeggli Unix Consulting [EMAIL PROTECTED] GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: ext2/3 files per directory limits
On Wed, 2005-02-23 at 22:11 -0500, Ron Peterson wrote: I would like to better understand ext2/3's performance characteristics. I'm specifically interested in how ext2/3 will handle a /var/spool/mail directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as 75,000 messages daily. Virtually all access is via imap, w/ approx ~1000 imapd processes running during peak load. Local delivery is via procmail, which by default uses both kernel-supported locking calls and .lock files. I understand that various tuning parameters will have an impact, e.g. putting the journal on a separate device, setting the noatime mount option, etc. I also understand that there are other mailbox formats and other strategies for locating mail spools (e.g. in user's home directories). I'm interested in people's thoughts on these issues, but I'm mostly interested in whether or not the scenario I described falls within ext2/3's designed capabilities. Yes, ext2 and ext3 can handle that load easily. You should not have to do any special tuning. The real question is why in the world you would want to use mbox format for this. It simply does not scale. Use maildir. Lee - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: ext2/3 files per directory limits
Ron Peterson [EMAIL PROTECTED] wrote: I would like to better understand ext2/3's performance characteristics. I'm specifically interested in how ext2/3 will handle a /var/spool/mail directory w/ ~6000 mbox format inboxes, handling approx 1GB delivered as 75,000 messages daily. Virtually all access is via imap, w/ approx ~1000 imapd processes running during peak load. Local delivery is via procmail, which by default uses both kernel-supported locking calls and .lock files. I understand that various tuning parameters will have an impact, e.g. putting the journal on a separate device, setting the noatime mount option, etc. I also understand that there are other mailbox formats and other strategies for locating mail spools (e.g. in user's home directories). I'm interested in people's thoughts on these issues, but I'm mostly interested in whether or not the scenario I described falls within ext2/3's designed capabilities. noatime will help. increasing the journal size _may_ help. With 6k files per directory you'll benefit from indexed directories (htree). Use `tune2fs -O dir_index'. dir_index isn't available for ext2. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
