Re: Tunning for large number of files in INBOX
On Mon, Jul 04, 2005 at 03:54:16PM +0200, Marco Colombo wrote: > > Is there a "real IMAP client" which is free software? > > I have seen this "downloading *all* headers" behaviour with every free > > imap client I have tried. > > > > Not that I'm suggesting it, but pine doesn't show the "all headers" > behavior, and it's free. :-) > > Too bad they forgot that an imap folder can hold both messages and > subfolders, and they had to add a late hack to allow the correct > browsing of a Cyrus server. No client is perfect. > > Being an old-time user of Pine, it's always a pain to use Thunderbird or > Evolution, clients so feature-full but w/o decent imap behavior: > sometimes I have to switch back to Pine to be able to handle 50k+ new > messages per folder in a decent time (Pine takes negligible time to open > them). Interesting to know, thanks. I'll consult with my local Pine users :) --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Tunning for large number of files in INBOX
On Thu, 2005-06-30 at 13:22 -0300, Andreas Hasenack wrote: > On Wed, Jun 29, 2005 at 02:56:58PM -0600, Michael Loftis wrote: > > clients retrofitted to squak IMAP. Get a real IMAP client like Mulberry > > that takes advantage of server side sorting, threading, and searching to > > allow for (nearly) limitless mailboxes but not download each and every > > header. > > Is there a "real IMAP client" which is free software? > I have seen this "downloading *all* headers" behaviour with every free > imap client I have tried. > Not that I'm suggesting it, but pine doesn't show the "all headers" behavior, and it's free. :-) Too bad they forgot that an imap folder can hold both messages and subfolders, and they had to add a late hack to allow the correct browsing of a Cyrus server. No client is perfect. Being an old-time user of Pine, it's always a pain to use Thunderbird or Evolution, clients so feature-full but w/o decent imap behavior: sometimes I have to switch back to Pine to be able to handle 50k+ new messages per folder in a decent time (Pine takes negligible time to open them). .TM. -- / / / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _/ _/ _/ [EMAIL PROTECTED] --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Tunning for large number of files in INBOX
-- Andreas Hasenack <[EMAIL PROTECTED]> is rumored to have mumbled on 30. Juni 2005 13:22:12 -0300 regarding Re: Tunning for large number of files in INBOX: On Wed, Jun 29, 2005 at 02:56:58PM -0600, Michael Loftis wrote: clients retrofitted to squak IMAP. Get a real IMAP client like Mulberry that takes advantage of server side sorting, threading, and searching to allow for (nearly) limitless mailboxes but not download each and every header. Is there a "real IMAP client" which is free software? I have seen this "downloading *all* headers" behaviour with every free imap client I have tried. So maybe it's time to try one that's not free? I'm all for OSS, but Mulberry is worth every cent. Cheers, Sebastian -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 pgpjvtEv8GKBF.pgp Description: PGP signature
Re: Tunning for large number of files in INBOX
On Wed, Jun 29, 2005 at 02:56:58PM -0600, Michael Loftis wrote: > clients retrofitted to squak IMAP. Get a real IMAP client like Mulberry > that takes advantage of server side sorting, threading, and searching to > allow for (nearly) limitless mailboxes but not download each and every > header. Is there a "real IMAP client" which is free software? I have seen this "downloading *all* headers" behaviour with every free imap client I have tried. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Tunning for large number of files in INBOX
On Wed, 2005-06-29 at 14:56 -0600, Michael Loftis wrote: > > --On June 29, 2005 4:30:06 PM -0400 Joel Nimety <[EMAIL PROTECTED]> > wrote: > > > Hello, > > > > I'm trying to set up cyrus-imap as a backend for an email archiving > > solution. I'm creating one account on the imap server for each customer > > domain(s) we'll be archiving mail for. I'm concerned that the number > > of emails that will end up in each INBOX will reach some limit (ext3 fs > > limit, practical limit, etc.) > > With EXT3 there are definite limits, not hard ones, but practical ones for > time to traverse/read the inode and list. Use ReiserFS. For Mail clients > most can't handle big folders because many of them are just POP3/NNTP > clients retrofitted to squak IMAP. Get a real IMAP client like Mulberry > that takes advantage of server side sorting, threading, and searching to > allow for (nearly) limitless mailboxes but not download each and every > header. > > With ReiserFS and UFS+Hashdirs (Linux and FreeBSD respectively) I > personally have many mailboxes that are well over 20k or 30k messages, and > have a few in the 200k range, haven't run into any performance problems. > With EXT3 I had serious problems in the 5k range, or less. When was that? AFAIK, EXT3 on Linux has indexed directory access since version 2.5.42 (11 Oct 2002). Any 2.6 kernel and any 2.4.20+ kernel includes it. You should see no problem in handling directories with > 100k files (biggest IMAP folder I've seen so far was about 50k messages). .TM. -- / / / / / / Marco Colombo ___/ ___ / / Technical Manager / / / ESI s.r.l. _/ _/ _/ [EMAIL PROTECTED] --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Tunning for large number of files in INBOX
--On June 29, 2005 2:52:30 PM -0700 Andrew Morgan <[EMAIL PROTECTED]> wrote: On Wed, 29 Jun 2005, Michael Loftis wrote: In the interest of completeness, under 2.6 linux kernels you can format an ext3 partition using the dir_index option. This enables a hash tree index for directories that supposedly improves lookups with very large directories. Here is the command I use to build my mail spool filesystem: I thought maybe there was but I wasn't sure. FreeBSD/BSD UFS is in that exact same manner, though it was first ;) same idea though, it's an extension that can be enabled/disabled via tunefs. It definitely helps. Thanks for chiming in on that. Not being sure, and not having time to investigate I felt it better to post what I knew rather than on conjecture. I'd imagine that the dir_index stuff helps EXT3 in the same way it helps UFS making it pretty close to the performance of ReiserFS in situations where without it, both Filesystems fall flat on their faces. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Tunning for large number of files in INBOX
On Wed, 2005-06-29 at 14:52 -0700, Andrew Morgan wrote: > In the interest of completeness, under 2.6 linux kernels you can format an > ext3 partition using the dir_index option. This enables a hash tree index > for directories that supposedly improves lookups with very large > directories. Here is the command I use to build my mail spool filesystem: > >mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1 > > I have not used other filesystems such as Reiser or XFS, so I cannot offer Later 2.4 kernels have hashed directory support also; in fact, I've just enabled it on my main Cyrus server, which is running CentOS 3.5 and kernel 2.4.21-27.0.2.ELsmp. (My e2fsprogs man pages had not been updated to reflect the new options also.) You don't have to recreate your filesystem or unmount them for that matter--you can enable it with: tune2fs -O dir_index /dev/foo You can also optimize your directories (which is probably a good idea if you're enabling it on an existing filesystem) with e2fsck: e2fsck -f -D /dev/foo (you'll need the -f if the filesystem is clean) Wil -- Wil Cooley [EMAIL PROTECTED] Naked Ape Consultinghttp://nakedape.cc * * * * Linux, UNIX, Networking and Security Solutions * * * * signature.asc Description: This is a digitally signed message part
Re: Tunning for large number of files in INBOX
On Wed, 29 Jun 2005, Michael Loftis wrote: --On June 29, 2005 4:30:06 PM -0400 Joel Nimety <[EMAIL PROTECTED]> wrote: Hello, I'm trying to set up cyrus-imap as a backend for an email archiving solution. I'm creating one account on the imap server for each customer domain(s) we'll be archiving mail for. I'm concerned that the number of emails that will end up in each INBOX will reach some limit (ext3 fs limit, practical limit, etc.) With EXT3 there are definite limits, not hard ones, but practical ones for time to traverse/read the inode and list. Use ReiserFS. For Mail clients most can't handle big folders because many of them are just POP3/NNTP clients retrofitted to squak IMAP. Get a real IMAP client like Mulberry that takes advantage of server side sorting, threading, and searching to allow for (nearly) limitless mailboxes but not download each and every header. With ReiserFS and UFS+Hashdirs (Linux and FreeBSD respectively) I personally have many mailboxes that are well over 20k or 30k messages, and have a few in the 200k range, haven't run into any performance problems. With EXT3 I had serious problems in the 5k range, or less. In the interest of completeness, under 2.6 linux kernels you can format an ext3 partition using the dir_index option. This enables a hash tree index for directories that supposedly improves lookups with very large directories. Here is the command I use to build my mail spool filesystem: mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1 I have not used other filesystems such as Reiser or XFS, so I cannot offer any performance comparisons. Andy --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Tunning for large number of files in INBOX
--On June 29, 2005 4:30:06 PM -0400 Joel Nimety <[EMAIL PROTECTED]> wrote: Hello, I'm trying to set up cyrus-imap as a backend for an email archiving solution. I'm creating one account on the imap server for each customer domain(s) we'll be archiving mail for. I'm concerned that the number of emails that will end up in each INBOX will reach some limit (ext3 fs limit, practical limit, etc.) With EXT3 there are definite limits, not hard ones, but practical ones for time to traverse/read the inode and list. Use ReiserFS. For Mail clients most can't handle big folders because many of them are just POP3/NNTP clients retrofitted to squak IMAP. Get a real IMAP client like Mulberry that takes advantage of server side sorting, threading, and searching to allow for (nearly) limitless mailboxes but not download each and every header. With ReiserFS and UFS+Hashdirs (Linux and FreeBSD respectively) I personally have many mailboxes that are well over 20k or 30k messages, and have a few in the 200k range, haven't run into any performance problems. With EXT3 I had serious problems in the 5k range, or less. Is there a way I can have cyrus hash the files within the INBOX directories into sub directories? If this isn't possible does anyone have a sieve script that can sort mail into folders by date? Any help is much appreciated. No, and no, but the latter should be something simple enough to create.A coworker did something llike this and found he had to create entries for every month, and then either manually swap them yearly or rewrite the rules yearly because Sieve has no variables or anything like that. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Tunning for large number of files in INBOX
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, I'm trying to set up cyrus-imap as a backend for an email archiving solution. I'm creating one account on the imap server for each customer domain(s) we'll be archiving mail for. I'm concerned that the number of emails that will end up in each INBOX will reach some limit (ext3 fs limit, practical limit, etc.) Is there a way I can have cyrus hash the files within the INBOX directories into sub directories? If this isn't possible does anyone have a sieve script that can sort mail into folders by date? Any help is much appreciated. Thanks. Joel Nimety Perimeter Internetworking Corp. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFCwwTO/uHyjmXLFuQRAl6JAKCOVWPN5iIBet1OVIlJLmTi7UgnzwCfdu8v wa1rZXcjC8yGSNQY6VDUVYM= =5apL -END PGP SIGNATURE- --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
