Re: Qmail - huge performance increase
On Wed, Jun 27, 2001 at 04:03:11PM +0200, Tomasz Papszun wrote: > "/bin/ls | wc" has taken 1 (one) second. "ls | wc" lasted 3 minutes and 26 > seconds. Yes, near 3 and a half minutes! > > This is because "ls" with additional information (e.g. file type, which is > needed to colour a listing) needs more time to gather this information. > I don't know what difference would be for reiserfs or xfs filesystems. yep. ls -l also takes a long time in large directories because it has to stat each file to get the info about it. i never bothered timing it, but in a directory with thousands of files it is significantly faster to run this: ls -1S | head -30 | xargs ls -lS ^ -- numeral 1, not letter l. than this: ls -lS | head -30 btw, in case you're wondering, i used to use commands like that to find the largest 30 files in /var/spool/mail craig -- craig sanders <[EMAIL PROTECTED]> Fabricati Diem, PVNC. -- motto of the Ankh-Morpork City Watch
Re: Qmail - huge performance increase
On Wed, Jun 27, 2001 at 04:03:11PM +0200, Tomasz Papszun wrote: > "/bin/ls | wc" has taken 1 (one) second. "ls | wc" lasted 3 minutes and 26 > seconds. Yes, near 3 and a half minutes! > > This is because "ls" with additional information (e.g. file type, which is > needed to colour a listing) needs more time to gather this information. > I don't know what difference would be for reiserfs or xfs filesystems. yep. ls -l also takes a long time in large directories because it has to stat each file to get the info about it. i never bothered timing it, but in a directory with thousands of files it is significantly faster to run this: ls -1S | head -30 | xargs ls -lS ^ -- numeral 1, not letter l. than this: ls -lS | head -30 btw, in case you're wondering, i used to use commands like that to find the largest 30 files in /var/spool/mail craig -- craig sanders <[EMAIL PROTECTED]> Fabricati Diem, PVNC. -- motto of the Ankh-Morpork City Watch -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Qmail - huge performance increase
And on an Ultra-60 running Solaris 7 w/UFS: bash-2.04$ time /bin/ls | wc 63975 63975 1971245 real0m2.213s user0m1.160s sys 0m0.890s bash-2.04$ time ls | wc 63975 63975 1971253 real2m19.965s user0m1.490s sys 0m16.340s bash-2.04$ Sped it up "just a little bit" :-) On Wednesday 27 June 2001 07:03, Tomasz Papszun wrote: > On Thu, 21 Jun 2001 at 13:25:17 +1000, Craig Sanders wrote: > > On Thu, Jun 21, 2001 at 01:45:23AM +0800, Jason Lim wrote: > > > SO... by increasing conf-split to 97 (from the default of 20 > > > something afaik), each directory ends up only having a hundred or so > > > files. Doing "ls" now is far speedier. > > > [...] > > > > this is actually a well-known limitation of ext2fs and similar > > file-systems - as soon as you get more than a thousand or so files in a > > directory, performance takes a nosedive. > > BTW, a tip: if you've got "ls" aliased (for instance as > 'ls --color=auto -F') then you can shorten this long execution of "ls". > Just issue "/bin/ls" instead of "ls". The difference is very big. It can > be as 1:200 (yeah!). I've just done a comparison in a directory > with > 33000 files. > > "/bin/ls | wc" has taken 1 (one) second. "ls | wc" lasted 3 minutes and 26 > seconds. Yes, near 3 and a half minutes! > > This is because "ls" with additional information (e.g. file type, which is > needed to colour a listing) needs more time to gather this information. > I don't know what difference would be for reiserfs or xfs filesystems. > > Hope it helps a little :-) . -- "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: Qmail - huge performance increase
On Thu, 21 Jun 2001 at 13:25:17 +1000, Craig Sanders wrote: > On Thu, Jun 21, 2001 at 01:45:23AM +0800, Jason Lim wrote: > > SO... by increasing conf-split to 97 (from the default of 20 > > something afaik), each directory ends up only having a hundred or so > > files. Doing "ls" now is far speedier. > > [...] > > this is actually a well-known limitation of ext2fs and similar > file-systems - as soon as you get more than a thousand or so files in a > directory, performance takes a nosedive. > BTW, a tip: if you've got "ls" aliased (for instance as 'ls --color=auto -F') then you can shorten this long execution of "ls". Just issue "/bin/ls" instead of "ls". The difference is very big. It can be as 1:200 (yeah!). I've just done a comparison in a directory with > 33000 files. "/bin/ls | wc" has taken 1 (one) second. "ls | wc" lasted 3 minutes and 26 seconds. Yes, near 3 and a half minutes! This is because "ls" with additional information (e.g. file type, which is needed to colour a listing) needs more time to gather this information. I don't know what difference would be for reiserfs or xfs filesystems. Hope it helps a little :-) . -- Tomasz Papszun SysAdm @ TP S.A. Lodz, Poland | And it's only [EMAIL PROTECTED] http://www.lodz.tpsa.pl/ | ones and zeros.
Re: Qmail - huge performance increase
And on an Ultra-60 running Solaris 7 w/UFS: bash-2.04$ time /bin/ls | wc 63975 63975 1971245 real0m2.213s user0m1.160s sys 0m0.890s bash-2.04$ time ls | wc 63975 63975 1971253 real2m19.965s user0m1.490s sys 0m16.340s bash-2.04$ Sped it up "just a little bit" :-) On Wednesday 27 June 2001 07:03, Tomasz Papszun wrote: > On Thu, 21 Jun 2001 at 13:25:17 +1000, Craig Sanders wrote: > > On Thu, Jun 21, 2001 at 01:45:23AM +0800, Jason Lim wrote: > > > SO... by increasing conf-split to 97 (from the default of 20 > > > something afaik), each directory ends up only having a hundred or so > > > files. Doing "ls" now is far speedier. > > > [...] > > > > this is actually a well-known limitation of ext2fs and similar > > file-systems - as soon as you get more than a thousand or so files in a > > directory, performance takes a nosedive. > > BTW, a tip: if you've got "ls" aliased (for instance as > 'ls --color=auto -F') then you can shorten this long execution of "ls". > Just issue "/bin/ls" instead of "ls". The difference is very big. It can > be as 1:200 (yeah!). I've just done a comparison in a directory > with > 33000 files. > > "/bin/ls | wc" has taken 1 (one) second. "ls | wc" lasted 3 minutes and 26 > seconds. Yes, near 3 and a half minutes! > > This is because "ls" with additional information (e.g. file type, which is > needed to colour a listing) needs more time to gather this information. > I don't know what difference would be for reiserfs or xfs filesystems. > > Hope it helps a little :-) . -- "To me vi is Zen. To use vi is to practice zen. Every command is a koan. Profound to the user, unintelligible to the uninitiated. You discover truth everytime you use it." [EMAIL PROTECTED] [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Qmail - huge performance increase
On Thu, 21 Jun 2001 at 13:25:17 +1000, Craig Sanders wrote: > On Thu, Jun 21, 2001 at 01:45:23AM +0800, Jason Lim wrote: > > SO... by increasing conf-split to 97 (from the default of 20 > > something afaik), each directory ends up only having a hundred or so > > files. Doing "ls" now is far speedier. > > [...] > > this is actually a well-known limitation of ext2fs and similar > file-systems - as soon as you get more than a thousand or so files in a > directory, performance takes a nosedive. > BTW, a tip: if you've got "ls" aliased (for instance as 'ls --color=auto -F') then you can shorten this long execution of "ls". Just issue "/bin/ls" instead of "ls". The difference is very big. It can be as 1:200 (yeah!). I've just done a comparison in a directory with > 33000 files. "/bin/ls | wc" has taken 1 (one) second. "ls | wc" lasted 3 minutes and 26 seconds. Yes, near 3 and a half minutes! This is because "ls" with additional information (e.g. file type, which is needed to colour a listing) needs more time to gather this information. I don't know what difference would be for reiserfs or xfs filesystems. Hope it helps a little :-) . -- Tomasz Papszun SysAdm @ TP S.A. Lodz, Poland | And it's only [EMAIL PROTECTED] http://www.lodz.tpsa.pl/ | ones and zeros. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Qmail - huge performance increase
On Thu, Jun 21, 2001 at 01:45:23AM +0800, Jason Lim wrote: > SO... by increasing conf-split to 97 (from the default of 20 > something afaik), each directory ends up only having a hundred or so > files. Doing "ls" now is far speedier. > > I couldn't find any documentation anywhere stating this, so I'll share > it with you all :-) this is actually a well-known limitation of ext2fs and similar file-systems - as soon as you get more than a thousand or so files in a directory, performance takes a nosedive. you can see this effect on news servers such as INN and also mail servers which use Maildir when the users keep thousands of messages in the one "folder", and in mail servers with many thousands of messages in the queue...in fact, in any application which has lots of files in one directory. i'd recommend using reiserfs or perhaps xfs for these applications. this is one of the major advantages that a modern filesystem like reiserfs has over ext2 - you can have as many files as you want in a directory and it doesn't suffer from this slowdown. reiserfs is included in kernel 2.4.x now, and SGI's linux port of their xfs is available as a patch. i've been using reiserfs on production servers for ages with no problems (even using it on mission critical mail servers), and have been trialling xfs since the 1.0 release...with no problems so far, so i'm starting to use it on non-mission-critical servers (e.g. my anonymous ftp server / debian mirror). if it lives up to expectations, i'll start replacing reiserfs with xfs over the next 6-12 months. BTW, even sendmail lets you set the hash queue depth (to use postfix terminology, which i am more comfortable with :) these days. sendmail used to have just one directory for all queue fileswhich made for lousy performance. craig -- craig sanders <[EMAIL PROTECTED]> Fabricati Diem, PVNC. -- motto of the Ankh-Morpork City Watch
Re: Qmail - huge performance increase
On Thu, Jun 21, 2001 at 01:45:23AM +0800, Jason Lim wrote: > SO... by increasing conf-split to 97 (from the default of 20 > something afaik), each directory ends up only having a hundred or so > files. Doing "ls" now is far speedier. > > I couldn't find any documentation anywhere stating this, so I'll share > it with you all :-) this is actually a well-known limitation of ext2fs and similar file-systems - as soon as you get more than a thousand or so files in a directory, performance takes a nosedive. you can see this effect on news servers such as INN and also mail servers which use Maildir when the users keep thousands of messages in the one "folder", and in mail servers with many thousands of messages in the queue...in fact, in any application which has lots of files in one directory. i'd recommend using reiserfs or perhaps xfs for these applications. this is one of the major advantages that a modern filesystem like reiserfs has over ext2 - you can have as many files as you want in a directory and it doesn't suffer from this slowdown. reiserfs is included in kernel 2.4.x now, and SGI's linux port of their xfs is available as a patch. i've been using reiserfs on production servers for ages with no problems (even using it on mission critical mail servers), and have been trialling xfs since the 1.0 release...with no problems so far, so i'm starting to use it on non-mission-critical servers (e.g. my anonymous ftp server / debian mirror). if it lives up to expectations, i'll start replacing reiserfs with xfs over the next 6-12 months. BTW, even sendmail lets you set the hash queue depth (to use postfix terminology, which i am more comfortable with :) these days. sendmail used to have just one directory for all queue fileswhich made for lousy performance. craig -- craig sanders <[EMAIL PROTECTED]> Fabricati Diem, PVNC. -- motto of the Ankh-Morpork City Watch -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
