On Sat, Apr 14, 2001 at 05:11:16PM +0800, Thomas Duterme <[EMAIL PROTECTED]>
wrote:
| So I've got a script which does some MAJOR file processing.
[...]
| Problem, I need to move these according to header to a
| certain directory. Now I've read various numbers that UNIX
| in general is not so happy when you overpopulate a
| directory. I've read that you should try to manage your
| directories so that there are less than 100 files per
| directory. (I've also heard 1000....which is it?)
|
| In terms of performance sacrifice, can someone please
| clarify what happens when you stuff too many files in a
| directory (in terms of read/write performance)?
On most platforms directories are searched linearly when you open
a file. (The dir is just a list.) I believe the ext2 fs uses linear
directories.
So if you have n files to process and they're all in one directory you
look at, on average, n/2 file entries in the directory when you try to
open or stat it. So total work is on the order of n*n/2, so it goes up
as the square of your number of files.
With many files, this gets expensive real fast.
So if you have high access requirements you make a subdirectory tree.
Eg 16 top level dirs, each with 16 (or 64 or whatever) subdirs, each
with a bunch of files therein. Pick some scheme which distributes your
files evenly across the subdirs.
This reduces your search cost to 16/2 + 16/2 + (n/( 16*16 ))/2
on average, and overall to n* that, so
n*(16*n)/512
which still scales as the square, but at about 1/256 the total cost.
Of course this is all very rough (the cost to open all those subdirs
creeps in, for example), but you get the idea. A win. Squid caches use
such a mechanism.
If course this only mitigates the search cost. If that's significant, you
win. If not, you're adding complexity to little gain.
Note, of course, that this goes as n squared if your directory is very
full. If you can keep pace with the influx of bounces (instead of doing
big batches of them) then the size of the direct at any one time _ought_
to be fairly small and these nasty scaling issues don't apply nearly as
much. It becomes n*m where "m" is the average directory size rather than
your total number of files. Of course, n == m for batch work :-(
Cheers,
--
Cameron Simpson, DoD#743 [EMAIL PROTECTED] http://www.zip.com.au/~cs/
Altruism is a fine motive, but if you want results, greed works much better.
- Henry Spencer @ U of Toronto Zoology, [EMAIL PROTECTED]
_______________________________________________
Redhat-list mailing list
[EMAIL PROTECTED]
https://listman.redhat.com/mailman/listinfo/redhat-list