+1 but since hadoop deals well with such directories currently, fuse-dfs will basically lock up on such directories - this is because ls -color=blah causes a stat on every file in a directory. There is a JIRA open for this but it is a pretty rare case although it has happened to me at facebook.
-- pete >It's good for a portable application to keep the #of files/directory low by having two levels of directory for storing files -just use a hash operation to determine which dir to store a specific file in. On 11/3/08 4:00 AM, "Steve Loughran" <[EMAIL PROTECTED]> wrote: Pete Wyckoff wrote: > It has come a long way since 0.18 and facebook keeps our (0.17) dfs mounted > via fuse and uses that for some operations. > > There have recently been some problems with fuse-dfs when used in a > multithreaded environment, but those have been fixed in 0.18.2 and 0.19. (do > not use 0.18 or 0.18.1) > > The current (known) issues are: > 2. When directories have 10s of thousands of files, performance can be very > poor. I've known other filesystems to top out at 64k-1 files per directory, even if they don't slow down. It's good for a portable application to keep the #of files/directory low by having two levels of directory for storing files -just use a hash operation to determine which dir to store a specific file in.
