On Jun 5, 2009, at 11:51 AM, Wasim Bari wrote:
Hi,
Does someone has some data regarding maximum possible number of
files over HDFS ?
Hey Wasim,
I don't think that there is a maximum limit. Remember:
1) Less is better. HDFS is optimized for big files.
2) The amount of memory the HDFS namenode needs is a function of the
number of files. If you have a huge number of files, you get a huge
memory requirement.
1-2 million files is fairly safe if you have a normal-looking namenode
server (8-16GB RAM). I know some of our UCSD colleagues just ran a
test where they were able to put more than .5M files in a single
directory and still have a useable file system.
Brian
my second question is, I created small files with small block size
up to one lac and read the files from HDFS, reading performance
remains almost unaffected with increasing number of files.
The possible reasons I could think are:
1 . One lac isn't a big number to disturb HDFS performance (I used
1 namenode and 4 data nodes)
2. As reading is done directly from datanode with first time
interaction with namenode, so reading from different nodes doesn't
affect the performance.
If someone could add or negate some information it will be highly
appreciated.
Cheers,
Wasim