Cagdas Gerede wrote:
In the system I am working, we have 6 million blocks total and the namenode heap size is about 600 MB and it takes about 5 minutes for namenode to leave the safemode.
How big is are your files? Are they several blocks on average? Hadoop is not designed for small files, but rather for larger files. An Archive system is currently being designed to help with this.
https://issues.apache.org/jira/browse/HADOOP-3307
I try to estimate what would be the heap size if we have 100 - 150 million blocks, and what would be the amount of time for namenode to leave the safemode.
At ~100M per block, 100M blocks would store 10PB. At ~1TB/node, this means a ~10,000 node system, larger than Hadoop currently supports well (for this and other reasons).
If your files are generally large, you can increase your block size to 250MB to decrease the number of blocks in the system.
Doug
