Doug:
If we use the heap as a cache and you have a large cluster then you will
have the memory on the NN to handle keeping all the namespace in memory.
We are looking for a way to support smaller clusters also that might over
run there heap size causing the cluster to crash.
So if the NN has the room to cache all the namespace then the larger
clusters will not see any disk hits once the namespace is fully loaded in to
memory.
Billy
"Doug Cutting" <[EMAIL PROTECTED]> wrote in
message news:[EMAIL PROTECTED]
Dennis Kubes wrote:
2) Besides possible slight degradation in performance, is there a reason
why the BlocksMap shouldn't or couldn't be stored on disk?
I think the assumption is that it would be considerably more than slight
degradation. I've seen the namenode benchmarked at over 50,000 opens per
second. If file data is on disk, and the namespace is considerably bigger
than RAM, then a seek would be required per access. At 10MS/seek, that
would give only 100 opens per second, or 500x slower. Flash storage today
peaks at around 5k seeks/second.
For smaller clusters the namenode might not need to be able to perform 50k
opens/second, but for larger clusters we do not want the namenode to
become a bottleneck.
Doug
