I would like to see something like this also I run 32bit servers so I am
limited on how much memory I can use for heap. Besides just storing to disk
I would like to see some sort of cache like a block cache that will cache
parts the BlocksMap this would help reduce the hits to disk for lookups and
still give us the ability to lower the memory requirement for the namenode.
Billy
"Dennis Kubes" <[EMAIL PROTECTED]> wrote in
message news:[EMAIL PROTECTED]
From time to time a message pops up on the mailing list about OOM errors
for the namenode because of too many files. Most recently there was a 1.7
million file installation that was failing. I know the simple solution to
this is to have a larger java heap for the namenode. But the non-simple
way would be to convert the BlocksMap for the NameNode to be stored on
disk and then queried and updated for operations. This would eliminate
memory problems for large file installations but also might degrade
performance slightly. Questions:
1) Is there any current work to allow the namenode to store on disk versus
is memory? This could be a configurable option.
2) Besides possible slight degradation in performance, is there a reason
why the BlocksMap shouldn't or couldn't be stored on disk?
I am willing to put forth the work to make this happen. Just want to make
sure I am not going down the wrong path to begin with.
Dennis
