> We can also try to mount the particular dir on ramfs
This could be an interesting project to replace the whole name-node
that is its FSDirectory part by ramfs.
If there are people interested in that kind of experiments we can discuss it.
>>> 2) Besides possible slight degradation in performance, is there a
>>> reason why the BlocksMap shouldn't or couldn't be stored on disk?
The degradation is expected to be substantial. But another reason is that it
will introduce a whole new layer in the name-node implementation responsible
of keeping the memory image in sync with the disk block map with caching etc.
As Raghu mentioned there was an idea of separating blockMap into a standalone
server, but currently there is no ongoing progress in this direction.
--Konstantin
Sagar Naik wrote:
We can also try to mount the particular dir on ramfs and reduce the
performance degradation
-Sagar
Billy Pearson wrote:
I would like to see something like this also I run 32bit servers so I
am limited on how much memory I can use for heap. Besides just storing
to disk I would like to see some sort of cache like a block cache that
will cache parts the BlocksMap this would help reduce the hits to disk
for lookups and still give us the ability to lower the memory
requirement for the namenode.
Billy
"Dennis Kubes" <[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
From time to time a message pops up on the mailing list about OOM
errors for the namenode because of too many files. Most recently
there was a 1.7 million file installation that was failing. I know
the simple solution to this is to have a larger java heap for the
namenode. But the non-simple way would be to convert the BlocksMap
for the NameNode to be stored on disk and then queried and updated
for operations. This would eliminate memory problems for large file
installations but also might degrade performance slightly. Questions:
1) Is there any current work to allow the namenode to store on disk
versus is memory? This could be a configurable option.
2) Besides possible slight degradation in performance, is there a
reason why the BlocksMap shouldn't or couldn't be stored on disk?
I am willing to put forth the work to make this happen. Just want to
make sure I am not going down the wrong path to begin with.
Dennis
