From time to time a message pops up on the mailing list about OOM errors for the namenode because of too many files. Most recently there was a 1.7 million file installation that was failing. I know the simple solution to this is to have a larger java heap for the namenode. But the non-simple way would be to convert the BlocksMap for the NameNode to be stored on disk and then queried and updated for operations. This would eliminate memory problems for large file installations but also might degrade performance slightly. Questions:

1) Is there any current work to allow the namenode to store on disk versus is memory? This could be a configurable option.

2) Besides possible slight degradation in performance, is there a reason why the BlocksMap shouldn't or couldn't be stored on disk?

I am willing to put forth the work to make this happen. Just want to make sure I am not going down the wrong path to begin with.

Dennis

Reply via email to