One more thing, Krishna, when using JounalNodes as opposed to the
native file system for the metadata storage you do get replication.
Chris
On 10/2/2013 12:52 AM, Krishna Kumaar Natarajan wrote:
Hi All,
While trying to understand federated HDFS in detail I had few doubts
and listing them down for your help.
1. In case of *_HDFS(without HDFS federation)_*, the metadata or the
data about the blocks belonging to the files in HDFS is maintained
in the main memory of the name node or it is stored on permanent
storage of the namenode and is brought in the main memory on
demand basis ?[Krishna] Based on my understanding, I assume the
entire metadata is in main memory which is an issue by itself.
Please correct me if my understanding is wrong.
2. In case of*_federated HDFS_*, the metadata or the data about the
blocks belonging to files in a particular namespace is maintained
in the main memory of the namenode or it is stored on the
permanent storage of the namenode and is brought in the main
memory on demand basis ?
3. Are the metadata information stored in separate cluster
nodes(block management layer separation) as discussed in Appendix
B of this document
?https://issues.apache.org/jira/secure/attachment/12453067/high-level-design.pdf
4. I would like to know if the following proposals are already
implemented in federated HDFS.
(http://www.slideshare.net/hortonworks/hdfs-futures-namenode-federation-for-improved-efficiency-and-scalability
slide-17)
* Separation of namespace and block management layers (same as qn.3)
* Partial namespace in memory for further scalability
* Move partial namespace from one namenode to another
Thanks,
Krishna
