[ https://issues.apache.org/jira/browse/HDFS-8286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083750#comment-15083750 ]
Haohui Mai commented on HDFS-8286: ---------------------------------- I have pushed the prototype that corresponds to my [Hadoop summit talk|http://www.slideshare.net/HaohuiMai/partial-nshadoopsummit2015] to the [feature-HDFS-8286|https://github.com/apache/hadoop/tree/feature-HDFS-8286] branch. > Scaling out the namespace using KV store > ---------------------------------------- > > Key: HDFS-8286 > URL: https://issues.apache.org/jira/browse/HDFS-8286 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Haohui Mai > Assignee: Haohui Mai > Attachments: hdfs-kv-design.pdf > > > Currently the NN keeps the namespace in the memory. To improve the > scalability of the namespace, users can scale up by using more RAM or scale > out using Federation (i.e., statically partitioning the namespace). > We would like to remove the limitation of scaling the global namespace. Our > vision is that that HDFS should adopt a scalable underlying architecture that > allows the global namespace scales linearly. > We propose to implement the HDFS namespace on top of a key-value (KV) store. > Adopting the KV store interfaces allows HDFS to leverage the capability of > modern KV store and to become much easier to scale. Going forward, the > architecture allows distributing the namespace across multiple machines, or > storing only the working set in the memory (HDFS-5389), both of which allows > HDFS to manage billions of files using the commodity hardware available today. -- This message was sent by Atlassian JIRA (v6.3.4#6332)