Haohui Mai created HDFS-8286:
--------------------------------
Summary: Scaling out the namespace using KV store
Key: HDFS-8286
URL: https://issues.apache.org/jira/browse/HDFS-8286
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Currently the NN keeps the namespace in the memory. To improve the scalability
of the namespace, users can scale up by using more RAM or scale out using
Federation (i.e., statically partitioning the namespace).
We would like to remove the limitation of scaling the global namespace. Our
vision is that that HDFS should adopt a scalable underlying architecture that
allows the global namespace scales linearly.
We propose to implement the HDFS namespace on top of a key-value (KV) store.
Adopting the KV store interfaces allows HDFS to leverage the capability of
modern KV store and to become much easier to scale. Going forward, the
architecture allows distributing the namespace across multiple machines, or
storing only the working set in the memory (HDFS-5389), both of which allows
HDFS to manage billions of files using the commodity hardware available today.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
