jay vyas created HDFS-5909:
------------------------------
Summary: NameNode : Can some features be implemented in a pure
YARN app, to synergize broader uniformity for HCFS distributed FS metadata
operations?
Key: HDFS-5909
URL: https://issues.apache.org/jira/browse/HDFS-5909
Project: Hadoop HDFS
Issue Type: Wish
Reporter: jay vyas
The HDFS NameNode is a major barrier to HCFS FileSystem compatibility. Magic
that occurs inside of it is :
1) Essential to certain performant applications of hadoop (i.e. hbase).
2) An extension of FileSystem interface properties which isn't necessarily
dependant on HDFS.
For example, in HDFS-5902, the idea of atomic directory swapping comes up.
This can be done in HDFS, but also, maybe in a generic YARN app which provides
general distributed FS metadata services to any HCFS provider.
In addition to making NameNode easier to maintain, this would gaurantee broader
compatibility of the HCFS ecosystem.
Now, just brainstorming here, but... Lets take a typical FS operation: High
volume writes. In a distributed file system, these can cause performance
issues when lots of files are created. YARN itself provides a framework which
is generic enough , that we can write a YARN service which, when running, could
generically implement certain distributed FS metadata operations. Then, HCFS
providers would be obligated to provide hooks in their API implementations to
update the running YARN distributed metadata services.
Obviously implementation details are complex.
But would it be possible to offload some of the complexity of NameNode into a
YARN utility which the HCFS community can leverage?
This is a very raw idea, and if its impossible or not practical, I'm open to
feedback. But in the end I think it could be significant innovation both for
HDFS (it would make it more maintainable by offloading shared meta data
operations), as well as other HCFS tools (because we would be able to share
some of the logic that HDFS implements for synchronizing distribtued FS
metadata).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
