[
https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603642#comment-13603642
]
Suresh Srinivas commented on HDFS-4489:
---------------------------------------
h2. Introduction
This jira proposes to introduce InodeID in HDFS. InodeID uniquely identifies an
instance of an inode such as directory, file, symbolic links etc. independent
of the file path name. This helps in several use cases:
# HDFS can evolve to support ID based protocols such as NFS. We plan to add an
experimental NFS V3 gateway to HDFS using this mechanism. Will post a github
link soon.
# InodeID can be used by the tools to track a single instance of a file, for
cacheing data or tracking and checking for modification based on INodeID, in
tools like distcp.
# Path cannot identify a unique instance of a file. This causes issues as
described in HDFS-4258 and HDFS-4437. It has also been a requirement of many
other jiras such as HDFS-385.
# Using InodeID as an identifier instead of path can be more efficient than
path bases accesses.
h4. New Inode identifier
It is 64 bit long number and has the following properties:
# Number 0 is reserved and would be used for identifying invalid/default value.
# Numbers 1-1023 are reserved for some unforeseen future requirements. The
InodeID starts from 1024.
# An InodeID is never re-used in a single namenode namespace.
h3. General overview of the changes required
Applications discover the InodeID by getting the FileStatus for an Inode or
when an Inode is created or opened for append. FileStatus will be changed to
include InodeID. Create and append will be changed to return FileStatus as well.
For other APIs that use path (Path or String) to identify a file we have two
choices:
# *API Change*: Add another variant of the API that uses InodeID to identify a
file or add additional parameter InodeID to the API.
# *No API Change*: Use the path to send the InodeID and keep the API changes to
a minimum. Example, one could use path {{/.inodes/<inodesID>}}, where ".inodes"
is a reserved name to identify the path that pass InodeID instead of regular
path. This similar to /proc used for special purposes on *nix.
h4. InodeID to Inode map
A new map (based on GSet) will be introduced in the namenode to map a given
InodeID to an Inode. This is in addition to existing map of BlockID to
BlockInfo.
h4. Additional memory consumption
Adding all this will require additional memory in the namenode.
* 8 byte InodeID into Inode object results in a cost of 8 bytes per Inode. As
proposed in HDFS-4258, this can be folded into existing modification and access
time.
* Introducing InodeID to Inode GSet results in additional memory of 16 bytes
per Inode:
** 8 * size of Gset (where size of GSet could be as big as number of Inodes)
** 8 bytes per Inode for a java reference (pointer to next element as required
by GSet)
Inodes and related objects consume approximately 1/3 of the JVM heap in a
system that is full. Inode size is ~180 bytes and this proposal adds 16-24
bytes per Inode. With change the JVM heap needs to be increased by 3% to 4.5%.
While further optimizations are possible to reduce this size further, it adds
needless code complexity.
h4. Security concerns
InodeID is an alternate name similar to path. All the existing security
mechanism that applies to path (that is ensuring permissions are checked from
the root to the Inode) will also be done for InodeID based access.
> Use InodeID as as an identifier of a file in HDFS protocols and APIs
> --------------------------------------------------------------------
>
> Key: HDFS-4489
> URL: https://issues.apache.org/jira/browse/HDFS-4489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Reporter: Brandon Li
> Assignee: Brandon Li
>
> The benefit of using InodeID to uniquely identify a file can be multiple
> folds. Here are a few of them:
> 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258,
> HDFS-4437.
> 2. modification checks in tools like distcp. Since a file could have been
> replaced or renamed to, the file name and size combination is no t reliable,
> but the combination of file id and size is unique.
> 3. id based protocol support (e.g., NFS)
> 4. to make the pluggable block placement policy use fileid instead of
> filename (HDFS-385).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
