[ https://issues.apache.org/jira/browse/HDFS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405654#comment-13405654 ]
Jesse Yates commented on HDFS-3370: ----------------------------------- Sorry for the slow reply, been a bit busy of late... @Daryn bq. Retaining ref-counted paths after deletion in the origin namespace requires an "inode id". A new api to reference paths based on the id is required. We aren't so soft anymore... That's why I'd argue for doing it in file meta-data with periodic rewrites so we can just do appends. We will still need to maintain references if we do hardlinks, so this is just a single method call to do the update - arguably a pretty simple code path that doesn't need to be that highly optimized for multi-writers since we can argue that hardlinks are "rare". bq. The inode id needs to be secured since it bypasses all parent dir permissions, Yeah, thats a bit of a pain... Maybe a bit more metadate to store with the file...? @Konstantin bq. Do I understand correctly that your hidden inodes can be regular HDFS files, and that then the whole implementation can be done on top of existing HDFS, as a stand alone library supporting calls Yeah, I guess that's a possibility. But you would probably need to have some sort of "namespace managers" to deal with handling hardlinks across different namespaces, which fits comfortably with the distributed namenode design. bq. ref-counted links, creating hidden "only accessible to the namenode" inodes, leases on arbitrated NN ownership, retention of deleted files with non-zero ref count, etc. Those aren't client-side operations. Since you keep the data along with the file (including the current file owner), you could do it all from a library. However, since the lease needs to be periodically regained, you will see temporary unavailability in the hardlinked files in the managed namespace. If you couple the hardlink management with the namenode managing the space, you can then do forced-resassign of the hardlinks to the back-up namendoe and still see the availability as you would files in that namespace, in terms of creating new hardlinks (reads would still work since all the important data can be replicated across the different namespaces). @Andy: I don't know if I've seen a compelling reason that we _need_ to have cross-namespace hardlinks, particularly since they are _hard_, to say the least. > HDFS hardlink > ------------- > > Key: HDFS-3370 > URL: https://issues.apache.org/jira/browse/HDFS-3370 > Project: Hadoop HDFS > Issue Type: New Feature > Reporter: Hairong Kuang > Assignee: Liyin Tang > Attachments: HDFS-HardLink.pdf > > > We'd like to add a new feature hardlink to HDFS that allows harlinked files > to share data without copying. Currently we will support hardlinking only > closed files, but it could be extended to unclosed files as well. > Among many potential use cases of the feature, the following two are > primarily used in facebook: > 1. This provides a lightweight way for applications like hbase to create a > snapshot; > 2. This also allows an application like Hive to move a table to a different > directory without breaking current running hive queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira