[ https://issues.apache.org/jira/browse/HDFS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127543#comment-15127543 ]
M. C. Srivas commented on HDFS-2139: ------------------------------------ Just a curiosity ... why was this method chosen? Why not simply implement hard-links at the HDFS file level? Then a simple NN transaction is sufficient to create the new pathname and add a ref count to the Inode. What is the benefit of doing it this way? Is it because we are going across two different NNs? If so, how do you prevent a simultaneous delete of the file at the srcNN from running ahead and removing some of the blocks of the src file? > Fast copy for HDFS. > ------------------- > > Key: HDFS-2139 > URL: https://issues.apache.org/jira/browse/HDFS-2139 > Project: Hadoop HDFS > Issue Type: New Feature > Reporter: Pritam Damania > Assignee: Rituraj > Attachments: HDFS-2139-For-2.7.1.patch, HDFS-2139.patch, > HDFS-2139.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > There is a need to perform fast file copy on HDFS. The fast copy mechanism > for a file works as > follows : > 1) Query metadata for all blocks of the source file. > 2) For each block 'b' of the file, find out its datanode locations. > 3) For each block of the file, add an empty block to the namesystem for > the destination file. > 4) For each location of the block, instruct the datanode to make a local > copy of that block. > 5) Once each datanode has copied over its respective blocks, they > report to the namenode about it. > 6) Wait for all blocks to be copied and exit. > This would speed up the copying process considerably by removing top of > the rack data transfers. > Note : An extra improvement, would be to instruct the datanode to create a > hardlink of the block file if we are copying a block on the same datanode -- This message was sent by Atlassian JIRA (v6.3.4#6332)