[ https://issues.apache.org/jira/browse/MAPREDUCE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728485#action_12728485 ]
Ravi Gummadi commented on MAPREDUCE-650: ---------------------------------------- Since with -update, after copying files to stagedir, moving from stagedir to actual destinationDir cannot be atomic. So am planning to change the patch to support -atomic <stageDir> option only without -update. If both options are used together, distcp would give error message and exits. > Add atomic move option > ---------------------- > > Key: MAPREDUCE-650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-650 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp > Reporter: Richard Theige > Assignee: Ravi Gummadi > Attachments: d_retries_atomic.patch, d_retries_atomic_v1.patch > > > Provide support for update to move directories/files atomically by copying > the src directory to a tmp directory (with random/unique name) then move the > directory to its target destination name after all subdirs/files are copied > and verified. > example option ideas > hadoop ... distcp -update -move src dst > or > hadoop ... distcp -update -atomic src dst > to assure file correctness at the destination, before distcp performs the > 'move' at the end of the copy process, it should first perform a strong > signature/cksum (e.g. MD4) on the files. > The issue/need for this is that applications may attempt to start processing > data (because files are present), prior to completion of a whole directory > copy -- resulting in work against an incomplete data set. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.