[ https://issues.apache.org/jira/browse/MAPREDUCE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mohammad Kamrul Islam updated MAPREDUCE-6713: --------------------------------------------- Description: *Current state and shortcoming* ======================= By default, distcp writes temporary files into $TARGET_PATH/.distcp.tmp/$taskatttempttid. (See RetriableFileCopyCommand#getTmpFile). There is no way a user can override this staging/tmp directory. The problem is obvious in S3 with versioning. For example, user wants to turn on S3 versioning only for his target directory but not the staging/tmp directory. Current distcp also creates versioning for staging directory which can contain a lot of temporary files. If user can override this path by a non-versioned S3 path for staging, it will make things cleaner. *Proposed solution* ============== Provide a new option(-stage) where user can optionally provide a path from target FS. Distcp mapper tasks will write distcp temporary files into that directory. *Possible Confusions* ================= There is another distcp option (-tmp) which can be assumed to serve the same purpose. But this option works only with "-atomic" option which has a different meaning of temporary files. Another confusion could be the staging directory used by mapreduce framework. The proposed temp directory is for distcp specific. Working on a patch to upload. was: *Current state and shortcoming* ======================= By default, distcp writes temporary files into $TARGET_PATH/.distcp.tmp/$taskatttempttid. (See RetriableFileCopyCommand#getTmpFile). There is no way a user can override this staging/tmp directory. The problem is obvious in S3 with versioning. For example, user wants to turn on S3 versioning only for his target directory but not the staging/tmp directory. Current distcp also creates versioning for staging directory which can contain a lot of temporary files. If user can override this path by a non-versioned S3 path for staging, it will make things cleaner. *Proposed solution* ============== Provide a new option(-stage) where user can optionally provide a path from target FS. Distcp mapper tasks will write distcp temporary files into that directory. *Possible Confusions* ================= There is another distcp option (-tmp) which can be assumed to serve the same purpose. But this option works only with "-atomic" option which has a different meaning of temporary files. Another confusion could be the staging directory used by mapreduce framework. The proposed temp directory is for distcp specific. > Distcp doesn't provide any option to override the default staging directory > --------------------------------------------------------------------------- > > Key: MAPREDUCE-6713 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6713 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp > Affects Versions: 2.5.1 > Reporter: Mohammad Kamrul Islam > > *Current state and shortcoming* > ======================= > By default, distcp writes temporary files into > $TARGET_PATH/.distcp.tmp/$taskatttempttid. (See > RetriableFileCopyCommand#getTmpFile). There is no way a user can override > this staging/tmp directory. The problem is obvious in S3 with versioning. For > example, user wants to turn on S3 versioning only for his target directory > but not the staging/tmp directory. Current distcp also creates versioning for > staging directory which can contain a lot of temporary files. If user can > override this path by a non-versioned S3 path for staging, it will make > things cleaner. > > *Proposed solution* > ============== > Provide a new option(-stage) where user can optionally provide a path from > target FS. Distcp mapper tasks will write distcp temporary files into that > directory. > *Possible Confusions* > ================= > There is another distcp option (-tmp) which can be assumed to serve the same > purpose. But this option works only with "-atomic" option which has a > different meaning of temporary files. > Another confusion could be the staging directory used by mapreduce framework. > The proposed temp directory is for distcp specific. > Working on a patch to upload. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org