[
https://issues.apache.org/jira/browse/MAPREDUCE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609095#comment-13609095
]
Srikanth Sundarrajan commented on MAPREDUCE-5014:
-------------------------------------------------
A revised patch is now uploaded.
With this patch, it should be possible to pass -Ddistcp.copy.listing.class
argument to the DistCp command (or on the passed configuration in the DistCp()
constructor if using API) to use the new CopyListing. The custom copy listing
if extending SimpleCopyListing would use the list of source paths as input by
default. If the custom copy listing, requires input to be passed any other way,
the same would have to be passed through OPTS (-D params) during invocation.
The custom copy listing class can be included in the classpath by including the
same via export HADOOP_CLASSPATH
> Extending DistCp through a custom CopyListing is not possible
> -------------------------------------------------------------
>
> Key: MAPREDUCE-5014
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5014
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
> Affects Versions: 0.23.0, 0.23.1, 0.23.3, trunk, 0.23.4, 0.23.5
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Attachments: MAPREDUCE-5014.patch, MAPREDUCE-5014.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> * While it is possible to implement a custom CopyListing in DistCp, DistCp
> driver class doesn't allow for using this custom CopyListing.
> * Allow SimpleCopyListing to provide an option to exclude files (For instance
> it is useful to exclude FileOutputCommiter.SUCCEEDED_FILE_NAME during copy as
> premature copy can indicate that the entire data is available at the
> destination)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
