[
https://issues.apache.org/jira/browse/MAPREDUCE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755406#action_12755406
]
Chris Douglas commented on MAPREDUCE-649:
-----------------------------------------
Just to make sure I understand the change, this validates the source and
destination checksums, retrying if they do not match. However, failure to match
the checksum doesn't count as a failed copy, but as an attempt in a different
retry mechanism. Are invalid/corrupt copies more common than other failures
that throw exceptions (e.g. (re)moved source files, IOExceptions, etc.)? This
overlapping retry logic- within a map and within a job- is a little odd. The
other refactoring changes look fine. Writing a unit test for this new
functionality would be both difficult and expensive, so as long as it's been
tested then that's OK.
> distcp should validate the data copied
> --------------------------------------
>
> Key: MAPREDUCE-649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-649
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: distcp
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Attachments: d_verify.patch, d_verify649.patch
>
>
> distcp should validate the files copied by checking the checksums, if the
> filesystem supports checksums.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
