[
https://issues.apache.org/jira/browse/MAPREDUCE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rosie Li updated MAPREDUCE-2257:
--------------------------------
Release Note:
copy file parallel by first chopping files into chunks, copy them and stitch
the file chunks back into files in the end.
By default, distcp.copy.by.chunk is set to true in the configuration. The user
can set it to false to use the original distcp. But the type of destination
will be checked afterward. distcp.copy.by.chunk will remain true only if the
destination file system is the distributed file system(where we can use the
concat method to stitch the file chunks together in the end)
was:
copy file parallel by first chopping files into chunks, copy them and stitch
the file chunks back into files in the end.
> distcp can copy blocks in parallel
> ----------------------------------
>
> Key: MAPREDUCE-2257
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2257
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: distcp
> Affects Versions: 0.21.0
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
