[
https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934045#comment-15934045
]
Omkar Aradhya K S edited comment on HADOOP-11794 at 3/21/17 3:46 AM:
---------------------------------------------------------------------
I was trying to evaluate your patch with ADLS:
Tried the bits on a HDInsight 3.5 cluster (this comes with hadoop 2.7)
Observed following compatibility issues:
a. You are checking for instance of *DistributedFileSystem* in many places
and all other *FileSystem* implementations don’t implement
*DistributedFileSystem*
i. Could this be changed to something more compatible with other
*FileSystem* implementations?
b. You are using the new *DFSUtilClient*, which makes DistCp incompatible
with older versions of Hadoop
i. Can this be changed to be backward compatible?
If the compatibility issues are addressed, the DistCp with your feature would
be available for other *FileSystem* implementations and also would be backward
compatible.
was (Author: omkarksa):
I was trying to evaluate your patch with ADLS:
Tried the bits on a HDInsight 3.5 cluster (this comes with hadoop 2.7)
Observed following compatibility issues:
a. You are checking for instance of {code}DistributedFileSystem{code} in
many places and all other {code}FileSystem{code} implementations don’t
implement {code}DistributedFileSystem{code}
i. Could this be changed to something more compatible with other
{code}FileSystem{code} implementations?
b. You are using the new {code}DFSUtilClient{code}, which makes DistCp
incompatible with older versions of Hadoop
i. Can this be changed to be backward compatible?
If the compatibility issues are addressed, the DistCp with your feature would
be available for other {code}FileSystem{code} implementations and also would be
backward compatible.
> distcp can copy blocks in parallel
> ----------------------------------
>
> Key: HADOOP-11794
> URL: https://issues.apache.org/jira/browse/HADOOP-11794
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
> Affects Versions: 0.21.0
> Reporter: dhruba borthakur
> Assignee: Yongjun Zhang
> Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch,
> HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch,
> HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch,
> MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are
> greater than 1 TB with a block size of 1 GB. If we use distcp to copy these
> files, the tasks either take a long long long time or finally fails. A better
> way for distcp would be to copy all the source blocks in parallel, and then
> stich the blocks back to files at the destination via the HDFS Concat API
> (HDFS-222)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
