[
https://issues.apache.org/jira/browse/MAPREDUCE-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140953#comment-16140953
]
Harsh J commented on MAPREDUCE-2149:
------------------------------------
I don't appear to have rights to change the resolution status here, but note
that the true resolution of this JIRA is not "Fixed" as indicated, but instead
"Duplicate" (of MAPREDUCE-2765, which introduces DistCp2 for YARN/MR2).
> Distcp : setup with update is too slow when latency is high
> -----------------------------------------------------------
>
> Key: MAPREDUCE-2149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2149
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.20.2, 0.21.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Attachments: MAPREDUCE-2149.patch
>
>
> If you run distcp with '-update' option, for _each of the files_ present on
> source cluster setup invokes a separate RPC to destination cluster to fetch
> file info.
> Usually this overhead is not very noticeable when both cluster are
> geographically close to each other. But if the latency is large, setup could
> take couple of orders of magnitude longer.
> E.g. : source has 10k directories, each with about 10 files, round trip
> latency between source and destination is 75 ms (typical for coast-to-coast
> clusters).
> If we run distcp on source cluster, set up would take about _2.5 hours_
> irrespective of whether destination has these files or not. '-lsr' on the
> same dest dir from source cluster would take up to 12 min (depending on how
> many directories already exist on dest).
> * A fairly simple fix to how setup() iterates should bring the set up time
> to same as '-lsr'. I will have a patch for this.. (though 12 min is too
> large).
> * A more scalable option is to differ update check to mappers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
