[
https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842604#action_12842604
]
Ravi Gummadi commented on MAPREDUCE-653:
----------------------------------------
Yes. But we need to make sure that a particular distcp-Job's task on a node is
not hogging the whole bandwidth available for the node(just to be fair to the
other job's tasks running on this node).
Also at the time of launching the distcp job, we will not have enough
information on the number of tasks of this job that will run paralelly on the
cluster at different points of time(because (a) at some point few tasks of this
distcp job could be running --- allowing tasks of this distcp job to use high
bandwidth per task and (b) at some other time, different number of tasks of
this distcp job running --- allowing tasks of this distcp job to use low
bandwidth per task).
So if sleeping by map task of distcp job looks bad, then distcp needs to find
out a way of identifying the number of map tasks to be run(actually, number of
map tasks that can run paralelly on the cluster at any point of time ?) so that
it can set the bandwidth usage per task properly with lower sleep time.
> distcp can support bandwidth limiting
> -------------------------------------
>
> Key: MAPREDUCE-653
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-653
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: distcp
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch
>
>
> distcp should support an option for user to specify the bandwidth limit for
> the distcp job.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
