[
https://issues.apache.org/jira/browse/MAPREDUCE-653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842218#action_12842218
]
eric baldeschwieler commented on MAPREDUCE-653:
-----------------------------------------------
So how will this behave if distcp gets 100 slots and is limited to 100mb? Will
you use 100 slots badly?
Typically we've handled the need for throttling by controlling the bandwidth /
client and the number of clients. This lets you reason about the utilization
of the clients & bandwidth. This would imply a fixed number of maps that pull
files to copy off a queue maintained by the distcp client. You'd need to
handle recovery carefully, but that could probably be done at the distcp client
(the slave signals done, the client does a final move and updates the queue).
> distcp can support bandwidth limiting
> -------------------------------------
>
> Key: MAPREDUCE-653
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-653
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: distcp
> Reporter: Ravi Gummadi
> Assignee: Ravi Gummadi
> Attachments: d_bw.patch, d_bw.v1.patch, d_bw.v2.patch
>
>
> distcp should support an option for user to specify the bandwidth limit for
> the distcp job.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
