[
https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726768#action_12726768
]
Scott Carey commented on HADOOP-5170:
-------------------------------------
bq. You're right that we can't support Jonathan's use case, but it isn't a
valid use case for Hadoop. Hadoop is about sharing a cluster with other users.
Please don't assume everyone uses Hadoop the same way. Many users are more
than willing to tune a specific job flow on a cluster for max compute power per
$ and have little or no ad-hoc job submission. For some such a workflow is
business critical and the cost/benefit of heavy hand-tuning is obvious.
Yes, these tuning knobs are not good for all use cases and especially ad-hoc
shared cluster use.
Being able to specifically tune a business critical workflow is a very good
thing for some. There is a reason this JIRA gained a lot of watchers fast and
has the most votes for it.
A resource model will be very useful, but it will never beat hand tuning for
performance -- it will only get part way there -- and the two can be
cooperative as well.
Some clusters will be used in ways such that hand tuning is possible, others
won't.
Perhaps in the long term there could be a shared scheduler component that
contains all the task limits with a default implementation, and each scheduler
can optionally use, or override it.
> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
> Key: HADOOP-5170
> URL: https://issues.apache.org/jira/browse/HADOOP-5170
> Project: Hadoop Common
> Issue Type: New Feature
> Components: mapred
> Reporter: Jonathan Gray
> Assignee: Matei Zaharia
> Fix For: 0.21.0
>
> Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch,
> tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch,
> tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this. The focus of this
> jira should be on finding what would be the simplest to implement that would
> satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide
> maximum. It seems that for most uses, the former is preferable however
> either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs. I am forced to keep the max
> map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the
> Datanode and Regionserver. I have other jobs that are network latency bound
> and would like to be able to run high numbers of them concurrently on each
> node. Though I can thread some jobs, there are some use cases that are
> difficult to thread (scanning from hbase) and there's significant complexity
> added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have
> multiple reducers on a single node but other nodes that received none. A
> limit of 1 reducer per node for that job would prevent that from happening.
> (only works with per-node limit)
> - Poor mans MR job virtualization. Since we can limit a jobs resources, this
> gives much more control in allocating and dividing up resources of a large
> cluster. (makes most sense w/ cluster-wide limit)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.