[
https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726674#action_12726674
]
Owen O'Malley commented on HADOOP-5170:
---------------------------------------
The problem with this patch is that it doesn't do anything useful and gets in
the way of real fixes to the problem. Limiting the number of running tasks per
a job is *not* what any users need. It is being used as an expedient way to
hack in global resource limits. If I launch two cpu hungry jobs (even with a
single user!), I will kill those nodes. That is *not* ok. While MAPREDUCE-516
addresses high-ram jobs, it takes the view that each slot has a set of
resources and that a given job's tasks may require multiple slots. This *is* a
reasonable model that is applicable for cpu, ram, or disk. It would be
reasonable to change the interface to MAPREDUCE-516 to be in terms of the slots
per a task instead of memory.
Clearly in the long term, I think that a more dynamic model that tracks the
resources being consumed and launches new tasks appropriately. In the mean
time, MAPREDUCE-516 is a much better approach to this.
{quote}
This is an optional feature - users don't have to use it and it is turned off
by default. It has no performance impact for those who don't enable it. On the
other hand there are a substantial number of users who are using it (or
interested in using it), and would be left with no immediate alternative if it
were pulled from the next release.
We can leave this feature in until there is a suitable replacement, after which
time it can be deprecated so users can migrate to the replacement. Could that
work?
{quote}
-1 Because once it is in, we need to support it. When it doesn't work for the
users, they'll try and fix it. We already have users confused by too many
knobs...
> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
> Key: HADOOP-5170
> URL: https://issues.apache.org/jira/browse/HADOOP-5170
> Project: Hadoop Common
> Issue Type: New Feature
> Components: mapred
> Reporter: Jonathan Gray
> Assignee: Matei Zaharia
> Fix For: 0.21.0
>
> Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch,
> tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch,
> tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this. The focus of this
> jira should be on finding what would be the simplest to implement that would
> satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide
> maximum. It seems that for most uses, the former is preferable however
> either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs. I am forced to keep the max
> map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the
> Datanode and Regionserver. I have other jobs that are network latency bound
> and would like to be able to run high numbers of them concurrently on each
> node. Though I can thread some jobs, there are some use cases that are
> difficult to thread (scanning from hbase) and there's significant complexity
> added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have
> multiple reducers on a single node but other nodes that received none. A
> limit of 1 reducer per node for that job would prevent that from happening.
> (only works with per-node limit)
> - Poor mans MR job virtualization. Since we can limit a jobs resources, this
> gives much more control in allocating and dividing up resources of a large
> cluster. (makes most sense w/ cluster-wide limit)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.