[ 
https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726602#action_12726602
 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

I wrote this patch because of demand for this feature from many users, 
including Cloudera customers, who had reasons to limit the number of tasks per 
job. If the patch interacts poorly with the capacity scheduler, wouldn't it 
make more sense to tell users of the capacity scheduler not to use this 
feature, or to improve MAPREDUCE-516 so that it can take into account task 
limits?

I agree that task limits are not the best solution for dealing with high-memory 
jobs, but that's not the only situation that this patch addresses. Here are 
some others:
* The situation described in the original task was that a task is very 
CPU-intensive. The user wanted to limit the number of those tasks running to 
less than the number of cores. However, there is no need to fill up all slots 
on the machine - the other slots could be used for less CPU-intensive tasks. 
Until there is a model for all system resources in the capacity scheduler, this 
patch lets users with that kind of problem achieve reasonable behavior.
* One situation in which you want to limit tasks on the whole cluster is if you 
have a housekeeping job with long tasks, e.g. a database import or export. If 
you don't have a limit on running tasks, such a job can take up the whole 
cluster and hold it for a long time. The capacity and fair  schedulers include 
a preemption feature to kill tasks, but preemption has some problems: First, 
the preemption timeout generally shouldn't be set to less than 5-10 minutes so 
it doesn't interact badly with faulty tasktracker timeouts, and second, 
preemption might complicate writing certain jobs (e.g. database import/export).

I also think the problems Arun brought up in the capacity scheduler aren't 
insurmountable. To deal with the locality problem, you can give up slots when 
you run out of local data on them, like MAPREDUCE-548. To deal with the pinning 
problem, preemption might help.

More importantly, I think Arun's issues will come up even if task limits aren't 
used. For example, suppose that a given queue's capacity is 25% of the cluster, 
say 250 slots, and that all the other queues are filled with jobs, so there is 
never excess capacity. Then if a high-memory job is submitted to the queue with 
25% capacity, it has to behave exactly as if it has a limit of 250 tasks 
cluster-wide. The same problems that Arun brought up will happen: locality will 
be poor, the job might make bad bets, etc. This situation will be far more 
common than a user explicitly setting a limit on running tasks for the job, so 
if these two concerns are serious, maybe more work needs to be put into 
MAPREDUCE-516 to prevent them from happening. In particular, I think limiting 
the number of "bets" based on the job's capacity or task limit, allowing the 
job to kill tasks if it has waited for too long on a bet, and implementing 
MAPREDUCE-548 would solve much of the problem.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, 
> tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, 
> tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this 
> jira should be on finding what would be the simplest to implement that would 
> satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide 
> maximum.  It seems that for most uses, the former is preferable however 
> either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max 
> map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the 
> Datanode and Regionserver.  I have other jobs that are network latency bound 
> and would like to be able to run high numbers of them concurrently on each 
> node.  Though I can thread some jobs, there are some use cases that are 
> difficult to thread (scanning from hbase) and there's significant complexity 
> added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have 
> multiple reducers on a single node but other nodes that received none.  A 
> limit of 1 reducer per node for that job would prevent that from happening. 
> (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this 
> gives much more control in allocating and dividing up resources of a large 
> cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to