[ 
https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726751#action_12726751
 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

{quote}
I have thousands of jobs submitted to my cluster every day and prior to this 
patch I was unable to get to 25% the capacity I have now because of the 
characteristics of a very small minority of the total number of jobs.

I need a way to prevent CPU intensive jobs from eating all the cores. If I have 
20 nodes and 100 CPU bound jobs, i only want 1 per node at a time. That strikes 
me as a common use case, not an invalid one. At the same time I'd like a way to 
allow 10 jobs per node of network bound jobs that do little else but wait 
around.
{quote}

Jonathan, Can you please sketch the scenario you face in more detail?

I'll hazard a *guess* and ask you to consider using high-ram jobs for your CPU 
intensive jobs and use MAPREDUCE-532  to limit how much of the cluster they 
occupy. (I'm reading that you have a small minority of such jobs from your 
comment).

For e.g. create a 'cpu-heavy' queue and limit it's capacity as a fraction of 
your total capacity using MAPREDUCE-532. Then configure your CPU-intensive jobs 
to use up a majority of the slots in each node (thus preventing 2 slots of the 
same job from being scheduled on the same node) and submit these jobs to the 
'cpu-heavy' queue.

Yes, you will not be able to run too many other tasks on those nodes. But with 
MAPREDUCE-532 you can ensure that it doesn't affect too many nodes in your 
cluster.

Will that work?



> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, 
> tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, 
> tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this 
> jira should be on finding what would be the simplest to implement that would 
> satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide 
> maximum.  It seems that for most uses, the former is preferable however 
> either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max 
> map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the 
> Datanode and Regionserver.  I have other jobs that are network latency bound 
> and would like to be able to run high numbers of them concurrently on each 
> node.  Though I can thread some jobs, there are some use cases that are 
> difficult to thread (scanning from hbase) and there's significant complexity 
> added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have 
> multiple reducers on a single node but other nodes that received none.  A 
> limit of 1 reducer per node for that job would prevent that from happening. 
> (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this 
> gives much more control in allocating and dividing up resources of a large 
> cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to