[
https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726179#action_12726179
]
Arun C Murthy commented on HADOOP-5170:
---------------------------------------
bq. I'm sorry I'm coming in late on this.
Ditto.
bq. Are these new knobs at the job level? I think this the wrong direction. In
particular, just limiting the number of slots won't do any good. The high ram
job processing is a much better model. So rather than declaring max number of
maps or reduces, we should allow "large task" jobs where each task is given
multiple slots.
Couldn't agree more. See MAPREDUCE-516.
I see the following issues:
mapred.max.{maps|reduces}.per.node
As of today, the framework will not be able to guard against users who take up
a node and consume all resources (cpu, memory, disk etc.) and starve other user
tasks running on the machines. This goes against the spirit of shared
compute/storage clusters. I can see an argument being made for this feature
once we can figure how to *charge* the user based on the task's total resource
consumption; however we are a long way away from this. We have taken a step
along this direction by introducing the High RAM Jobs feature in the Capacity
Scheduler, we have a long way to go.
mapred.max.{maps|reduces}.per.cluster
Given that we mostly agree that the high-ram jobs are the right model, these
features interact very badly with each other.
Consider a high-ram job which has mapred.max.maps.per.cluster set to 100 and a
few thousand tasks (a fraction of which is sufficient to exhaust the capacity
of it's queue).
Currently the CapacityScheduler starts 'reserving' slots (after charging the
user, queue etc.) to satisfy the resource requirements of the job.
Now we have 2 choices if we choose to incorporate
mapred.max.{map|reduces}.per.cluster:
# Do not 'reserve' more tasktrackers than mapred.max.maps.per.cluster
# Reserve as much as needed but start 'unreserving' as soon as we have
scheduled 'mapred.max.maps.per.cluster' number of maps.
Either way we are in trouble.
# The high-ram job is seriously starved. The scheduler has to pick only 100
nodes and no more. If they happened to be bad bets (other long running tasks
etc.) the high-ram job needs to wait for a long while. Furthermore, the other
bad side-effect is that the high-ram job gets *pinned* to the first 100 nodes
and this really hurts locality of its tasks.
# The cluster is severely under-utilized since we may reserve all of the
queue's capacity and then realize that we have to start 'unreserving'. Once the
first wave of maps of the high-ram job are done we rinse and repeat for each
wave of mapred.max.maps.per.cluster maps, there-by keeping the cluster idle for
large amounts of time.
bq. I think we should revert this before it is released.
Unfortunately, yes. +1
> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
> Key: HADOOP-5170
> URL: https://issues.apache.org/jira/browse/HADOOP-5170
> Project: Hadoop Common
> Issue Type: New Feature
> Components: mapred
> Reporter: Jonathan Gray
> Assignee: Matei Zaharia
> Fix For: 0.21.0
>
> Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch,
> tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch,
> tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this. The focus of this
> jira should be on finding what would be the simplest to implement that would
> satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide
> maximum. It seems that for most uses, the former is preferable however
> either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs. I am forced to keep the max
> map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the
> Datanode and Regionserver. I have other jobs that are network latency bound
> and would like to be able to run high numbers of them concurrently on each
> node. Though I can thread some jobs, there are some use cases that are
> difficult to thread (scanning from hbase) and there's significant complexity
> added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have
> multiple reducers on a single node but other nodes that received none. A
> limit of 1 reducer per node for that job would prevent that from happening.
> (only works with per-node limit)
> - Poor mans MR job virtualization. Since we can limit a jobs resources, this
> gives much more control in allocating and dividing up resources of a large
> cluster. (makes most sense w/ cluster-wide limit)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.