[
https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228430#comment-13228430
]
Thomas Graves commented on MAPREDUCE-3893:
------------------------------------------
I think its more of being able to configure it differently per queue and get
better utilization then stopping deadlocks (Although it could also work around
deadlock issues). Right now there is only one setting so if you have queues
that are running very different jobs, its not very efficient. For instance,
what if one queue is running all uberAM's, You want that queue to have
maximum-am-resource-percent close to 100%, but if another queue on the same
cluster is running jobs with lots of maps/reduces per job you may only want it
at 10% because starting other AM's will just waste memory as they will have to
wait for the first job anyway. So lets say you set it at 50% to get somewhere
in the middle. That means you are wasting half of the queue that wants to run
all uberAM's. That 50% may also be to high if you have yet another queue that
is set to use 1% of the cluster capacity. Note that the calculation for
maxActiveApps is using the minimumAllocation setting, so if a user sets its AM
memory to be much more then that and the queue is small, you could deadlock
again.
Note that I'm mostly interested in maximum-am-resource-percent. The
maximum-applications we can leave alone.
I believe the way Eric has it only makes it more complicated if you want to
configure it per queue. The global is still there but if you want to override
then you can set it per queue. Obviously more settings does introduce the
chance for more mis-config but I don't see it anymore complicated then what is
already there per queue. I'm worried without this we won't be using the
cluster very efficiently or we will run into deadlocks because we set it high
but the smaller capacity queues won't be able to handle that.
If you have other ideas or thoughts please let us know.
> allow capacity scheduler configs maximum-applications and
> maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and
> maximum-am-resource-percent are currently configured globally and then made
> proportional to each queue based on its capacity. There are times when this
> may not work well. some exampless - if you have a queue that is running on
> uberAM jobs, the jobs a queue is running always has a small number of
> containers, and then you have the opposite where in a queue with very small
> capacity, you may want to limit the am resources even more so you don't end
> up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
