[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627188#comment-16627188
]
Antal Bálint Steinbach commented on YARN-8468:
----------------------------------------------
Hi [~cheersyang] ,
I think I got your point. We do normalization stuff in
DefaultAMSProcessor.allocate() at #239 then call scheduler.allocate from there
where we do normalization again. Furthermore, in DefaultAMSProcessor an
exception will be thrown if the allocation is not correct so we will never get
to the scheduler.allocate() call. The question is that will ever
scheduler.allocate() be called with non-normalized requests?
My main idea was during my implementation is to prevent any change in
functionality or structure and protect Non-FairScheduler scheduler
implementations from changes. What I did is that I followed the usage of the
configuration called "yarn.scheduler.maximum-allocation-mb". Everywhere where I
found scheduler level capacity calculation I replaced it with queue or
scheduler level calculation.
This way I think the patch does what it was planned to do, also It did not
break older functionality and does not have side effects.
Also, I totally agree with you [~cheersyang] , we should revisit the
normalization and validation as It is clearly something which is a bit
overcomplicated. So what do you think about opening a new Jira (assign it to
me) for resolving your concerns, because this patch is already big enough and
the problem to me looks like a refactoring old behavior issue?
> Enable the use of queue based maximum container allocation limit and
> implement it in FairScheduler
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.1.0
> Reporter: Antal Bálint Steinbach
> Assignee: Antal Bálint Steinbach
> Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch,
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch,
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch,
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch,
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch,
> YARN-8468.014.patch, YARN-8468.015.patch, YARN-8468.016.patch,
> YARN-8468.017.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
> The goal of this ticket is to allow this value to be set on a per queue basis.
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
> Suggested solution:
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability(String queueName) in both
> FSParentQueue and FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * Enforce the use of queue based maximum allocation limit if it is
> available, if not use the general scheduler level setting
> ** Use it during validation and normalization of requests in
> scheduler.allocate, app submit and resource request
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
