[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623426#comment-16623426
]
Weiwei Yang commented on YARN-8468:
-----------------------------------
Thanks [~bsteinbach] for the updates.
One thing I am not sure, as {{ApplicationMasterProtocol}} is the only contract
we have for requests, isn't enough to just do the normalization in AMS
processor's allocate path? Why we need to do that again in scheduler? I know
this is not introduced by this patch, but just want to understand the idea of
normalization and see if it is possible to avoid changing the scheduler API
{{YarnScheduler}}.
And now it seems not very consistent in FS and CS
CS doesn't normalize with max allocation at queue level
{code:java}
public Allocation allocate(...) {
....
// Sanity check for new allocation requests
normalizeResourceRequests(ask);
// Normalize scheduling requests
normalizeSchedulingRequests(schedulingRequests);
...
}{code}
But in FS it does,
{code:java}
public Allocation allocate(....) {
....
// Sanity check
normalizeResourceRequests(ask, queue.getName());
....
}{code}
am I missing anything [~leftnoteasy], [~bsteinbach]?
And some other minor comments:
1) TestApplicationMasterServiceWithFS
{code:java}
String TEST_DIR = new File(System.getProperty("test.build.data",
"/tmp")).getAbsolutePath();
{code}
can be replaced by
{code:java}
GenericTestUtils.getTestDir("xxx");
{code}
also pls remember to cleanup the test dir in teardown. This comment applies to
{{TestAppManagerWithFairScheduler}}
2) Instead of pulling {{TestRMAppManager}} out as a separate class, is it
better to create a class {{AppManagerTestBase}}, and move that into it as a
inner class. Then let {{TestAppManager}} and
{{TestAppManagerWithFairScheduler}} both extend {{AppManagerTestBase.}}
{{TestRMAppManager}} sounds like a test class too much.
Please take one more look at the checkstyle issue, and fix as best as you can.
e.g I see some unused imports in e.g {{TestRMAppManager}} and
{{TestAppManager}}.
Thanks.
> Enable the use of queue based maximum container allocation limit and
> implement it in FairScheduler
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.1.0
> Reporter: Antal Bálint Steinbach
> Assignee: Antal Bálint Steinbach
> Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch,
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch,
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch,
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch,
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch,
> YARN-8468.014.patch, YARN-8468.015.patch, YARN-8468.016.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
> The goal of this ticket is to allow this value to be set on a per queue basis.
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
> Suggested solution:
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability(String queueName) in both
> FSParentQueue and FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * Enforce the use of queue based maximum allocation limit if it is
> available, if not use the general scheduler level setting
> ** Use it during validation and normalization of requests in
> scheduler.allocate, app submit and resource request
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
