[
https://issues.apache.org/jira/browse/YARN-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252632#comment-17252632
]
Siddharth Ahuja commented on YARN-10528:
----------------------------------------
Hey [~snemeth],
Nice catch! Indeed, if there is no exception thrown from the source code even
with a test setup that is designed to have the exception thrown (because
maxAMShare is defined inside a parent queue), then, the tests would still pass
incorrectly because there is no fail(...) in test case logic to double check
the "throwing of the exception". As such, these tests will fail to identify a
bad update to the source code that may not result in any exception.
I have gone ahead and updated the tests as per your suggestion now.
In regards to doing a backport to the earlier branches, I will apply the patch
to these now and run the JUnits. Once these are passed I will upload the
patches for the relative branches as well soon.
Thanks again for reviewing!
> maxAMShare should only be accepted for leaf queues, not parent queues
> ---------------------------------------------------------------------
>
> Key: YARN-10528
> URL: https://issues.apache.org/jira/browse/YARN-10528
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Siddharth Ahuja
> Assignee: Siddharth Ahuja
> Priority: Major
> Attachments: YARN-10528.001.patch, YARN-10528.002.patch, maxAMShare
> for root.users (parent queue) has no effect as child queue does not inherit
> it.png
>
>
> Based on [Hadoop
> documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html],
> it is clear that {{maxAMShare}} property can only be used for *leaf queues*.
> This is similar to the {{reservation}} setting.
> However, existing code only ensures that the reservation setting is not
> accepted for "parent" queues (see
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L226
> and
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java#L233)
> but it is missing the checks for {{maxAMShare}}. Due to this, it is
> currently possible to have an allocation similar to below:
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <allocations>
> <queue name="root">
> <weight>1.0</weight>
> <schedulingPolicy>drf</schedulingPolicy>
> <aclSubmitApps>*</aclSubmitApps>
> <aclAdministerApps>*</aclAdministerApps>
> <queue name="default">
> <weight>1.0</weight>
> <schedulingPolicy>drf</schedulingPolicy>
> </queue>
> <queue name="users" type="parent">
> <weight>1.0</weight>
> <schedulingPolicy>drf</schedulingPolicy>
> <maxAMShare>1.0</maxAMShare>
> </queue>
> </queue>
> <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
> <queuePlacementPolicy>
> <rule name="specified" create="true"/>
> <rule name="nestedUserQueue" create="true">
> <rule name="default" create="true" queue="users"/>
> </rule>
> <rule name="default"/>
> </queuePlacementPolicy>
> </allocations>
> {code}
> where {{maxAMShare}} is 1.0f meaning, it is possible allocate 100% of the
> queue's resources for Application Masters. Notice above that root.users is a
> parent queue, however, it still gladly accepts {{maxAMShare}}. This is
> contrary to the documentation and in fact, it is very misleading because the
> child queues like root.users.<user> actually do not inherit this setting at
> all and they still go on and use the default of 0.5 instead of 1.0, see the
> attached screenshot as an example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
