[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264616#comment-17264616
]
zhuqi commented on YARN-10506:
------------------------------
[~wangda] [~gandras]
I have updated a patch according to [~wangda] comment, i think it's more
reasonable in
*1) How we deal with "create" flag of ApplicationPlacementContext?*
1.autoCreateLeafQueue:
{code:java}
private LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
String leafQueueName = placementContext.getQueue();
String parentQueueName = placementContext.getParentQueue();
ApplicationPlacementContext apc =
new ApplicationPlacementContext(placementContext);
if (!StringUtils.isEmpty(parentQueueName)) {
CSQueue parentQueue = getQueue(parentQueueName);
CSQueue leafQueue = getQueue(leafQueueName);
// Check if parent should not be null
// when cant't auto create parent.
if (parentQueue == null && !apc.isCreateParentQueue()) {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping specifies an invalid parent queue "
+ "which does not exist or can't auto create" +
parentQueueName);
}
if (parentQueue != null &&
conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
} else {
// Now the parentQueue will not be null later in autoCreateQueue.
// Because that: if parentQueue null, apc.isCreateParentQueue() is
enabled.
// Otherwise the parentQueue will not be null.
// Just check the leafQueue will handle special case.
if (!apc.isCreateLeafQueue()) {
if (leafQueue != null && leafQueue instanceof LeafQueue) {
return (LeafQueue) leafQueue;
} else {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ "which auto create leaf not enabed. " +
"Meanwhile it does not exist or not a leaf queue.");
}
}
// Now can create parent with leaf.
// Other check will be in autoCreateQueue.
return autoQueueHandler.autoCreateQueue(apc);
}
}
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping does not specify"
+ " which parent queue it needs to be created under.");
}
}
{code}
I handle all the case, may some corner case i don't mind it.
I also fill the test cases in
{code:java}
@Test
public void testAutoQueueCreationOnAppSubmission() throws Exception {
startScheduler();
createBasicQueueStructureAndValidate();
// Parent exists, allow create leaf will pass
submitApp(cs, USER0, USER0, "root.e-auto", true, false);
AbstractCSQueue e = (AbstractCSQueue) cs.getQueue("root.e-auto");
Assert.assertNotNull(e);
Assert.assertTrue(e.isDynamicQueue());
AbstractCSQueue user0 = (AbstractCSQueue) cs.getQueue(
"root.e-auto." + USER0);
Assert.assertNotNull(user0);
Assert.assertTrue(user0.isDynamicQueue());
// Parent not exists (null), allow create parent and allow create leaf will
pass
submitApp(cs, USER1, USER1, "root.e-auto2", true, true);
AbstractCSQueue e2 = (AbstractCSQueue) cs.getQueue("root.e-auto2");
Assert.assertNotNull(e2);
Assert.assertTrue(e2.isDynamicQueue());
AbstractCSQueue user1 = (AbstractCSQueue) cs.getQueue(
"root.e-auto2." + USER1);
Assert.assertNotNull(user0);
Assert.assertTrue(user1.isDynamicQueue());
// Parent exists, don't allow create leaf, but leaf queue exists will pass
submitApp(cs, USER0, USER0, "root.e-auto", false, false);
Assert.assertNotNull(e);
Assert.assertTrue(e.isDynamicQueue());
Assert.assertNotNull(user0);
Assert.assertTrue(user0.isDynamicQueue());
// Parent not exists, don't allow create parent will not pass
submitApp(cs, USER2, USER2, "root.e-auto3", true, false);
// e3 will be null
AbstractCSQueue e3 = (AbstractCSQueue) cs.getQueue("root.e-auto3");
Assert.assertNull(e3);
AbstractCSQueue user3 = (AbstractCSQueue) cs.getQueue(
"root.e-auto3." + USER1);
// user3 will be null
Assert.assertNull(user3);
}
{code}
Add a submit with auto create leaf/parent flag in
TestCapacitySchedulerAutoCreatedQueueBase.
Reg. *2) How we deal with the queue's auto-queue-creation configuration flag?*
*I have changed the related to queue-path.auto-queue-creation-v2.enabled.*
*If you any other advice for commit?*
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506.001.patch, YARN-10506.002.patch,
> YARN-10506.003.patch, YARN-10506.004.patch, YARN-10506.005.patch,
> YARN-10506.006-combined.patch, YARN-10506.006.patch, YARN-10506.007.patch,
> YARN-10506.009.patch, YARN-10506.011.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
