[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279769#comment-17279769
]
Andras Gyori commented on YARN-10532:
-------------------------------------
Hi [~zhuqi] and thanks for addressing the feedbacks. I have tried this feature
on a cluster and did not work flawlessly. My observations are:
* The monitoringInterval is in milisecond, but the configuration value is in
second. This is a critical discrepancy and resulted in RM spamming. This should
be mitigated by:
{code:java}
monitoringInterval =
csConfig.getLong(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME,
CapacitySchedulerConfiguration.
DEFAULT_AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME) * 1000;
{code}
* The queue-expiration-time is not prefixed, thus making the whole property
named auto-queue-creation-v2.queue-expiration-time instead of
yarn.scheduler.capacity.auto-queue-creation-v2.queue-expiration-time.
* There should be a separation of testing, and I would advise to change the
existing auto deletion tests in TestCapacitySchedulerNewQueueAutoCreation not
to invoke the policy itself, but rely only on CapacityScheduler like the
following example:
{code:java}
@Test
public void testCapacitySchedulerAutoQueueDeletion() throws Exception {
startScheduler();
csConf.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
csConf.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
AutoCreatedQueueDeletionPolicy.class.getCanonicalName());
csConf.setAutoExpiredDeletionTime(2);
cs.reinitialize(csConf, mockRM.getRMContext());
Set<String> policies = new HashSet<>();
policies.add(
AutoCreatedQueueDeletionPolicy.class.getCanonicalName());
Assert.assertTrue(
"No AutoCreatedQueueDeletionPolicy is present in running monitors",
cs.getSchedulingMonitorManager().isSameConfiguredPolicies(policies));
ApplicationAttemptId app = submitApp(cs, USER0,
"a2-auto", "root.a.a1-auto");
Thread.sleep(1000);
AbstractCSQueue a2 = (AbstractCSQueue) cs.getQueue(
"root.a.a1-auto.a2-auto");
Assert.assertNotNull("a2 is not present", a2);
Assert.assertTrue("a2 is not a dynamic queue",
a2.isDynamicQueue());
finishApp(app);
Thread.sleep(2000);
a2 = (AbstractCSQueue) cs.getQueue("root.a.a1-auto.a2-auto");
Assert.assertNull("a2 is not deleted", a2);
Thread.sleep(2000);
AbstractCSQueue a1 = (AbstractCSQueue) cs.getQueue("root.a.a1-auto");
Assert.assertNull("a1 is not deleted", a1);
csConf.setAutoExpiredDeletionTime(2);
cs.reinitialize(csConf, mockRM.getRMContext());
}
{code}
* The policy itself should only be invoked in its own test class, as you have
already done it.
* As I see it, the general way of disabling time sensitive values is to set it
to -1. This way, you would not need to handle two different property.
> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is
> not being used
> --------------------------------------------------------------------------------------------
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Qi Zhu
> Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch,
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch,
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch,
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch,
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch,
> YARN-10532.015.patch, YARN-10532.016.patch
>
>
> It's better if we can delete auto-created queues when they are not in use for
> a period of time (like 5 mins). It will be helpful when we have a large
> number of auto-created queues (e.g. from 500 users), but only a small subset
> of queues are actively used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
