[jira] [Commented] (YARN-10674) fs2cs: should support auto created queue deletion.

[Peter Bacsko (Jira)]

    [ 
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298112#comment-17298112
 ] 

Peter Bacsko commented on YARN-10674:
-------------------------------------

[~zhuqi] I have the following comments:

1. This change seems to always enable "RM monitors":
{noformat}
            // This should be always true to trigger dynamic queue auto deletion
            // when expired.
            yarnSiteConfig.setBoolean(
                YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
{noformat}
But I don't think this is necessary. We need to enable it in two cases: 
preemption is enabled OR we're in weight mode. We don't have auto-queue delete 
in percentage mode (fs2cs can still convert to percentages with a command line 
switch).
 So I suggest that you pass an extra boolean "usePercentages".

Invocation from {{FSConfigToCSConfigConverter}}:
{noformat}
    siteConverter.convertSiteProperties(inputYarnSiteConfig,
        convertedYarnSiteConfig, drfUsed,
        conversionOptions.isEnableAsyncScheduler(), usePercentages);  <-- last 
argument is new
{noformat}
Then in the site converter:
{noformat}
if (conf.getBoolean(FairSchedulerConfiguration.PREEMPTION,
        FairSchedulerConfiguration.DEFAULT_PREEMPTION)) {
      yarnSiteConfig.setBoolean(
          YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
      preemptionEnabled = true;
          ...
}

if (!usePercentages) {
    yarnSiteConfig.setBoolean(
        YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);           // 
setting it again is OK

    String policies =
        yarnSiteConfig.get(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES);
    if (policies == null) {
      policies = AutoCreatedQueueDeletionPolicy.
          class.getCanonicalName();
    } else {
      policies += "," + AutoCreatedQueueDeletionPolicy.
          class.getCanonicalName();
    }

    yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
        policies);

    // Set the expired for deletion interval to 10s, consistent with fs.
    yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
        AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
If I think about it, {{yarnSiteConfig}} is the output config. So this cannot 
happen:
{noformat}
    } else {
      policies += "," + AutoCreatedQueueDeletionPolicy.
          class.getCanonicalName();
    }
{noformat}
This {{Configuration}} object is created with no entries. The {{else}} branch 
will never be taken.

So it can be simplified to:
{noformat}
if (!usePercentages) {
    yarnSiteConfig.setBoolean(
        YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);

    String policy = AutoCreatedQueueDeletionPolicy.
          class.getCanonicalName();

    yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
        policy);

    // Set the expired for deletion interval to 10s, consistent with fs.
    yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
        AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
2. This also means two separate test cases:
 * When usePercentages = false, then {{RM_SCHEDULER_ENABLE_MONITORS}} and 
{{RM_SCHEDULER_MONITOR_POLICIES}} should be set (with preemption = false)
 * When usePercentages = true, then\{{RM_SCHEDULER_ENABLE_MONITORS}} and 
{{RM_SCHEDULER_MONITOR_POLICIES}} should NOT be set (with preemption = false)

I recommend the following naming:
 {{testRmMonitorsAndPoliciesSetWhenUsingWeights()}} - first scenario
 {{testRmMonitorsAndPoliciesSetWhenUsingPercentages()}} - second scenario

> fs2cs: should support auto created queue deletion.
> --------------------------------------------------
>
>                 Key: YARN-10674
>                 URL: https://issues.apache.org/jira/browse/YARN-10674
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Qi Zhu
>            Assignee: Qi Zhu
>            Priority: Major
>              Labels: fs2cs
>         Attachments: YARN-10674.001.patch, YARN-10674.002.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
>   queueMgr.removeEmptyDynamicQueues();
>   queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
>   try {
>     synchronized (this) {
>       reloadListener.onCheck();
>     }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org