[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298112#comment-17298112
]
Peter Bacsko edited comment on YARN-10674 at 3/9/21, 3:23 PM:
--------------------------------------------------------------
[~zhuqi] I have the following comments:
1. This change seems to always enable "RM monitors":
{noformat}
// This should be always true to trigger dynamic queue auto deletion
// when expired.
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
{noformat}
But I don't think this is necessary. We need to enable it in two cases:
preemption is enabled OR we're in weight mode. We don't have auto-queue delete
in percentage mode (fs2cs can still convert to percentages with a command line
switch).
So I suggest that you pass an extra boolean "usePercentages".
Invocation from {{FSConfigToCSConfigConverter}}:
{noformat}
siteConverter.convertSiteProperties(inputYarnSiteConfig,
convertedYarnSiteConfig, drfUsed,
conversionOptions.isEnableAsyncScheduler(), usePercentages); <-- last
argument is new
{noformat}
Then in the site converter:
{noformat}
if (conf.getBoolean(FairSchedulerConfiguration.PREEMPTION,
FairSchedulerConfiguration.DEFAULT_PREEMPTION)) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
preemptionEnabled = true;
...
}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true); //
setting it again is OK
String policies =
yarnSiteConfig.get(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES);
if (policies == null) {
policies = AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
} else {
policies += "," + AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
}
yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
policies);
// Set the expired for deletion interval to 10s, consistent with fs.
yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
If I think about it, {{yarnSiteConfig}} is the output config. So this cannot
happen:
{noformat}
} else {
policies += "," + AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
}
{noformat}
This {{Configuration}} object is created with no entries. The {{else}} branch
will never be taken.
So it can be simplified to:
{noformat}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
String policy = AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
policy);
// Set the expired for deletion interval to 10s, consistent with fs.
yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
2. This also means two separate test cases:
* When usePercentages = false, then {{RM_SCHEDULER_ENABLE_MONITORS}} and
{{RM_SCHEDULER_MONITOR_POLICIES}} should be set (with preemption = false)
* When usePercentages = true, then {{RM_SCHEDULER_ENABLE_MONITORS}} and
{{RM_SCHEDULER_MONITOR_POLICIES}} should NOT be set (with preemption = false)
I recommend the following naming:
{{testRmMonitorsAndPoliciesSetWhenUsingWeights()}} - first scenario
{{testRmMonitorsAndPoliciesSetWhenUsingPercentages()}} - second scenario
was (Author: pbacsko):
[~zhuqi] I have the following comments:
1. This change seems to always enable "RM monitors":
{noformat}
// This should be always true to trigger dynamic queue auto deletion
// when expired.
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
{noformat}
But I don't think this is necessary. We need to enable it in two cases:
preemption is enabled OR we're in weight mode. We don't have auto-queue delete
in percentage mode (fs2cs can still convert to percentages with a command line
switch).
So I suggest that you pass an extra boolean "usePercentages".
Invocation from {{FSConfigToCSConfigConverter}}:
{noformat}
siteConverter.convertSiteProperties(inputYarnSiteConfig,
convertedYarnSiteConfig, drfUsed,
conversionOptions.isEnableAsyncScheduler(), usePercentages); <-- last
argument is new
{noformat}
Then in the site converter:
{noformat}
if (conf.getBoolean(FairSchedulerConfiguration.PREEMPTION,
FairSchedulerConfiguration.DEFAULT_PREEMPTION)) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
preemptionEnabled = true;
...
}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true); //
setting it again is OK
String policies =
yarnSiteConfig.get(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES);
if (policies == null) {
policies = AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
} else {
policies += "," + AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
}
yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
policies);
// Set the expired for deletion interval to 10s, consistent with fs.
yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
If I think about it, {{yarnSiteConfig}} is the output config. So this cannot
happen:
{noformat}
} else {
policies += "," + AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
}
{noformat}
This {{Configuration}} object is created with no entries. The {{else}} branch
will never be taken.
So it can be simplified to:
{noformat}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
String policy = AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
policy);
// Set the expired for deletion interval to 10s, consistent with fs.
yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
2. This also means two separate test cases:
* When usePercentages = false, then {{RM_SCHEDULER_ENABLE_MONITORS}} and
{{RM_SCHEDULER_MONITOR_POLICIES}} should be set (with preemption = false)
* When usePercentages = true, then\{{RM_SCHEDULER_ENABLE_MONITORS}} and
{{RM_SCHEDULER_MONITOR_POLICIES}} should NOT be set (with preemption = false)
I recommend the following naming:
{{testRmMonitorsAndPoliciesSetWhenUsingWeights()}} - first scenario
{{testRmMonitorsAndPoliciesSetWhenUsingPercentages()}} - second scenario
> fs2cs: should support auto created queue deletion.
> --------------------------------------------------
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Qi Zhu
> Assignee: Qi Zhu
> Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
