[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304228#comment-17304228
]
Qi Zhu edited comment on YARN-10674 at 3/22/21, 3:45 AM:
-
[~gandras]
Now i understand you, we can just use the code:
{code:java}
checkDisablePreemption(preemptionMode, !cliParser.hasOption(CliOption.
DISABLE_PREEMPTION.shortSwitch));
{code}
{code:java}
private static void checkDisablePreemption(FSConfigToCSConfigConverterParams.
PreemptionMode preemptionMode, boolean enabled) {
if (preemptionMode == null && !enabled) {
throw new PreconditionException(
"Specified disable-preemption mode is illegal, " +
" use nopolicy or observeonly.");
}
}
{code}
PreemptionMode.ENABLED is not necessary, i updated in latest patch.
I am glad that i make sense now.
[~pbacsko]
If you any other advice? :D
Thanks.
was (Author: zhuqi):
[~gandras]
Now i understand you, we can just use the code:
{code:java}
checkDisablePreemption(preemptionMode, !cliParser.hasOption(CliOption.
DISABLE_PREEMPTION.shortSwitch));
{code}
{code:java}
private static void checkDisablePreemption(FSConfigToCSConfigConverterParams.
PreemptionMode preemptionMode, boolean enabled) {
if (preemptionMode == null && !enabled) {
throw new PreconditionException(
"Specified disable-preemption mode is illegal, " +
" use nopolicy or observeonly.");
}
}
{code}
PreemptionMode.ENABLED is not necessary, i updated in latest patch.
I am glad that i make sense now.
[~pbacsko]
If you any other advice?
Thanks.
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch,
> YARN-10674.015.patch, YARN-10674.016.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304142#comment-17304142
]
Qi Zhu edited comment on YARN-10674 at 3/18/21, 1:29 PM:
-
Thanks [~gandras] for reply.
If we don't have PreemptionMode.ENABLED, we can use the has option to know if
this is enabled and passed to PreemptionMode enabled field.
{code:java}
public static PreemptionMode fromString(String cliOption,
boolean enabled) {
if (enabled) {
return PreemptionMode.ENABLED;
} else {
if (StringUtils.isEmpty(cliOption)) {
return PreemptionMode.NO_POLICY;
} else {
if (cliOption.trim().
equals(PreemptionMode.OBSERVE_ONLY.getCliOption())) {
return PreemptionMode.OBSERVE_ONLY;
} else if (cliOption.trim().
equals(PreemptionMode.NO_POLICY.getCliOption())) {
return PreemptionMode.NO_POLICY;
} else {
return null;
}
}
}
}
{code}
If return null:
{code:java}
private static void checkDisablePreemption(FSConfigToCSConfigConverterParams.
PreemptionMode preemptionMode) {
if (preemptionMode == null) {
throw new PreconditionException(
"Specified disable-preemption mode is illegal, " +
" use nopolicy or observeonly.");
}
}
{code}
But fromString should return a value to make it used later, if it will return
null , it will confused with the case that we disabled but print not nopolicy
or observeonly. I think the flag will make this clear that we have four case
return value:
# null mean that we use illegal value
# PreemptionMode.ENABLED
# PreemptionMode.OBSERVE_ONLY
# PreemptionMode.NO_POLICY
What's your opinion about this?
was (Author: zhuqi):
Thanks [~gandras] for reply.
If we don't have PreemptionMode.ENABLED, we can use the has option to know if
this is enabled and passed to PreemptionMode enabled field.
{code:java}
public static PreemptionMode fromString(String cliOption,
boolean enabled) {
if (enabled) {
return PreemptionMode.ENABLED;
} else {
if (StringUtils.isEmpty(cliOption)) {
return PreemptionMode.NO_POLICY;
} else {
if (cliOption.trim().
equals(PreemptionMode.OBSERVE_ONLY.getCliOption())) {
return PreemptionMode.OBSERVE_ONLY;
} else if (cliOption.trim().
equals(PreemptionMode.NO_POLICY.getCliOption())) {
return PreemptionMode.NO_POLICY;
} else {
return null;
}
}
}
}
{code}
But fromString should return a value to make it used later, if it will return
null , it will confused with the case that we disabled but print not nopolicy
or observeonly. I think the flag will make this clear that we have four case
return value:
# null mean that we use illegal value
# PreemptionMode.ENABLED
# PreemptionMode.OBSERVE_ONLY
# PreemptionMode.NO_POLICY
What's your opinion about this?
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch,
> YARN-10674.015.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304142#comment-17304142
]
Qi Zhu edited comment on YARN-10674 at 3/18/21, 1:26 PM:
-
Thanks [~gandras] for reply.
If we don't have PreemptionMode.ENABLED, we can use the has option to know if
this is enabled and passed to PreemptionMode enabled field.
{code:java}
public static PreemptionMode fromString(String cliOption,
boolean enabled) {
if (enabled) {
return PreemptionMode.ENABLED;
} else {
if (StringUtils.isEmpty(cliOption)) {
return PreemptionMode.NO_POLICY;
} else {
if (cliOption.trim().
equals(PreemptionMode.OBSERVE_ONLY.getCliOption())) {
return PreemptionMode.OBSERVE_ONLY;
} else if (cliOption.trim().
equals(PreemptionMode.NO_POLICY.getCliOption())) {
return PreemptionMode.NO_POLICY;
} else {
return null;
}
}
}
}
{code}
But fromString should return a value to make it used later, if it will return
null , it will confused with the case that we disabled but print not nopolicy
or observeonly. I think the flag will make this clear that we have four case
return value:
# null mean that we use illegal value
# PreemptionMode.ENABLED
# PreemptionMode.OBSERVE_ONLY
# PreemptionMode.NO_POLICY
What's your opinion about this?
was (Author: zhuqi):
Thanks [~gandras] for reply.
If we don't have PreemptionMode.ENABLED, we can use the has option to know if
this is enabled and passed to PreemptionMode enabled field.
{code:java}
public static PreemptionMode fromString(String cliOption,
boolean enabled) {
if (enabled) {
return PreemptionMode.ENABLED;
} else {
if (StringUtils.isEmpty(cliOption)) {
return PreemptionMode.NO_POLICY;
} else {
if (cliOption.trim().
equals(PreemptionMode.OBSERVE_ONLY.getCliOption())) {
return PreemptionMode.OBSERVE_ONLY;
} else if (cliOption.trim().
equals(PreemptionMode.NO_POLICY.getCliOption())) {
return PreemptionMode.NO_POLICY;
} else {
return null;
}
}
}
}
{code}
But fromString should return a value to make it used later, if it will return
null , it will confused with the case that we disabled but print not nopolicy
or observeonly. I think the flag will make this clear.
What's your opinion about this?
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch,
> YARN-10674.015.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304142#comment-17304142
]
Qi Zhu edited comment on YARN-10674 at 3/18/21, 1:24 PM:
-
Thanks [~gandras] for reply.
If we don't have PreemptionMode.ENABLED, we can use the has option to know if
this is enabled and passed to PreemptionMode enabled field.
{code:java}
public static PreemptionMode fromString(String cliOption,
boolean enabled) {
if (enabled) {
return PreemptionMode.ENABLED;
} else {
if (StringUtils.isEmpty(cliOption)) {
return PreemptionMode.NO_POLICY;
} else {
if (cliOption.trim().
equals(PreemptionMode.OBSERVE_ONLY.getCliOption())) {
return PreemptionMode.OBSERVE_ONLY;
} else if (cliOption.trim().
equals(PreemptionMode.NO_POLICY.getCliOption())) {
return PreemptionMode.NO_POLICY;
} else {
return null;
}
}
}
}
{code}
But fromString should return a value to make it used later, if it will return
null , it will confused with the case that we disabled but print not nopolicy
or observeonly. I think the flag will make this clear.
What's your opinion about this?
was (Author: zhuqi):
Thanks [~gandras] for reply.
If we don't have PreemptionMode.ENABLED, we can use the has option to know if
this is enabled and passed to PreemptionMode enabled field.
But fromString should return a value to make it used later, if it will return
null , it will confused with the case that we disabled but print not nopolicy
or observeonly. I think the flag will make this clear.
What's your opinion about this?
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch,
> YARN-10674.015.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17303804#comment-17303804
]
Qi Zhu edited comment on YARN-10674 at 3/18/21, 2:20 AM:
-
[~pbacsko] [~gandras]
Fixed the checkstyle and change to assertTrue in
testSiteDisabledPreemptionWithObserveOnlyConversion in latest patch.
Thanks.:D
was (Author: zhuqi):
[~pbacsko] [~gandras]
Fixed the checkstyle and change to assertTrue in
testSiteDisabledPreemptionWithObserveOnlyConversion.
Thanks.:D
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300319#comment-17300319
]
Peter Bacsko edited comment on YARN-10674 at 3/12/21, 1:34 PM:
---
[~zhuqi] I didn't have too much time to deeply review the patch, but your
change ignores the "observeonly" setting. So, if I use "\-\-disablepreemption
observeonly", nothing happens. Could you insert this to
{{FSConfigToCSConfigConverter}}? I believe that is the best place for it.
was (Author: pbacsko):
[~zhuqi] I didn't have too much time to deeply review the patch, but your
change ignore the "observeonly" setting. So, if I use "\-\-disablepreemption
observeonly", nothing happens. Could you insert this to
{{FSConfigToCSConfigConverter}}? I believe that is the best place for it.
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299602#comment-17299602
]
Peter Bacsko edited comment on YARN-10674 at 3/11/21, 3:25 PM:
---
Ok, I did some research, I think we have 3 options to completely disable
preemption:
1) Set disable_preemption to "root", which will propagate down to other queues.
2) Remove "ProportionalCapacityPreemptionPolicy" from the list of policies.
3) Enable "observe_only" property.
I think #1 is not really good, because it relies on a side-effect (propagation
of a setting). The intention is not clear.
#2 is perfectly acceptable and this goes to {{yarn-site.xml}} so it should be
in {{FSYarnSiteConverter}}.
#3 is also OK, but that goes to {{capacity-scheduler.xml}} and NOT to
{{yarn-site.xml}}, I just verified it. So this should be placed somewhere else.
So we can do:
1) Vote for what's best
2) Introduce a command line switch like "-dp" "\-\-disable-preemption" with
values like "nopolicy" or "observeonly" and we pick a default value, eg.
"nopolicy". So we can do something like:
{noformat}
yarn fs2cs --disable-preemption observeonly --yarnsiteconfig
/path/to/yarn-site.xml
{noformat}
[~gandras] [~zhuqi] what do you think?
was (Author: pbacsko):
Ok, I did some research, I think we 3 options to completely disable preemption:
1) Set disable_preemption to "root", which will propagate down to other queues.
2) Remove "ProportionalCapacityPreemptionPolicy" from the list of policies.
3) Enable "observe_only" property.
I think #1 is not really good, because it relies on a side-effect (propagation
of a setting). The intention is not clear.
#2 is perfectly acceptable and this goes to {{yarn-site.xml}} so it should be
in {{FSYarnSiteConverter}}.
#3 is also OK, but that goes to {{capacity-scheduler.xml}} and NOT to
{{yarn-site.xml}}, I just verified it. So this should be placed somewhere else.
So we can do:
1) Vote for what's best
2) Introduce a command line switch like "-dp" "\-\-disable-preemption" with
values like "nopolicy" or "observeonly" and we pick a default value, eg.
"nopolicy". So we can do something like:
{noformat}
yarn fs2cs --disable-preemption observeonly --yarnsiteconfig
/path/to/yarn-site.xml
{noformat}
[~gandras] [~zhuqi] what do you think?
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299602#comment-17299602
]
Peter Bacsko edited comment on YARN-10674 at 3/11/21, 2:43 PM:
---
Ok, I did some research, I think we 3 options to completely disable preemption:
1) Set disable_preemption to "root", which will propagate down to other queues.
2) Remove "ProportionalCapacityPreemptionPolicy" from the list of policies.
3) Enable "observe_only" property.
I think #1 is not really good, because it relies on a side-effect (propagation
of a setting). The intention is not clear.
#2 is perfectly acceptable and this goes to {{yarn-site.xml}} so it should be
in {{FSYarnSiteConverter}}.
#3 is also OK, but that goes to {{capacity-scheduler.xml}} and NOT to
{{yarn-site.xml}}, I just verified it. So this should be placed somewhere else.
So we can do:
1) Vote for what's best
2) Introduce a command line switch like "-dp" "\-\-disable-preemption" with
values like "nopolicy" or "observeonly" and we pick a default value, eg.
"nopolicy". So we can do something like:
{noformat}
yarn fs2cs --disable-preemption observeonly --yarnsiteconfig
/path/to/yarn-site.xml
{noformat}
[~gandras] [~zhuqi] what do you think?
was (Author: pbacsko):
Ok, I did some research, I think we 3 options to completely disable preemption:
1) Set disable_preemption to "root", which will propagate down to other queues.
2) Remove "ProportionalCapacityPreemptionPolicy" from the list of policies.
3) Enable "observe_only" property.
I think #1 is not really good, because it relies on a side-effect (propagation
of a setting). The intention is not clear.
#2 is perfectly acceptable and this goes to {{yarn-site.xml}} so it should be
in {{FSYarnSiteConverter}}.
#3 is also OK, but that goes to {{capacity-scheduler.xml}} and NOT in
{{yarn-site.xml}}, I just verified it. So this should be placed somewhere else.
So we can do:
1) Vote for what's best
2) Introduce a command line switch like "-dp" "\-\-disable-preemption" with
values like "nopolicy" or "observeonly" and we pick a default value, eg.
"nopolicy". So we can do something like:
{noformat}
yarn fs2cs --disable-preemption observeonly --yarnsiteconfig
/path/to/yarn-site.xml
{noformat}
[~gandras] [~zhuqi] what do you think?
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299461#comment-17299461
]
Qi Zhu edited comment on YARN-10674 at 3/11/21, 10:07 AM:
--
[~pbacsko] [~gandras]
Ah, Now it makes sense to me. The preemption also have a policy default, we
just need to remove it, when we disable the preemption?
"The following configuration parameters can be configured in yarn-site.xml to
control the preemption of containers when
{{ProportionalCapacityPreemptionPolicy}} class is configured for
{{yarn.resourcemanager.scheduler.monitor.policies"}}
Am i right? :D
Thanks.
was (Author: zhuqi):
[~pbacsko] [~gandras]
Ah, Now it makes sense to me. The preemption also have a policy default, we
just need to remove it, when we disable the preemption?
Am i right? :D
Thanks.
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298776#comment-17298776
]
Peter Bacsko edited comment on YARN-10674 at 3/10/21, 12:03 PM:
[~zhuqi] thanks for the patch. I found a new property which is probably good
for us if preemption is completely disabled on the FS side. I have to check if
it is really acceptable.
was (Author: pbacsko):
[~zhuqi] thanks for the patch. I found a new property which is probably good
for us if preemption is completely disabled on the FS side. I have to check if
it is good for us.
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298587#comment-17298587
]
Qi Zhu edited comment on YARN-10674 at 3/10/21, 7:36 AM:
-
Thanks a lot [~pbacsko] for reply.
Your advice make sense to me , and it's the only solution to solve it now,
because the preemption is related to the enable option.
“However, you already added a policy in YARN-10623, so looks like this property
always has to be enabled in weight mode.”
Actually this is used in all mode for auto refresh.
I will update it in next patch.
Thanks.
was (Author: zhuqi):
Thanks a lot [~pbacsko] for reply.
Your advice make sense to me , and it's the only solution to solve it now,
because the preemption is related to the enable option.
I will update it in next patch.
Thanks.
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298112#comment-17298112
]
Peter Bacsko edited comment on YARN-10674 at 3/9/21, 3:23 PM:
--
[~zhuqi] I have the following comments:
1. This change seems to always enable "RM monitors":
{noformat}
// This should be always true to trigger dynamic queue auto deletion
// when expired.
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
{noformat}
But I don't think this is necessary. We need to enable it in two cases:
preemption is enabled OR we're in weight mode. We don't have auto-queue delete
in percentage mode (fs2cs can still convert to percentages with a command line
switch).
So I suggest that you pass an extra boolean "usePercentages".
Invocation from {{FSConfigToCSConfigConverter}}:
{noformat}
siteConverter.convertSiteProperties(inputYarnSiteConfig,
convertedYarnSiteConfig, drfUsed,
conversionOptions.isEnableAsyncScheduler(), usePercentages); <-- last
argument is new
{noformat}
Then in the site converter:
{noformat}
if (conf.getBoolean(FairSchedulerConfiguration.PREEMPTION,
FairSchedulerConfiguration.DEFAULT_PREEMPTION)) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
preemptionEnabled = true;
...
}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true); //
setting it again is OK
String policies =
yarnSiteConfig.get(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES);
if (policies == null) {
policies = AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
} else {
policies += "," + AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
}
yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
policies);
// Set the expired for deletion interval to 10s, consistent with fs.
yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
If I think about it, {{yarnSiteConfig}} is the output config. So this cannot
happen:
{noformat}
} else {
policies += "," + AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
}
{noformat}
This {{Configuration}} object is created with no entries. The {{else}} branch
will never be taken.
So it can be simplified to:
{noformat}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
String policy = AutoCreatedQueueDeletionPolicy.
class.getCanonicalName();
yarnSiteConfig.set(YarnConfiguration.RM_SCHEDULER_MONITOR_POLICIES,
policy);
// Set the expired for deletion interval to 10s, consistent with fs.
yarnSiteConfig.setInt(CapacitySchedulerConfiguration.
AUTO_CREATE_CHILD_QUEUE_EXPIRED_TIME, 10);
}
{noformat}
2. This also means two separate test cases:
* When usePercentages = false, then {{RM_SCHEDULER_ENABLE_MONITORS}} and
{{RM_SCHEDULER_MONITOR_POLICIES}} should be set (with preemption = false)
* When usePercentages = true, then {{RM_SCHEDULER_ENABLE_MONITORS}} and
{{RM_SCHEDULER_MONITOR_POLICIES}} should NOT be set (with preemption = false)
I recommend the following naming:
{{testRmMonitorsAndPoliciesSetWhenUsingWeights()}} - first scenario
{{testRmMonitorsAndPoliciesSetWhenUsingPercentages()}} - second scenario
was (Author: pbacsko):
[~zhuqi] I have the following comments:
1. This change seems to always enable "RM monitors":
{noformat}
// This should be always true to trigger dynamic queue auto deletion
// when expired.
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
{noformat}
But I don't think this is necessary. We need to enable it in two cases:
preemption is enabled OR we're in weight mode. We don't have auto-queue delete
in percentage mode (fs2cs can still convert to percentages with a command line
switch).
So I suggest that you pass an extra boolean "usePercentages".
Invocation from {{FSConfigToCSConfigConverter}}:
{noformat}
siteConverter.convertSiteProperties(inputYarnSiteConfig,
convertedYarnSiteConfig, drfUsed,
conversionOptions.isEnableAsyncScheduler(), usePercentages); <-- last
argument is new
{noformat}
Then in the site converter:
{noformat}
if (conf.getBoolean(FairSchedulerConfiguration.PREEMPTION,
FairSchedulerConfiguration.DEFAULT_PREEMPTION)) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MONITORS, true);
preemptionEnabled = true;
...
}
if (!usePercentages) {
yarnSiteConfig.setBoolean(
YarnConfiguration.RM_SCHEDULER_ENABLE_MO
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298076#comment-17298076
]
Qi Zhu edited comment on YARN-10674 at 3/9/21, 2:10 PM:
Thanks [~pbacsko] for review.
It not depends on YARN-10682 , "," already supported now, YARN-10682 fixed the
"," with space error , just like YARN-10497 we should fix, such as the case:
"a, b" is not supported now, but "a,b" is supported.:D
I updated the description for YARN-10682.
was (Author: zhuqi):
Thanks [~pbacsko] for review.
It not depends on YARN-10682 , "," already supported now, YARN-10682 fixed the
"," with space error such as the case:
"a, b" is not supported now, but "a,b" is supported.:D
I updated the description for YARN-10682.
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10674) fs2cs: should support auto created queue deletion.
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298076#comment-17298076
]
Qi Zhu edited comment on YARN-10674 at 3/9/21, 1:54 PM:
Thanks [~pbacsko] for review.
It not depends on YARN-10682 , "," already supported now, YARN-10682 fixed the
"," with space error such as the case:
"a, b" is not supported now, but "a,b" is supported.:D
I updated the description for YARN-10682.
was (Author: zhuqi):
Thanks [~pbacsko] for review.
It not depends on YARN-10682 , "," already supported now, YARN-10682 fixed the
"," with space error such as the case:
"a, b" is not supported now, but "a,b" is supported.:D
> fs2cs: should support auto created queue deletion.
> --
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
