[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[ https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692031#comment-17692031 ] ASF GitHub Bot commented on YARN-11408: --- brumi1024 merged PR #5278: URL: https://github.com/apache/hadoop/pull/5278 > Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor > method > > > Key: YARN-11408 > URL: https://issues.apache.org/jira/browse/YARN-11408 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Susheel Gupta >Assignee: Susheel Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > It is required to add user-limit-factor to -1 only for those queues which are > leafqueue and auto-queue-creation is disabled. > Follow-up of YARN-11393 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[ https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692030#comment-17692030 ] ASF GitHub Bot commented on YARN-11408: --- brumi1024 commented on PR #5278: URL: https://github.com/apache/hadoop/pull/5278#issuecomment-1439635512 Thanks @susheel-gupta, LGTM. Merged to trunk. > Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor > method > > > Key: YARN-11408 > URL: https://issues.apache.org/jira/browse/YARN-11408 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Susheel Gupta >Assignee: Susheel Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > It is required to add user-limit-factor to -1 only for those queues which are > leafqueue and auto-queue-creation is disabled. > Follow-up of YARN-11393 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689360#comment-17689360
]
ASF GitHub Bot commented on YARN-11408:
---
hadoop-yetus commented on PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#issuecomment-1432097035
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 35s | | trunk passed |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| -1 :x: | javadoc | 0m 54s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/3/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 1m 1s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| -1 :x: | javadoc | 0m 40s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/3/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 27s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 220m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5278 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c73991bf875b 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2d132a48361bd247f118463ba1a8feb1995edd55 |
| Default Java | Private
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688524#comment-17688524
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1105904503
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
Update: checked the code, because I was under the impression we're
automatically adding the auto-create in percentage conversion as well, however
it's not the case:
[https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/had[…]ger/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java](https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java#L192)
Sorry for that, please disregard my comment, and feel free to remove the
original check for this.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679731#comment-17679731
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1083821014
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
Yes exactly. The basic purpose of a test like this would be to check if the
auto-create-child-queue.enabled is true the user-limit-factor = -1 isn't added.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679730#comment-17679730
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1083821014
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
Yes please. The basic purpose of a test like this would be to check if the
auto-create-child-queue.enabled is true the user-limit-factor = -1 isn't added.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679024#comment-17679024
]
ASF GitHub Bot commented on YARN-11408:
---
susheel-gupta commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1071075406
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
In class TestFSQueueConverter, there is method
testQueueWithNoAutoCreateChildQueue which checks assert
.auto-create-child-queue.enabled to null but according to above comment I need
to add a property of auto-create-child-queue.enabled to true.
So do I need to add a another set of queues where
auto-create-child-queue.enabled is true.
```
testQueueWithNoAutoCreateChildQueue() {
converter = builder
.withCapacitySchedulerConfig(csConfig)
.build();
converter.convertQueueHierarchy(rootQueue);
assertNoValueForQueues(ALL_QUEUES, ".auto-create-child-queue.enabled",
csConfig);
}
```
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677328#comment-17677328
]
ASF GitHub Bot commented on YARN-11408:
---
susheel-gupta commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1071075406
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
In class TestFSQueueConverter, there is method
testQueueWithNoAutoCreateChildQueue which checks assert
.auto-create-child-queue.enabled to null but according to above comment I need
to add a property of auto-create-child-queue.enabled to true.
So do I need to add a another set of queues where
auto-create-child-queue.enabled is true or is this check of
checkAutoCreateChildQueueDisabled really required?.
```
testQueueWithNoAutoCreateChildQueue() {
converter = builder
.withCapacitySchedulerConfig(csConfig)
.build();
converter.convertQueueHierarchy(rootQueue);
assertNoValueForQueues(ALL_QUEUES, ".auto-create-child-queue.enabled",
csConfig);
}
```
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677304#comment-17677304
]
ASF GitHub Bot commented on YARN-11408:
---
susheel-gupta commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1071075406
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
In class TestFSQueueConverter, there is method
testQueueWithNoAutoCreateChildQueue which checks wheather
.auto-create-child-queue.enabled is null or not but according to above comment
I need to add a property of auto-create-child-queue.enabled is true.
So do I need to add a another set of queues where
auto-create-child-queue.enabled is true or is this check really required?.
```
testQueueWithNoAutoCreateChildQueue() {
converter = builder
.withCapacitySchedulerConfig(csConfig)
.build();
converter.convertQueueHierarchy(rootQueue);
assertNoValueForQueues(ALL_QUEUES, ".auto-create-child-queue.enabled",
csConfig);
}
```
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17672298#comment-17672298
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1066823284
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -194,6 +199,10 @@ public void testDefaultUserLimitFactor() throws Exception {
assertNull("root.users user-limit-factor should be null",
conf.get(PREFIX + "root.users." + USER_LIMIT_FACTOR));
+assertEquals("root.users auto-queue-creation-v2.enabled", "true",
+conf.get(PREFIX + "root.users.auto-queue-creation-v2.enabled"));
+assertNull( "root.users auto-create-child-queue.enabled should be null",
+conf.get(PREFIX + "root.users.auto-create-child-queue.enabled"));
Review Comment:
Can you please add one testcase for checking the user-limit-factor in a
conversion where the original auto-create-child-queue.enabled is true? Because
currently it's not validated, so the test will pass if that check is removed.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17672295#comment-17672295
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1066819718
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java:
##
@@ -309,6 +314,18 @@ private void checkMaxChildCapacitySetting(FSQueue queue) {
}
}
+ private boolean checkAutoQueueCreationV2Disabled(String queueName) {
+return !capacitySchedulerConfig.getBoolean(
+PREFIX + queueName + DOT + AUTO_QUEUE_CREATION_V2_ENABLED,
+DEFAULT_AUTO_QUEUE_CREATION_ENABLED);
+ }
+
+ private boolean checkAutoCreateChildQueueDisabled(String queueName) {
+return !capacitySchedulerConfig.getBoolean(
+PREFIX + queueName + DOT + AUTO_CREATE_CHILD_QUEUE_ENABLED,
+DEFAULT_AUTO_CREATE_CHILD_QUEUE_ENABLED);
+ }
Review Comment:
Ohh, the tricky thing is that capacitySchedulerConfig is in reality a
Configuration object, and not a CapacitySchedulerConfiguration object. This is
misleading (and it means that every convenience getter will be reimplemented
here), so I've created a follow-up:
https://issues.apache.org/jira/browse/YARN-11416. Resolving this conversation
for this patch.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656620#comment-17656620
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1065729095
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java:
##
@@ -309,6 +314,18 @@ private void checkMaxChildCapacitySetting(FSQueue queue) {
}
}
+ private boolean checkAutoQueueCreationV2Disabled(String queueName) {
+return !capacitySchedulerConfig.getBoolean(
+PREFIX + queueName + DOT + AUTO_QUEUE_CREATION_V2_ENABLED,
+DEFAULT_AUTO_QUEUE_CREATION_ENABLED);
+ }
+
+ private boolean checkAutoCreateChildQueueDisabled(String queueName) {
+return !capacitySchedulerConfig.getBoolean(
+PREFIX + queueName + DOT + AUTO_CREATE_CHILD_QUEUE_ENABLED,
+DEFAULT_AUTO_CREATE_CHILD_QUEUE_ENABLED);
+ }
Review Comment:
These methods already exist in CapacitySchedulerConfiguration, mentioned the
method names in my previous comments. Please use those.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656546#comment-17656546
]
ASF GitHub Bot commented on YARN-11408:
---
hadoop-yetus commented on PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#issuecomment-1376979850
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 2s | | trunk passed |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| -1 :x: | javadoc | 0m 54s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/2/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 38s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| -1 :x: | javadoc | 0m 38s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/2/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 103m 5s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 220m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5278 |
| Optional
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655484#comment-17655484
]
ASF GitHub Bot commented on YARN-11408:
---
brumi1024 commented on code in PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#discussion_r1063534266
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java:
##
@@ -309,6 +311,11 @@ private void checkMaxChildCapacitySetting(FSQueue queue) {
}
}
+ private boolean checkAutoQueueCreationV2Disabled(String queueName) {
+return !Objects.equals(capacitySchedulerConfig.get(
+PREFIX + queueName + DOT + AUTO_QUEUE_CREATION_V2_ENABLED), "true");
Review Comment:
This won't return true because Object.equals only returns true if both of
the compared objects refer to the same object, String.equals compares the
values themselves. But CapacitySchedulerConfiguration already has a helper
method which parses the value to a boolean, so I suggest using that. See
isAutoQueueCreationV2Enabled.
Also if percentage mode is used when converting it'll have the legacy Auto
Queue Creation property, which too has a helper method, called
isAutoCreateChildQueueEnabled. Both of these should be checked.
> Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor
> method
>
>
> Key: YARN-11408
> URL: https://issues.apache.org/jira/browse/YARN-11408
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Susheel Gupta
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> It is required to add user-limit-factor to -1 only for those queues which are
> leafqueue and auto-queue-creation is disabled.
> Follow-up of YARN-11393
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[
https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655141#comment-17655141
]
ASF GitHub Bot commented on YARN-11408:
---
hadoop-yetus commented on PR #5278:
URL: https://github.com/apache/hadoop/pull/5278#issuecomment-1372803205
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| -1 :x: | javadoc | 0m 54s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/1/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 56s | | the patch passed |
| -1 :x: | javadoc | 0m 41s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 38s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 210m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5278/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5278 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6cac3270c046 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-11408) Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor method
[ https://issues.apache.org/jira/browse/YARN-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655080#comment-17655080 ] ASF GitHub Bot commented on YARN-11408: --- susheel-gupta opened a new pull request, #5278: URL: https://github.com/apache/hadoop/pull/5278 Change-Id: If1e36c5969d270c1b81a4bbd2e883fa819c81f20 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add a check of autoQueueCreation is disabled for emitDefaultUserLimitFactor > method > > > Key: YARN-11408 > URL: https://issues.apache.org/jira/browse/YARN-11408 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Susheel Gupta >Assignee: Susheel Gupta >Priority: Major > Fix For: 3.4.0 > > > It is required to add user-limit-factor to -1 only for those queues which are > leafqueue and auto-queue-creation is disabled. > Follow-up of YARN-11393 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
