[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810871#comment-17810871
]
ASF GitHub Bot commented on YARN-11641:
---
brumi1024 merged PR #6435:
URL: https://github.com/apache/hadoop/pull/6435
> Can't update a queue hierarchy in absolute mode when the configured
> capacities are zero
> ---
>
> Key: YARN-11641
> URL: https://issues.apache.org/jira/browse/YARN-11641
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.5.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: hierarchy.png
>
>
> h2. Error symptoms
> It is not possible to modify a queue hierarchy in absolute mode when the
> parent or every child queue of the parent has 0 min resource configured.
> {noformat}
> 2024-01-05 15:38:59,016 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager:
> Initialized queue: root.a.c
> 2024-01-05 15:38:59,016 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : Parent=root.a: When absolute
> minResource is used, we must make sure both parent and child all use absolute
> minResource
> {noformat}
> h2. Reproduction
> capacity-scheduler.xml
> {code:xml}
>
>
>
> yarn.scheduler.capacity.root.queues
> default,a
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960, vcores=16]
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.a.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.queues
> b,c
>
>
> yarn.scheduler.capacity.root.a.b.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.b.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.c.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.c.maximum-capacity
> [memory=39936, vcores=15]
>
>
> {code}
> !hierarchy.png!
> updatequeue.xml
> {code:xml}
>
>
>
> root.a
>
>
> capacity
> [memory=1024,vcores=1]
>
>
> maximum-capacity
> [memory=39936,vcores=15]
>
>
>
>
> {code}
> {code}
> $ curl -X PUT -H 'Content-Type: application/xml' -d @updatequeue.xml
> http://localhost:8088/ws/v1/cluster/scheduler-conf\?user.name\=yarn
> Failed to re-init queues : Parent=root.a: When absolute minResource is used,
> we must make sure both parent and child all use absolute minResource
> {code}
> h2. Root cause
> setChildQueues is called during reinit, where:
> {code:java}
> void setChildQueues(Collection childQueues) throws IOException {
> writeLock.lock();
> try {
> boolean isLegacyQueueMode =
> queueContext.getConfiguration().isLegacyQueueMode();
> if (isLegacyQueueMode) {
> QueueCapacityType childrenCapacityType =
> getCapacityConfigurationTypeForQueues(childQueues);
> QueueCapacityType parentCapacityType =
> getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
> if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
> || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
> // We don't allow any mixed absolute + {weight, percentage} between
> // children and parent
> if (childrenCapacityType != parentCapacityType &&
> !this.getQueuePath()
> .equals(CapacitySchedulerConfiguration.ROOT)) {
> throw new IOException("Parent=" + this.getQueuePath()
> + ": When absolute minResource is used, we must make sure
> both "
> + "parent and child all use absolute minResource");
> }
> {code}
> The parent or childrenCapacityType will be considered as PERCENTAGE, because
> getCapacityConfigurationTypeForQueues fails to detect the absolute mode, here:
> {code:java}
> if
> (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
> .equals(Resources.none())) {
> absoluteMinResSet = true;
> {code}
> (It only happens in legacy queue mode.)
> h2. Possible fixes
> Possible fix in AbstractParentQueue.getCapacityConfigurationTypeForQueues
> using the capacityVector:
> {code:java}
> for (CSQueue queue : queues) {
> for (String
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810781#comment-17810781
]
ASF GitHub Bot commented on YARN-11641:
---
hadoop-yetus commented on PR #6435:
URL: https://github.com/apache/hadoop/pull/6435#issuecomment-1909795271
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 85m 56s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not
generate ASF License warnings. |
| | | 174m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6435 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint |
| uname | Linux dcafbda4b026 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 14a0c971aa3a443a9f54e96b0119eda54adf5552 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/5/testReport/ |
| Max. process+thread count | 915 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
|
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810465#comment-17810465
]
ASF GitHub Bot commented on YARN-11641:
---
hadoop-yetus commented on PR #6435:
URL: https://github.com/apache/hadoop/pull/6435#issuecomment-1908385303
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 1s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 13m 36s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 28s |
[/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 30s |
[/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| -1 :x: | mvnsite | 0m 29s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in trunk failed. |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 0m 28s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in trunk failed. |
| -1 :x: | shadedclient | 7m 12s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 0m 21s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/4/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 21s |
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810425#comment-17810425
]
ASF GitHub Bot commented on YARN-11641:
---
hadoop-yetus commented on PR #6435:
URL: https://github.com/apache/hadoop/pull/6435#issuecomment-1908266142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 29 unchanged - 0 fixed = 32 total (was 29) |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 101m 31s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 182m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel |
| | hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivitiesWithMultiNodesEnabled
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6435 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807083#comment-17807083
]
Shilun Fan commented on YARN-11641:
---
Set target to 3.5.0.
> Can't update a queue hierarchy in absolute mode when the configured
> capacities are zero
> ---
>
> Key: YARN-11641
> URL: https://issues.apache.org/jira/browse/YARN-11641
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: hierarchy.png
>
>
> h2. Error symptoms
> It is not possible to modify a queue hierarchy in absolute mode when the
> parent or every child queue of the parent has 0 min resource configured.
> {noformat}
> 2024-01-05 15:38:59,016 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager:
> Initialized queue: root.a.c
> 2024-01-05 15:38:59,016 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : Parent=root.a: When absolute
> minResource is used, we must make sure both parent and child all use absolute
> minResource
> {noformat}
> h2. Reproduction
> capacity-scheduler.xml
> {code:xml}
>
>
>
> yarn.scheduler.capacity.root.queues
> default,a
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960, vcores=16]
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.a.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.queues
> b,c
>
>
> yarn.scheduler.capacity.root.a.b.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.b.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.c.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.c.maximum-capacity
> [memory=39936, vcores=15]
>
>
> {code}
> !hierarchy.png!
> updatequeue.xml
> {code:xml}
>
>
>
> root.a
>
>
> capacity
> [memory=1024,vcores=1]
>
>
> maximum-capacity
> [memory=39936,vcores=15]
>
>
>
>
> {code}
> {code}
> $ curl -X PUT -H 'Content-Type: application/xml' -d @updatequeue.xml
> http://localhost:8088/ws/v1/cluster/scheduler-conf\?user.name\=yarn
> Failed to re-init queues : Parent=root.a: When absolute minResource is used,
> we must make sure both parent and child all use absolute minResource
> {code}
> h2. Root cause
> setChildQueues is called during reinit, where:
> {code:java}
> void setChildQueues(Collection childQueues) throws IOException {
> writeLock.lock();
> try {
> boolean isLegacyQueueMode =
> queueContext.getConfiguration().isLegacyQueueMode();
> if (isLegacyQueueMode) {
> QueueCapacityType childrenCapacityType =
> getCapacityConfigurationTypeForQueues(childQueues);
> QueueCapacityType parentCapacityType =
> getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
> if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
> || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
> // We don't allow any mixed absolute + {weight, percentage} between
> // children and parent
> if (childrenCapacityType != parentCapacityType &&
> !this.getQueuePath()
> .equals(CapacitySchedulerConfiguration.ROOT)) {
> throw new IOException("Parent=" + this.getQueuePath()
> + ": When absolute minResource is used, we must make sure
> both "
> + "parent and child all use absolute minResource");
> }
> {code}
> The parent or childrenCapacityType will be considered as PERCENTAGE, because
> getCapacityConfigurationTypeForQueues fails to detect the absolute mode, here:
> {code:java}
> if
> (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
> .equals(Resources.none())) {
> absoluteMinResSet = true;
> {code}
> (It only happens in legacy queue mode.)
> h2. Possible fixes
> Possible fix in AbstractParentQueue.getCapacityConfigurationTypeForQueues
> using the capacityVector:
> {code:java}
> for (CSQueue queue : queues) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> Set
>
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805574#comment-17805574
]
ASF GitHub Bot commented on YARN-11641:
---
hadoop-yetus commented on PR #6435:
URL: https://github.com/apache/hadoop/pull/6435#issuecomment-1887138976
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 44 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 26s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 34 unchanged - 0 fixed = 35 total (was 34) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 74m 38s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not
generate ASF License warnings. |
| | | 160m 15s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes
|
| |
hadoop.yarn.server.resourcemanager.scheduler.constraint.algorithm.TestLocalAllocationTagsManager
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigConverterMain
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerMultiNodes
|
| |
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805232#comment-17805232
]
ASF GitHub Bot commented on YARN-11641:
---
hadoop-yetus commented on PR #6435:
URL: https://github.com/apache/hadoop/pull/6435#issuecomment-1885276837
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 43 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 25s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 29 unchanged - 0 fixed = 34 total (was 29) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 11s | | the patch passed |
| -1 :x: | shadedclient | 29m 58s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 15m 49s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +0 :ok: | asflicense | 0m 22s | | ASF License check generated no
output? |
| | | 120m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6435 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint |
| uname | Linux c2206127136b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805189#comment-17805189
]
ASF GitHub Bot commented on YARN-11641:
---
tomicooler opened a new pull request, #6435:
URL: https://github.com/apache/hadoop/pull/6435
### Description of PR
WIP: until the other 2 ticket is merged, I'll rebase this PR.
Details in the Jira:
[YARN-11641](https://issues.apache.org/jira/browse/YARN-11641)
Note: it is not possible to rely on the capacityVectors (at least not for
the root queue, which is always in percentage mode with 100%). So I decided to
go with the `checkConfigTypeIsAbsoluteResource` approach.
### How was this patch tested?
Tested manually and added a unit test.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'YARN-11641 Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Can't update a queue hierarchy in absolute mode when the configured
> capacities are zero
> ---
>
> Key: YARN-11641
> URL: https://issues.apache.org/jira/browse/YARN-11641
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Attachments: hierarchy.png
>
>
> h2. Error symptoms
> It is not possible to modify a queue hierarchy in absolute mode when the
> parent or every child queue of the parent has 0 min resource configured.
> {noformat}
> 2024-01-05 15:38:59,016 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager:
> Initialized queue: root.a.c
> 2024-01-05 15:38:59,016 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : Parent=root.a: When absolute
> minResource is used, we must make sure both parent and child all use absolute
> minResource
> {noformat}
> h2. Reproduction
> capacity-scheduler.xml
> {code:xml}
>
>
>
> yarn.scheduler.capacity.root.queues
> default,a
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960, vcores=16]
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.a.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.queues
> b,c
>
>
> yarn.scheduler.capacity.root.a.b.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.b.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.c.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.c.maximum-capacity
> [memory=39936, vcores=15]
>
>
> {code}
> !hierarchy.png!
> updatequeue.xml
> {code:xml}
>
>
>
> root.a
>
>
> capacity
> [memory=1024,vcores=1]
>
>
> maximum-capacity
> [memory=39936,vcores=15]
>
>
>
>
> {code}
> {code}
> $ curl -X PUT -H 'Content-Type: application/xml' -d @updatequeue.xml
> http://localhost:8088/ws/v1/cluster/scheduler-conf\?user.name\=yarn
> Failed to re-init queues : Parent=root.a: When absolute minResource is used,
> we must make sure both parent and child all use absolute minResource
> {code}
> h2. Root cause
> setChildQueues is called during reinit, where:
> {code:java}
> void setChildQueues(Collection childQueues) throws IOException {
> writeLock.lock();
> try {
> boolean isLegacyQueueMode =
> queueContext.getConfiguration().isLegacyQueueMode();
> if (isLegacyQueueMode) {
> QueueCapacityType childrenCapacityType =
> getCapacityConfigurationTypeForQueues(childQueues);
> QueueCapacityType parentCapacityType =
> getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
> if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
> || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
> // We don't allow any mixed
