[jira] [Commented] (YARN-11634) Speed-up TestTimelineClient
[
https://issues.apache.org/jira/browse/YARN-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805347#comment-17805347
]
ASF GitHub Bot commented on YARN-11634:
---
hadoop-yetus commented on PR #6419:
URL: https://github.com/apache/hadoop/pull/6419#issuecomment-1886092782
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 1s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6419/6/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant
spotbugs warnings. |
| +1 :green_heart: | shadedclient | 19m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6419/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 1
new + 23 unchanged - 3 fixed = 24 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) |
| +1 :green_heart: | shadedclient | 19m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 26s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 21s | | The patch does not
generate ASF License warnings. |
| | | 85m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6419/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6419 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b484a7962e19 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aeaa452863fb337ea2971cce4d0918dd77c27315 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-open
[jira] [Commented] (YARN-11634) Speed-up TestTimelineClient
[ https://issues.apache.org/jira/browse/YARN-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805338#comment-17805338 ] ASF GitHub Bot commented on YARN-11634: --- slfan1989 commented on PR #6419: URL: https://github.com/apache/hadoop/pull/6419#issuecomment-1886011678 @brumi1024 @K0K0V0K Can you help review the code again? Thank you very much! > Speed-up TestTimelineClient > --- > > Key: YARN-11634 > URL: https://issues.apache.org/jira/browse/YARN-11634 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Bence Kosztolnik >Assignee: Bence Kosztolnik >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > The TimelineConnector.class has a hardcoded 1-minute connection time out, > which makes the TestTimelineClient a long-running test (~15:30 min). > Decreasing the timeout to 10ms will speed up the test run (~56 sec). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11634) Speed-up TestTimelineClient
[
https://issues.apache.org/jira/browse/YARN-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805337#comment-17805337
]
ASF GitHub Bot commented on YARN-11634:
---
slfan1989 commented on code in PR #6419:
URL: https://github.com/apache/hadoop/pull/6419#discussion_r1448159809
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java:
##
@@ -89,7 +89,7 @@ public void tearDown() throws Exception {
if (isSSLConfigured()) {
KeyStoreTestUtil.cleanupSSLConfig(keystoresDir, sslConfDir);
}
-TimelineConnector.DEFAULT_SOCKET_TIMEOUT = 60_000;
+client.getConnector().setSocketTimeOut(60_1000);
Review Comment:
@K0K0V0K Thanks for the review! I will improve it.
> Speed-up TestTimelineClient
> ---
>
> Key: YARN-11634
> URL: https://issues.apache.org/jira/browse/YARN-11634
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Bence Kosztolnik
>Assignee: Bence Kosztolnik
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The TimelineConnector.class has a hardcoded 1-minute connection time out,
> which makes the TestTimelineClient a long-running test (~15:30 min).
> Decreasing the timeout to 10ms will speed up the test run (~56 sec).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11645) Fix flaky json assert tests in TestRMWebServices
[
https://issues.apache.org/jira/browse/YARN-11645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805267#comment-17805267
]
ASF GitHub Bot commented on YARN-11645:
---
hadoop-yetus commented on PR #6432:
URL: https://github.com/apache/hadoop/pull/6432#issuecomment-1885452493
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 34 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 46m 14s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/2/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 41s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 102m 33s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 238m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedMode
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreation
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/2/artifact/out/Dockerfile
|
| GITHUB PR | https://git
[jira] [Commented] (YARN-11645) Fix flaky json assert tests in TestRMWebServices
[
https://issues.apache.org/jira/browse/YARN-11645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805253#comment-17805253
]
ASF GitHub Bot commented on YARN-11645:
---
hadoop-yetus commented on PR #6432:
URL: https://github.com/apache/hadoop/pull/6432#issuecomment-1885403725
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 34 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 1s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 40s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 101m 47s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 245m 11s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedulerMixedMode
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedLegacyQueueCreation
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6432/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6432 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient sp
[jira] [Commented] (YARN-11646) QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
[
https://issues.apache.org/jira/browse/YARN-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805240#comment-17805240
]
ASF GitHub Bot commented on YARN-11646:
---
hadoop-yetus commented on PR #6433:
URL: https://github.com/apache/hadoop/pull/6433#issuecomment-1885332277
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 5s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 167m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6433 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux da84b7c3b3c3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 29d5dd4b7c5efa93901e2f398af0a36d233e476b |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/2/testReport/ |
| Max. process+thread count | 941 (vs. ulimit of 5
[jira] [Commented] (YARN-11646) QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
[
https://issues.apache.org/jira/browse/YARN-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805238#comment-17805238
]
ASF GitHub Bot commented on YARN-11646:
---
hadoop-yetus commented on PR #6433:
URL: https://github.com/apache/hadoop/pull/6433#issuecomment-1885328661
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 84m 56s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 21s | | The patch does not
generate ASF License warnings. |
| | | 168m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6433 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0d3fc268fb08 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 29d5dd4b7c5efa93901e2f398af0a36d233e476b |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6433/1/testReport/ |
| Max. process+thread count | 953 (vs. ulimit of 5
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805232#comment-17805232
]
ASF GitHub Bot commented on YARN-11641:
---
hadoop-yetus commented on PR #6435:
URL: https://github.com/apache/hadoop/pull/6435#issuecomment-1885276837
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 43 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 25s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 29 unchanged - 0 fixed = 34 total (was 29) |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 11s | | the patch passed |
| -1 :x: | shadedclient | 29m 58s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 15m 49s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +0 :ok: | asflicense | 0m 22s | | ASF License check generated no
output? |
| | | 120m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestLeaderElectorService |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6435/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6435 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint |
| uname | Linux c2206127136b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git re
[jira] [Commented] (YARN-11634) Speed-up TestTimelineClient
[
https://issues.apache.org/jira/browse/YARN-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805210#comment-17805210
]
ASF GitHub Bot commented on YARN-11634:
---
hadoop-yetus commented on PR #6419:
URL: https://github.com/apache/hadoop/pull/6419#issuecomment-1885164744
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 47s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 0s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6419/5/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 1 extant
spotbugs warnings. |
| +1 :green_heart: | shadedclient | 19m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6419/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 2
new + 23 unchanged - 3 fixed = 25 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) |
| +1 :green_heart: | shadedclient | 19m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 27s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 85m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6419/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6419 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c572935bbf5a 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 506efb6df641d5797541869452a42bb23397fb58 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-open
[jira] [Commented] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805189#comment-17805189
]
ASF GitHub Bot commented on YARN-11641:
---
tomicooler opened a new pull request, #6435:
URL: https://github.com/apache/hadoop/pull/6435
### Description of PR
WIP: until the other 2 ticket is merged, I'll rebase this PR.
Details in the Jira:
[YARN-11641](https://issues.apache.org/jira/browse/YARN-11641)
Note: it is not possible to rely on the capacityVectors (at least not for
the root queue, which is always in percentage mode with 100%). So I decided to
go with the `checkConfigTypeIsAbsoluteResource` approach.
### How was this patch tested?
Tested manually and added a unit test.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'YARN-11641 Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Can't update a queue hierarchy in absolute mode when the configured
> capacities are zero
> ---
>
> Key: YARN-11641
> URL: https://issues.apache.org/jira/browse/YARN-11641
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Attachments: hierarchy.png
>
>
> h2. Error symptoms
> It is not possible to modify a queue hierarchy in absolute mode when the
> parent or every child queue of the parent has 0 min resource configured.
> {noformat}
> 2024-01-05 15:38:59,016 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager:
> Initialized queue: root.a.c
> 2024-01-05 15:38:59,016 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : Parent=root.a: When absolute
> minResource is used, we must make sure both parent and child all use absolute
> minResource
> {noformat}
> h2. Reproduction
> capacity-scheduler.xml
> {code:xml}
>
>
>
> yarn.scheduler.capacity.root.queues
> default,a
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960, vcores=16]
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.a.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.queues
> b,c
>
>
> yarn.scheduler.capacity.root.a.b.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.b.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.c.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.c.maximum-capacity
> [memory=39936, vcores=15]
>
>
> {code}
> !hierarchy.png!
> updatequeue.xml
> {code:xml}
>
>
>
> root.a
>
>
> capacity
> [memory=1024,vcores=1]
>
>
> maximum-capacity
> [memory=39936,vcores=15]
>
>
>
>
> {code}
> {code}
> $ curl -X PUT -H 'Content-Type: application/xml' -d @updatequeue.xml
> http://localhost:8088/ws/v1/cluster/scheduler-conf\?user.name\=yarn
> Failed to re-init queues : Parent=root.a: When absolute minResource is used,
> we must make sure both parent and child all use absolute minResource
> {code}
> h2. Root cause
> setChildQueues is called during reinit, where:
> {code:java}
> void setChildQueues(Collection childQueues) throws IOException {
> writeLock.lock();
> try {
> boolean isLegacyQueueMode =
> queueContext.getConfiguration().isLegacyQueueMode();
> if (isLegacyQueueMode) {
> QueueCapacityType childrenCapacityType =
> getCapacityConfigurationTypeForQueues(childQueues);
> QueueCapacityType parentCapacityType =
> getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
> if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
> || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
> // We don't al
[jira] [Updated] (YARN-11641) Can't update a queue hierarchy in absolute mode when the configured capacities are zero
[
https://issues.apache.org/jira/browse/YARN-11641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11641:
--
Labels: pull-request-available (was: )
> Can't update a queue hierarchy in absolute mode when the configured
> capacities are zero
> ---
>
> Key: YARN-11641
> URL: https://issues.apache.org/jira/browse/YARN-11641
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: hierarchy.png
>
>
> h2. Error symptoms
> It is not possible to modify a queue hierarchy in absolute mode when the
> parent or every child queue of the parent has 0 min resource configured.
> {noformat}
> 2024-01-05 15:38:59,016 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager:
> Initialized queue: root.a.c
> 2024-01-05 15:38:59,016 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : Parent=root.a: When absolute
> minResource is used, we must make sure both parent and child all use absolute
> minResource
> {noformat}
> h2. Reproduction
> capacity-scheduler.xml
> {code:xml}
>
>
>
> yarn.scheduler.capacity.root.queues
> default,a
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960, vcores=16]
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=1024, vcores=1]
>
>
> yarn.scheduler.capacity.root.a.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.queues
> b,c
>
>
> yarn.scheduler.capacity.root.a.b.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.b.maximum-capacity
> [memory=39936, vcores=15]
>
>
> yarn.scheduler.capacity.root.a.c.capacity
> [memory=0, vcores=0]
>
>
> yarn.scheduler.capacity.root.a.c.maximum-capacity
> [memory=39936, vcores=15]
>
>
> {code}
> !hierarchy.png!
> updatequeue.xml
> {code:xml}
>
>
>
> root.a
>
>
> capacity
> [memory=1024,vcores=1]
>
>
> maximum-capacity
> [memory=39936,vcores=15]
>
>
>
>
> {code}
> {code}
> $ curl -X PUT -H 'Content-Type: application/xml' -d @updatequeue.xml
> http://localhost:8088/ws/v1/cluster/scheduler-conf\?user.name\=yarn
> Failed to re-init queues : Parent=root.a: When absolute minResource is used,
> we must make sure both parent and child all use absolute minResource
> {code}
> h2. Root cause
> setChildQueues is called during reinit, where:
> {code:java}
> void setChildQueues(Collection childQueues) throws IOException {
> writeLock.lock();
> try {
> boolean isLegacyQueueMode =
> queueContext.getConfiguration().isLegacyQueueMode();
> if (isLegacyQueueMode) {
> QueueCapacityType childrenCapacityType =
> getCapacityConfigurationTypeForQueues(childQueues);
> QueueCapacityType parentCapacityType =
> getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
> if (childrenCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE
> || parentCapacityType == QueueCapacityType.ABSOLUTE_RESOURCE) {
> // We don't allow any mixed absolute + {weight, percentage} between
> // children and parent
> if (childrenCapacityType != parentCapacityType &&
> !this.getQueuePath()
> .equals(CapacitySchedulerConfiguration.ROOT)) {
> throw new IOException("Parent=" + this.getQueuePath()
> + ": When absolute minResource is used, we must make sure
> both "
> + "parent and child all use absolute minResource");
> }
> {code}
> The parent or childrenCapacityType will be considered as PERCENTAGE, because
> getCapacityConfigurationTypeForQueues fails to detect the absolute mode, here:
> {code:java}
> if
> (!queue.getQueueResourceQuotas().getConfiguredMinResource(nodeLabel)
> .equals(Resources.none())) {
> absoluteMinResSet = true;
> {code}
> (It only happens in legacy queue mode.)
> h2. Possible fixes
> Possible fix in AbstractParentQueue.getCapacityConfigurationTypeForQueues
> using the capacityVector:
> {code:java}
> for (CSQueue queue : queues) {
> for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
> Set
> definedCap
[jira] [Commented] (YARN-11634) Speed-up TestTimelineClient
[
https://issues.apache.org/jira/browse/YARN-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805183#comment-17805183
]
ASF GitHub Bot commented on YARN-11634:
---
K0K0V0K commented on code in PR #6419:
URL: https://github.com/apache/hadoop/pull/6419#discussion_r1447513712
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java:
##
@@ -89,7 +89,7 @@ public void tearDown() throws Exception {
if (isSSLConfigured()) {
KeyStoreTestUtil.cleanupSSLConfig(keystoresDir, sslConfDir);
}
-TimelineConnector.DEFAULT_SOCKET_TIMEOUT = 60_000;
+client.getConnector().setSocketTimeOut(60_1000);
Review Comment:
Hi @slfan1989 !
Thanks for the patch!
If i see well we have a typo here, 60_1000 seems too much.
> Speed-up TestTimelineClient
> ---
>
> Key: YARN-11634
> URL: https://issues.apache.org/jira/browse/YARN-11634
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Bence Kosztolnik
>Assignee: Bence Kosztolnik
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The TimelineConnector.class has a hardcoded 1-minute connection time out,
> which makes the TestTimelineClient a long-running test (~15:30 min).
> Decreasing the timeout to 10ms will speed up the test run (~56 sec).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11646) QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
[
https://issues.apache.org/jira/browse/YARN-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805176#comment-17805176
]
ASF GitHub Bot commented on YARN-11646:
---
tomicooler opened a new pull request, #6433:
URL: https://github.com/apache/hadoop/pull/6433
Change-Id: Id00a5382d982dcf33ec8a3769fc33419b660968b
### Description of PR
There is no reason to ignore the configured capacity if the memory is 0 in
the configuration.
It makes it impossible to configure a zero absolute resource capacity.
Example:
```
root.default.capacity=[memory=0, vcores=0]
root.default.maximum-capacity=[memory=2048, vcores=2]
```
### How was this patch tested?
Ran the unit tests.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'YARN-11646. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
>
>
> Key: YARN-11646
> URL: https://issues.apache.org/jira/browse/YARN-11646
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
>
> There is no reason to ignore the configured capacity if the memory is 0 in
> the configuration.
> It makes it impossible to configure a zero absolute resource capacity.
> Example:
> {noformat}
> root.default.capacity=[memory=0, vcores=0]
> root.default.maximum-capacity=[memory=2048, vcores=2]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11646) QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
[
https://issues.apache.org/jira/browse/YARN-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11646:
--
Labels: pull-request-available (was: )
> QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
>
>
> Key: YARN-11646
> URL: https://issues.apache.org/jira/browse/YARN-11646
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
>
> There is no reason to ignore the configured capacity if the memory is 0 in
> the configuration.
> It makes it impossible to configure a zero absolute resource capacity.
> Example:
> {noformat}
> root.default.capacity=[memory=0, vcores=0]
> root.default.maximum-capacity=[memory=2048, vcores=2]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11646) QueueCapacityConfigParser shouldn't ignore capacity config with 0 memory
Tamas Domok created YARN-11646:
--
Summary: QueueCapacityConfigParser shouldn't ignore capacity
config with 0 memory
Key: YARN-11646
URL: https://issues.apache.org/jira/browse/YARN-11646
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 3.4.0
Reporter: Tamas Domok
Assignee: Tamas Domok
There is no reason to ignore the configured capacity if the memory is 0 in the
configuration.
It makes it impossible to configure a zero absolute resource capacity.
Example:
{noformat}
root.default.capacity=[memory=0, vcores=0]
root.default.maximum-capacity=[memory=2048, vcores=2]
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11645) Fix flaky json assert tests in TestRMWebServices
[
https://issues.apache.org/jira/browse/YARN-11645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805162#comment-17805162
]
ASF GitHub Bot commented on YARN-11645:
---
tomicooler opened a new pull request, #6432:
URL: https://github.com/apache/hadoop/pull/6432
Change-Id: Iff7c8d0522003f9ae9d5c01a1070158e45903d3c
### Description of PR
The JSON assert tests that were using the `createMutableRM` were flaky the
queue order changed in the "queue": [ {}, {} ] lists. We had a workaround like
`reinitAfterNodeChane` (with a typo) but it didn't work well.
Since there is a comment in the production code in
`CapacitySchedulerInfo.getQueues` about a marshalling issue,
I didn't introduce sorting there, just here for the tests.
`git grep -rn '"queuePath" :'`
```
dynamic-testAbsoluteMode-0.json:21: "queuePath" : "root",
dynamic-testAbsoluteMode-0.json:27: "queuePath" : "root.default",
dynamic-testAbsoluteMode-0.json:505: "queuePath" : "root.test1",
dynamic-testAbsoluteMode-0.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testAbsoluteMode-0.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testAbsoluteMode-0.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testAbsoluteMode-0.json:2271: "queuePath" : "root.test2",
dynamic-testAbsoluteMode-16.json:21: "queuePath" : "root",
dynamic-testAbsoluteMode-16.json:27: "queuePath" : "root.default",
dynamic-testAbsoluteMode-16.json:505: "queuePath" : "root.test1",
dynamic-testAbsoluteMode-16.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testAbsoluteMode-16.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testAbsoluteMode-16.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testAbsoluteMode-16.json:2271: "queuePath" : "root.test2",
dynamic-testAbsoluteMode-32.json:21: "queuePath" : "root",
dynamic-testAbsoluteMode-32.json:27: "queuePath" : "root.default",
dynamic-testAbsoluteMode-32.json:505: "queuePath" : "root.test1",
dynamic-testAbsoluteMode-32.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testAbsoluteMode-32.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testAbsoluteMode-32.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testAbsoluteMode-32.json:2271: "queuePath" : "root.test2",
dynamic-testAbsoluteMode-legacy-0.json:21: "queuePath" : "root",
dynamic-testAbsoluteMode-legacy-0.json:27: "queuePath" :
"root.default",
dynamic-testAbsoluteMode-legacy-0.json:505: "queuePath" :
"root.test1",
dynamic-testAbsoluteMode-legacy-0.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testAbsoluteMode-legacy-0.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testAbsoluteMode-legacy-0.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testAbsoluteMode-legacy-0.json:2271: "queuePath" :
"root.test2",
dynamic-testPercentageMode-0.json:21: "queuePath" : "root",
dynamic-testPercentageMode-0.json:27: "queuePath" : "root.default",
dynamic-testPercentageMode-0.json:505: "queuePath" : "root.test1",
dynamic-testPercentageMode-0.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testPercentageMode-0.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testPercentageMode-0.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testPercentageMode-0.json:2271: "queuePath" : "root.test2",
dynamic-testPercentageMode-16.json:21: "queuePath" : "root",
dynamic-testPercentageMode-16.json:27: "queuePath" : "root.default",
dynamic-testPercentageMode-16.json:505: "queuePath" : "root.test1",
dynamic-testPercentageMode-16.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testPercentageMode-16.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testPercentageMode-16.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testPercentageMode-16.json:2271: "queuePath" : "root.test2",
dynamic-testPercentageMode-32.json:21: "queuePath" : "root",
dynamic-testPercentageMode-32.json:27: "queuePath" : "root.default",
dynamic-testPercentageMode-32.json:505: "queuePath" : "root.test1",
dynamic-testPercentageMode-32.json:522: "queuePath" :
"root.test1.test1_1",
dynamic-testPercentageMode-32.json:1001: "queuePath" :
"root.test1.test1_2",
dynamic-testPercentageMode-32.json:1480: "queuePath" :
"root.test1.test1_3",
dynamic-testPercentageMode-32.json:2271: "queuePa
[jira] [Updated] (YARN-11645) Fix flaky json assert tests in TestRMWebServices
[ https://issues.apache.org/jira/browse/YARN-11645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11645: -- Labels: pull-request-available (was: ) > Fix flaky json assert tests in TestRMWebServices > > > Key: YARN-11645 > URL: https://issues.apache.org/jira/browse/YARN-11645 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.4.0 >Reporter: Tamas Domok >Assignee: Tamas Domok >Priority: Major > Labels: pull-request-available > > TestRMWebServicesCapacitySchedDynamicConfig and > TestRMWebServicesCapacitySchedulerMixedMode are flaky due to changes in the > queue order. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11645) Fix flaky json assert tests in TestRMWebServices
Tamas Domok created YARN-11645: -- Summary: Fix flaky json assert tests in TestRMWebServices Key: YARN-11645 URL: https://issues.apache.org/jira/browse/YARN-11645 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.4.0 Reporter: Tamas Domok Assignee: Tamas Domok TestRMWebServicesCapacitySchedDynamicConfig and TestRMWebServicesCapacitySchedulerMixedMode are flaky due to changes in the queue order. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11643) Skip unnecessary pre-check in Multi Node Placement
[
https://issues.apache.org/jira/browse/YARN-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805023#comment-17805023
]
ASF GitHub Bot commented on YARN-11643:
---
hadoop-yetus commented on PR #6426:
URL: https://github.com/apache/hadoop/pull/6426#issuecomment-1884422714
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 1s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 168m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6426/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f212591512f7 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4c340e66690d361073ab7a746dbe4a9e7c446f2e |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6426/2/testReport/ |
| Max. process+thread count | 970 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6426/2/console |
| ve
