[jira] [Commented] (YARN-10174) Add colored policies to enable manual load balancing across sub clusters
[ https://issues.apache.org/jira/browse/YARN-10174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769401#comment-17769401 ] zhengchenyu commented on YARN-10174: [~slfan1989] Thanks for your reply. Adjusting parameters on the client side is indeed a unsafe behavior. I will implement this according to your suggestion. > Add colored policies to enable manual load balancing across sub clusters > > > Key: YARN-10174 > URL: https://issues.apache.org/jira/browse/YARN-10174 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Young Chen >Assignee: zhengchenyu >Priority: Major > > Add colored policies to enable manual load balancing across sub clusters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8862) [GPG] Add Yarn Registry cleanup in ApplicationCleaner
[
https://issues.apache.org/jira/browse/YARN-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769353#comment-17769353
]
ASF GitHub Bot commented on YARN-8862:
--
goiri commented on code in PR #6083:
URL: https://github.com/apache/hadoop/pull/6083#discussion_r1337823389
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/GlobalPolicyGenerator.java:
##
@@ -214,6 +230,10 @@ protected void serviceStart() throws Exception {
@Override
protected void serviceStop() throws Exception {
+if (this.registry != null) {
+ this.registry.stop();
Review Comment:
Set to null?
> [GPG] Add Yarn Registry cleanup in ApplicationCleaner
> -
>
> Key: YARN-8862
> URL: https://issues.apache.org/jira/browse/YARN-8862
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-8862-YARN-7402.v1.patch,
> YARN-8862-YARN-7402.v2.patch, YARN-8862-YARN-7402.v3.patch,
> YARN-8862-YARN-7402.v4.patch, YARN-8862-YARN-7402.v5.patch,
> YARN-8862-YARN-7402.v6.patch
>
>
> In Yarn Federation, we use Yarn Registry to use the AMToken for UAMs in
> secondary sub-clusters. Because of potential more app attempts later,
> AMRMProxy cannot kill the UAM and delete the tokens when one local attempt
> finishes. So similar to the StateStore application table, we need
> ApplicationCleaner in GPG to cleanup the finished app entries in Yarn
> Registry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11498) Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml
[
https://issues.apache.org/jira/browse/YARN-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769317#comment-17769317
]
ASF GitHub Bot commented on YARN-11498:
---
hadoop-yetus commented on PR #6063:
URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1736116392
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 46s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 40m 16s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/4/artifact/out/branch-mvninstall-root.txt)
| root in branch-3.3 failed. |
| +1 :green_heart: | compile | 19m 33s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 8m 28s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 6m 24s | | branch-3.3 passed |
| -1 :x: | shadedclient | 129m 5s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 26m 16s | | the patch passed |
| +1 :green_heart: | compile | 18m 51s | | the patch passed |
| +1 :green_heart: | javac | 18m 51s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 8m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 19s | | the patch passed |
| -1 :x: | shadedclient | 56m 14s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 18m 4s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 4m 49s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 22m 50s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | unit | 4m 11s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| -1 :x: | unit | 108m 50s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| -1 :x: | unit | 1m 6s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt)
| hadoop-yarn-applications-catalog-webapp in the patch failed. |
| -1 :x: | unit | 0m 43s |
[/patch-unit-hadoop-tools_hadoop-resourceestimator.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/4/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt)
| hadoop-resourceestimator in the patch failed. |
| -1 :x: | unit | 0m 43s |
[/patch-unit-hadoop-client-modules_hadoop-client-runtime.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/4/artifact/out/patch-unit-hadoop-client-modules_hadoop-client-runtime.txt)
| hadoop-client-runtime in the patch failed. |
| -1 :x: | unit | 0m 41s |
[/patch-unit-hadoop-client-modules_hadoop-client-minicluster.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6063/4/artifact/out/patch-unit-hadoop-client-modules_hadoop-client-minicluster.txt)
| hadoop-client-minicluster in the patch failed. |
| +0 :ok: | asflicense | 0m 46s | | ASF License check generated no
output? |
| | | 410m 19s | | |
| Reason | Tes
[jira] [Commented] (YARN-8862) [GPG] Add Yarn Registry cleanup in ApplicationCleaner
[
https://issues.apache.org/jira/browse/YARN-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769297#comment-17769297
]
ASF GitHub Bot commented on YARN-8862:
--
hadoop-yetus commented on PR #6083:
URL: https://github.com/apache/hadoop/pull/6083#issuecomment-1736001048
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 31s | | trunk passed |
| +1 :green_heart: | compile | 2m 35s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 2m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 40s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 3s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 2m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 2m 23s | | the patch passed |
| +1 :green_heart: | compile | 2m 13s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 2m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 33s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 0m 56s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 149m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6083/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6083 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 67b6f698229e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 101e20d0933c66b998c3a8b0c5674b3bd8ecbb7a |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6083/7/testReport/ |
| Max. process+thread c
[jira] [Commented] (YARN-8862) [GPG] Add Yarn Registry cleanup in ApplicationCleaner
[
https://issues.apache.org/jira/browse/YARN-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769250#comment-17769250
]
ASF GitHub Bot commented on YARN-8862:
--
slfan1989 commented on code in PR #6083:
URL: https://github.com/apache/hadoop/pull/6083#discussion_r1337380994
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##
@@ -761,7 +761,7 @@ public AppsInfo getApps(HttpServletRequest hsr, String
stateQuery,
});
if (apps.getApps().isEmpty()) {
- return null;
+ return apps;
Review Comment:
Thank you very much for your help in reviewing the code! I was doing the
integration test of YARN Federation recently and accidentally mixed some code
into YARN-8862. I will fix this part of the code.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/TestRouterWebServicesREST.java:
##
@@ -373,6 +373,8 @@ public void testClusterInfoXML() throws Exception {
assertEquals(
rmResponse.getRMVersion(),
routerResponse.getRMVersion());
+
+Thread.sleep(1000);
Review Comment:
I will fix it.
> [GPG] Add Yarn Registry cleanup in ApplicationCleaner
> -
>
> Key: YARN-8862
> URL: https://issues.apache.org/jira/browse/YARN-8862
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-8862-YARN-7402.v1.patch,
> YARN-8862-YARN-7402.v2.patch, YARN-8862-YARN-7402.v3.patch,
> YARN-8862-YARN-7402.v4.patch, YARN-8862-YARN-7402.v5.patch,
> YARN-8862-YARN-7402.v6.patch
>
>
> In Yarn Federation, we use Yarn Registry to use the AMToken for UAMs in
> secondary sub-clusters. Because of potential more app attempts later,
> AMRMProxy cannot kill the UAM and delete the tokens when one local attempt
> finishes. So similar to the StateStore application table, we need
> ApplicationCleaner in GPG to cleanup the finished app entries in Yarn
> Registry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8862) [GPG] Add Yarn Registry cleanup in ApplicationCleaner
[ https://issues.apache.org/jira/browse/YARN-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769251#comment-17769251 ] ASF GitHub Bot commented on YARN-8862: -- slfan1989 commented on code in PR #6083: URL: https://github.com/apache/hadoop/pull/6083#discussion_r1337382027 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/resources/yarn-site.xml: ## @@ -39,7 +39,7 @@ yarn.resourcemanager.resource-profiles.source-file profiles/sample-profiles-1.json - + Review Comment: I will fix it. > [GPG] Add Yarn Registry cleanup in ApplicationCleaner > - > > Key: YARN-8862 > URL: https://issues.apache.org/jira/browse/YARN-8862 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Labels: pull-request-available > Attachments: YARN-8862-YARN-7402.v1.patch, > YARN-8862-YARN-7402.v2.patch, YARN-8862-YARN-7402.v3.patch, > YARN-8862-YARN-7402.v4.patch, YARN-8862-YARN-7402.v5.patch, > YARN-8862-YARN-7402.v6.patch > > > In Yarn Federation, we use Yarn Registry to use the AMToken for UAMs in > secondary sub-clusters. Because of potential more app attempts later, > AMRMProxy cannot kill the UAM and delete the tokens when one local attempt > finishes. So similar to the StateStore application table, we need > ApplicationCleaner in GPG to cleanup the finished app entries in Yarn > Registry. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11498) Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml
[ https://issues.apache.org/jira/browse/YARN-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769146#comment-17769146 ] ASF GitHub Bot commented on YARN-11498: --- pjfanning commented on PR #6063: URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1735420751 TestFSConfigToCSConfigConverterMain crashed in the last run. I've restarted the build to see if it was a one-off. > Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml > -- > > Key: YARN-11498 > URL: https://issues.apache.org/jira/browse/YARN-11498 > Project: Hadoop YARN > Issue Type: Task > Components: build >Reporter: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > This exclusion is done to reduce CVEs present due to an older version of > Jettison(1.1) being pulled in with jersey-json artifact. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11498) Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml
[ https://issues.apache.org/jira/browse/YARN-11498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769133#comment-17769133 ] ASF GitHub Bot commented on YARN-11498: --- steveloughran commented on PR #6063: URL: https://github.com/apache/hadoop/pull/6063#issuecomment-1735378182 any suggestions as to why the client is reported as a failure? the console output doesn't highlight any problems > Exclude Jettison from jersey-json artifact in hadoop-yarn-common's pom.xml > -- > > Key: YARN-11498 > URL: https://issues.apache.org/jira/browse/YARN-11498 > Project: Hadoop YARN > Issue Type: Task > Components: build >Reporter: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > This exclusion is done to reduce CVEs present due to an older version of > Jettison(1.1) being pulled in with jersey-json artifact. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
