[jira] [Commented] (YARN-11425) [Federation] Router Supports SubClusterCleaner
[
https://issues.apache.org/jira/browse/YARN-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683676#comment-17683676
]
ASF GitHub Bot commented on YARN-11425:
---
hadoop-yetus commented on PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#issuecomment-1415055787
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 35s | | trunk passed |
| +1 :green_heart: | compile | 9m 56s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 8m 38s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 9s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 39s | | the patch passed |
| +1 :green_heart: | compile | 9m 18s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 9m 18s | | the patch passed |
| +1 :green_heart: | compile | 8m 32s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 35s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5326/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 177 unchanged
- 0 fixed = 179 total (was 177) |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 47s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 12s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 43s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 20s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 0m 45s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 187m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5326/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5326 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux ba4bbe43228c 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 264c2bae5da8bc1f33e39870a2ffceadb3aa549a |
| Default Java | Private Build-1.8.0_352-8u352-g
[jira] [Commented] (YARN-11425) [Federation] Router Supports SubClusterCleaner
[
https://issues.apache.org/jira/browse/YARN-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683668#comment-17683668
]
ASF GitHub Bot commented on YARN-11425:
---
hadoop-yetus commented on PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#issuecomment-1414990116
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 22s | | trunk passed |
| +1 :green_heart: | compile | 9m 54s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 9m 9s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 15s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 48s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 47s | | the patch passed |
| +1 :green_heart: | compile | 9m 54s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 9m 54s | | the patch passed |
| +1 :green_heart: | compile | 8m 45s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 36s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5326/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 177 unchanged
- 0 fixed = 178 total (was 177) |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 47s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 42s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 51s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 22s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 0m 40s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 188m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5326/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5326 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux aa70df0a8832 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b75454f4f6216c74f8562787ff17f443e33b5af6 |
| Default Java | Private Build-1.8.0_352-8u352-g
[jira] [Commented] (YARN-11419) Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc
[
https://issues.apache.org/jira/browse/YARN-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683663#comment-17683663
]
ASF GitHub Bot commented on YARN-11419:
---
hadoop-yetus commented on PR #5309:
URL: https://github.com/apache/hadoop/pull/5309#issuecomment-1414980064
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| -1 :x: | javadoc | 0m 41s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5309/4/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-client in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| -1 :x: | javadoc | 0m 22s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5309/4/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-client in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 28m 14s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 129m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5309/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5309 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 06bd08ac6e2e 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 05b14c1f60b61487ed06fabbba84b6c1ac373f35 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-
[jira] [Commented] (YARN-11178) Avoid CPU busy idling and resource wasting in DelegationTokenRenewerPoolTracker thread
[
https://issues.apache.org/jira/browse/YARN-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683638#comment-17683638
]
ASF GitHub Bot commented on YARN-11178:
---
slfan1989 commented on PR #4435:
URL: https://github.com/apache/hadoop/pull/4435#issuecomment-1414736328
@LennonChin Thank you for your contribution! Is the unit test error caused
by this pr?
> Avoid CPU busy idling and resource wasting in
> DelegationTokenRenewerPoolTracker thread
> --
>
> Key: YARN-11178
> URL: https://issues.apache.org/jira/browse/YARN-11178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, security
>Affects Versions: 3.3.1, 3.3.2, 3.3.3, 3.3.4
> Environment: Hadoop 3.3.3 with Kerberos, Ranger 2.1.0, Hive 2.3.7 and
> Spark 3.0.3
>Reporter: Lennon Chin
>Priority: Minor
> Labels: pull-request-available
> Attachments: YARN-11178.CPU idling busy 100% before optimized.png,
> YARN-11178.CPU normal after optimized.png, YARN-11178.CPU profile for idling
> busy 100% before optimized.html, YARN-11178.CPU profile for idling busy 100%
> before optimized.png, YARN-11178.CPU profile for normal after optimized.html,
> YARN-11178.CPU profile for normal after optimized.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The DelegationTokenRenewerPoolTracker thread is busy wasting CPU resource in
> empty poll iterate when there is no delegation token renewer event task in
> the futures map:
> {code:java}
> //
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.DelegationTokenRenewerPoolTracker#run
> @Override
> public void run() {
> // this while true loop is busy when the `futures` is empty
> while (true) {
> for (Map.Entry> entry : futures
> .entrySet()) {
> DelegationTokenRenewerEvent evt = entry.getKey();
> Future future = entry.getValue();
> try {
> future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
> } catch (TimeoutException e) {
> // Cancel thread and retry the same event in case of timeout
> if (future != null && !future.isDone() && !future.isCancelled()) {
> future.cancel(true);
> futures.remove(evt);
> if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
> renewalTimer.schedule(
> getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
> tokenRenewerThreadRetryInterval);
> } else {
> LOG.info(
> "Exhausted max retry attempts {} in token renewer "
> + "thread for {}",
> tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
> }
> }
> } catch (Exception e) {
> LOG.info("Problem in submitting renew tasks in token renewer "
> + "thread.", e);
> }
> }
> }
> }{code}
> A better way to avoid CPU idling is waiting for some time when the `futures`
> map is empty, and when the renewer task done or cancelled, we should remove
> the task future in `futures` map to avoid memory leak:
> {code:java}
> @Override
> public void run() {
> while (true) {
> // waiting for some time when futures map is empty
> if (futures.isEmpty()) {
> synchronized (this) {
> try {
> // waiting for tokenRenewerThreadTimeout milliseconds
> long waitingTimeMs = Math.min(1, Math.max(500,
> tokenRenewerThreadTimeout));
> LOG.info("Delegation token renewer pool is empty, waiting for {}
> ms.", waitingTimeMs);
> wait(waitingTimeMs);
> } catch (InterruptedException e) {
> LOG.warn("Delegation token renewer pool tracker waiting interrupt
> occurred.");
> Thread.currentThread().interrupt();
> }
> }
> if (futures.isEmpty()) {
> continue;
> }
> }
> for (Map.Entry> entry : futures
> .entrySet()) {
> DelegationTokenRenewerEvent evt = entry.getKey();
> Future future = entry.getValue();
> try {
> future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
> } catch (TimeoutException e) {
> // Cancel thread and retry the same event in case of timeout
> if (future != null && !future.isDone() && !future.isCancelled()) {
> future.cancel(true);
> futures.remove(evt);
> if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
> renewalTimer.schedule(
> getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
> tokenRenewerThreadRetryInterval);
> } else {
> LOG.info(
> "Exhausted max retry attempts {} in token renewe
[jira] [Commented] (YARN-11178) Avoid CPU busy idling and resource wasting in DelegationTokenRenewerPoolTracker thread
[
https://issues.apache.org/jira/browse/YARN-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683636#comment-17683636
]
ASF GitHub Bot commented on YARN-11178:
---
slfan1989 commented on code in PR #4435:
URL: https://github.com/apache/hadoop/pull/4435#discussion_r1095306416
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java:
##
@@ -996,6 +996,22 @@ public void run() {
@Override
public void run() {
while (true) {
+if (futures.isEmpty()) {
+ synchronized (this) {
+try {
+ // waiting for tokenRenewerThreadTimeout milliseconds
+ long waitingTimeMs = Math.min(1, Math.max(500,
tokenRenewerThreadTimeout));
Review Comment:
> @slfan1989 what do you suggest increasing them to?
Sorry for the late reply, I hope the configuration can be added, not
hardcoded.
> Avoid CPU busy idling and resource wasting in
> DelegationTokenRenewerPoolTracker thread
> --
>
> Key: YARN-11178
> URL: https://issues.apache.org/jira/browse/YARN-11178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, security
>Affects Versions: 3.3.1, 3.3.2, 3.3.3, 3.3.4
> Environment: Hadoop 3.3.3 with Kerberos, Ranger 2.1.0, Hive 2.3.7 and
> Spark 3.0.3
>Reporter: Lennon Chin
>Priority: Minor
> Labels: pull-request-available
> Attachments: YARN-11178.CPU idling busy 100% before optimized.png,
> YARN-11178.CPU normal after optimized.png, YARN-11178.CPU profile for idling
> busy 100% before optimized.html, YARN-11178.CPU profile for idling busy 100%
> before optimized.png, YARN-11178.CPU profile for normal after optimized.html,
> YARN-11178.CPU profile for normal after optimized.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The DelegationTokenRenewerPoolTracker thread is busy wasting CPU resource in
> empty poll iterate when there is no delegation token renewer event task in
> the futures map:
> {code:java}
> //
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.DelegationTokenRenewerPoolTracker#run
> @Override
> public void run() {
> // this while true loop is busy when the `futures` is empty
> while (true) {
> for (Map.Entry> entry : futures
> .entrySet()) {
> DelegationTokenRenewerEvent evt = entry.getKey();
> Future future = entry.getValue();
> try {
> future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
> } catch (TimeoutException e) {
> // Cancel thread and retry the same event in case of timeout
> if (future != null && !future.isDone() && !future.isCancelled()) {
> future.cancel(true);
> futures.remove(evt);
> if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
> renewalTimer.schedule(
> getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
> tokenRenewerThreadRetryInterval);
> } else {
> LOG.info(
> "Exhausted max retry attempts {} in token renewer "
> + "thread for {}",
> tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
> }
> }
> } catch (Exception e) {
> LOG.info("Problem in submitting renew tasks in token renewer "
> + "thread.", e);
> }
> }
> }
> }{code}
> A better way to avoid CPU idling is waiting for some time when the `futures`
> map is empty, and when the renewer task done or cancelled, we should remove
> the task future in `futures` map to avoid memory leak:
> {code:java}
> @Override
> public void run() {
> while (true) {
> // waiting for some time when futures map is empty
> if (futures.isEmpty()) {
> synchronized (this) {
> try {
> // waiting for tokenRenewerThreadTimeout milliseconds
> long waitingTimeMs = Math.min(1, Math.max(500,
> tokenRenewerThreadTimeout));
> LOG.info("Delegation token renewer pool is empty, waiting for {}
> ms.", waitingTimeMs);
> wait(waitingTimeMs);
> } catch (InterruptedException e) {
> LOG.warn("Delegation token renewer pool tracker waiting interrupt
> occurred.");
> Thread.currentThread().interrupt();
> }
> }
> if (futures.isEmpty()) {
> continue;
> }
> }
> for (Map.Entry> entry : futures
> .entrySet()) {
> DelegationTokenRenewerEvent evt = entry.getKey();
> Future future = entry.getValue();
> try {
> future.get(tokenRenewerThre
[jira] [Commented] (YARN-11426) Improve YARN NodeLabel Memory Display
[
https://issues.apache.org/jira/browse/YARN-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683528#comment-17683528
]
ASF GitHub Bot commented on YARN-11426:
---
cnauroth commented on code in PR #5335:
URL: https://github.com/apache/hadoop/pull/5335#discussion_r1094945678
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/api/records/TestResource.java:
##
@@ -42,4 +42,70 @@ void testCastToIntSafely() {
"Cast to Integer.MAX_VALUE if the long is greater than "
+ "Integer.MAX_VALUE");
}
+
+ @Test
+ public void testResourceFormatted() {
+// We set 10MB
+String expectedResult1 = "";
+MockResource capability1 = new MockResource();
+capability1.setMemory(10);
+capability1.setVirtualCores(1);
+assertEquals(capability1.toFormattedString(), expectedResult1);
+
+// We set 1024 MB = 1GB
+String expectedResult2 = "";
+MockResource capability2 = new MockResource();
+capability2.setMemory(1024);
+capability2.setVirtualCores(1);
+assertEquals(capability2.toFormattedString(), expectedResult2);
+
+// We set 1024 * 1024 MB = 1024 GB = 1TB
+String expectedResult3 = "";
+MockResource capability3 = new MockResource();
+capability3.setMemory(1024 * 1024);
+capability3.setVirtualCores(1);
+assertEquals(capability3.toFormattedString(), expectedResult3);
+
+// We set 1024 * 1024 * 1024 MB = 1024 * 1024 GB = 1 * 1024 TB = 1 PB
+String expectedResult4 = "";
+MockResource capability4 = new MockResource();
+capability4.setMemory(1024 * 1024 * 1024);
+capability4.setVirtualCores(1);
+assertEquals(capability4.toFormattedString(), expectedResult4);
+ }
+
+ class MockResource extends Resource {
Review Comment:
Could we switch to a Mockito mock to shorten this code?
> Improve YARN NodeLabel Memory Display
> -
>
> Key: YARN-11426
> URL: https://issues.apache.org/jira/browse/YARN-11426
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Attachments: before improvement.png, image-2023-02-01-17-55-59-037.png
>
>
> NodeLabel displays the memory information of the cluster, directly displays
> the numbers, does not display the unit information, and is not easy to
> understand.
> - before improvement
> !before improvement.png!
> * after improvement
> !image-2023-02-01-17-55-59-037.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11426) Improve YARN NodeLabel Memory Display
[
https://issues.apache.org/jira/browse/YARN-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683369#comment-17683369
]
ASF GitHub Bot commented on YARN-11426:
---
hadoop-yetus commented on PR #5335:
URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1413619060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 13s | | trunk passed |
| +1 :green_heart: | compile | 10m 28s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 8m 58s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 10s | | trunk passed |
| -1 :x: | javadoc | 0m 54s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/3/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 52s | | the patch passed |
| +1 :green_heart: | compile | 10m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 10m 24s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/3/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 2 new + 721
unchanged - 0 fixed = 723 total (was 721) |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 8m 35s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/3/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08
with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 2 new + 636
unchanged - 0 fixed = 638 total (was 636) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 33s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 3 unchanged -
0 fixed = 6 total (was 3) |
| +1 :green_heart: | mvnsite | 2m 6s | | the patch passed |
| -1 :x: | javadoc | 0m 57s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/3/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed wi
[jira] [Commented] (YARN-11426) Improve YARN NodeLabel Memory Display
[
https://issues.apache.org/jira/browse/YARN-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683332#comment-17683332
]
ASF GitHub Bot commented on YARN-11426:
---
hadoop-yetus commented on PR #5335:
URL: https://github.com/apache/hadoop/pull/5335#issuecomment-1413527028
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 30s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 9s | | trunk passed |
| +1 :green_heart: | compile | 9m 36s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 8m 30s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 18s | | trunk passed |
| -1 :x: | javadoc | 0m 53s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/2/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 45s | | the patch passed |
| +1 :green_heart: | compile | 9m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 9m 32s | | the patch passed |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 32s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 58s | | the patch passed |
| -1 :x: | javadoc | 0m 49s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/2/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 6s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 99m 26s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 30s | | The patch does not
generate ASF License warnings. |
| | | 264m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5335/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5335 |
| Optional Tests | dupname asflicense compile javac jav
[jira] [Commented] (YARN-11217) [Federation] Add dumpSchedulerLogs REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683318#comment-17683318
]
ASF GitHub Bot commented on YARN-11217:
---
hadoop-yetus commented on PR #5272:
URL: https://github.com/apache/hadoop/pull/5272#issuecomment-1413478935
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 53s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 22s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 39s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 128m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5272 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2377193a9aed 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 55d7c592ee0859c1329d79665f2629ec85d5f65a |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/17/testReport/ |
| Max. process+thread count | 614 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibr
