[
https://issues.apache.org/jira/browse/YARN-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709969#comment-17709969
]
ASF GitHub Bot commented on YARN-11178:
---------------------------------------
hadoop-yetus commented on PR #4435:
URL: https://github.com/apache/hadoop/pull/4435#issuecomment-1501121315
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 35m 39s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 33s | | trunk passed |
| +1 :green_heart: | compile | 8m 57s | | trunk passed |
| +1 :green_heart: | checkstyle | 1m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 1s | | trunk passed |
| +1 :green_heart: | spotbugs | 6m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 35s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | | the patch passed |
| +1 :green_heart: | compile | 8m 17s | | the patch passed |
| +1 :green_heart: | javac | 8m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 41s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 7s | | the patch passed |
| +1 :green_heart: | xmllint | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed |
| +1 :green_heart: | spotbugs | 6m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 40s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 16s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 50s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 100m 27s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 278m 47s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4435 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 18558f9ceeee 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 84d4d555f62e29fdc19947869d02ffd33b9ac4c5 |
| Default Java | Red Hat, Inc.-1.8.0_362-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/testReport/ |
| Max. process+thread count | 943 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4435/5/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 xmllint=20901 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Avoid CPU busy idling and resource wasting in
> DelegationTokenRenewerPoolTracker thread
> --------------------------------------------------------------------------------------
>
> Key: YARN-11178
> URL: https://issues.apache.org/jira/browse/YARN-11178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, security
> Affects Versions: 3.3.1, 3.3.2, 3.3.3, 3.3.4
> Environment: Hadoop 3.3.3 with Kerberos, Ranger 2.1.0, Hive 2.3.7 and
> Spark 3.0.3
> Reporter: Lennon Chin
> Priority: Minor
> Labels: pull-request-available
> Attachments: YARN-11178.CPU idling busy 100% before optimized.png,
> YARN-11178.CPU normal after optimized.png, YARN-11178.CPU profile for idling
> busy 100% before optimized.html, YARN-11178.CPU profile for idling busy 100%
> before optimized.png, YARN-11178.CPU profile for normal after optimized.html,
> YARN-11178.CPU profile for normal after optimized.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The DelegationTokenRenewerPoolTracker thread is busy wasting CPU resource in
> empty poll iterate when there is no delegation token renewer event task in
> the futures map:
> {code:java}
> //
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.DelegationTokenRenewerPoolTracker#run
> @Override
> public void run() {
> // this while true loop is busy when the `futures` is empty
> while (true) {
> for (Map.Entry<DelegationTokenRenewerEvent, Future<?>> entry : futures
> .entrySet()) {
> DelegationTokenRenewerEvent evt = entry.getKey();
> Future<?> future = entry.getValue();
> try {
> future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
> } catch (TimeoutException e) {
> // Cancel thread and retry the same event in case of timeout
> if (future != null && !future.isDone() && !future.isCancelled()) {
> future.cancel(true);
> futures.remove(evt);
> if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
> renewalTimer.schedule(
> getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
> tokenRenewerThreadRetryInterval);
> } else {
> LOG.info(
> "Exhausted max retry attempts {} in token renewer "
> + "thread for {}",
> tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
> }
> }
> } catch (Exception e) {
> LOG.info("Problem in submitting renew tasks in token renewer "
> + "thread.", e);
> }
> }
> }
> }{code}
> A better way to avoid CPU idling is waiting for some time when the `futures`
> map is empty, and when the renewer task done or cancelled, we should remove
> the task future in `futures` map to avoid memory leak:
> {code:java}
> @Override
> public void run() {
> while (true) {
> // waiting for some time when futures map is empty
> if (futures.isEmpty()) {
> synchronized (this) {
> try {
> // waiting for tokenRenewerThreadTimeout milliseconds
> long waitingTimeMs = Math.min(10000, Math.max(500,
> tokenRenewerThreadTimeout));
> LOG.info("Delegation token renewer pool is empty, waiting for {}
> ms.", waitingTimeMs);
> wait(waitingTimeMs);
> } catch (InterruptedException e) {
> LOG.warn("Delegation token renewer pool tracker waiting interrupt
> occurred.");
> Thread.currentThread().interrupt();
> }
> }
> if (futures.isEmpty()) {
> continue;
> }
> }
> for (Map.Entry<DelegationTokenRenewerEvent, Future<?>> entry : futures
> .entrySet()) {
> DelegationTokenRenewerEvent evt = entry.getKey();
> Future<?> future = entry.getValue();
> try {
> future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
> } catch (TimeoutException e) {
> // Cancel thread and retry the same event in case of timeout
> if (future != null && !future.isDone() && !future.isCancelled()) {
> future.cancel(true);
> futures.remove(evt);
> if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
> renewalTimer.schedule(
> getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
> tokenRenewerThreadRetryInterval);
> } else {
> LOG.info(
> "Exhausted max retry attempts {} in token renewer "
> + "thread for {}",
> tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
> }
> }
> } catch (Exception e) {
> LOG.info("Problem in submitting renew tasks in token renewer "
> + "thread.", e);
> }
> // remove done and cancelled task
> if (future.isDone() || future.isCancelled()) {
> try {
> futures.remove(evt);
> LOG.info("Removed done or cancelled renew tasks of {} in token
> renewer thread.", evt.getApplicationId());
> } catch (Exception e) {
> LOG.warn("Problem in removing done or cancelled renew tasks in
> token renewer thread.", e);
> }
> }
> }
> }
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
