LennonChin opened a new pull request, #4435:
URL: https://github.com/apache/hadoop/pull/4435
### Description of PR
The DelegationTokenRenewerPoolTracker thread is busy wasting CPU resource in
empty poll iterate when there is no delegation token renewer event task in the
futures map:
```java
//
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.DelegationTokenRenewerPoolTracker#run
@Override
public void run() {
// this while true loop is busy when the `futures` is empty
while (true) {
for (Map.Entry<DelegationTokenRenewerEvent, Future<?>> entry : futures
.entrySet()) {
DelegationTokenRenewerEvent evt = entry.getKey();
Future<?> future = entry.getValue();
try {
future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
// Cancel thread and retry the same event in case of timeout
if (future != null && !future.isDone() && !future.isCancelled()) {
future.cancel(true);
futures.remove(evt);
if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
renewalTimer.schedule(
getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
tokenRenewerThreadRetryInterval);
} else {
LOG.info(
"Exhausted max retry attempts {} in token renewer "
+ "thread for {}",
tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
}
}
} catch (Exception e) {
LOG.info("Problem in submitting renew tasks in token renewer "
+ "thread.", e);
}
}
}
}
```
A better way to avoid CPU idling is waiting for some time when the `futures`
map is empty, and when the renewer task done or cancelled, we should remove the
task future in `futures` map to avoid memory leak:
```java
@Override
public void run() {
while (true) {
// waiting for some time when futures map is empty
if (futures.isEmpty()) {
synchronized (this) {
try {
// waiting for tokenRenewerThreadTimeout milliseconds
long waitingTimeMs = Math.min(10000, Math.max(500,
tokenRenewerThreadTimeout));
LOG.info("Delegation token renewer pool is empty, waiting for {}
ms.", waitingTimeMs);
wait(waitingTimeMs);
} catch (InterruptedException e) {
LOG.warn("Delegation token renewer pool tracker waiting interrupt
occurred.");
Thread.currentThread().interrupt();
}
}
if (futures.isEmpty()) {
continue;
}
}
for (Map.Entry<DelegationTokenRenewerEvent, Future<?>> entry : futures
.entrySet()) {
DelegationTokenRenewerEvent evt = entry.getKey();
Future<?> future = entry.getValue();
try {
future.get(tokenRenewerThreadTimeout, TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
// Cancel thread and retry the same event in case of timeout
if (future != null && !future.isDone() && !future.isCancelled()) {
future.cancel(true);
futures.remove(evt);
if (evt.getAttempt() < tokenRenewerThreadRetryMaxAttempts) {
renewalTimer.schedule(
getTimerTask((AbstractDelegationTokenRenewerAppEvent) evt),
tokenRenewerThreadRetryInterval);
} else {
LOG.info(
"Exhausted max retry attempts {} in token renewer "
+ "thread for {}",
tokenRenewerThreadRetryMaxAttempts, evt.getApplicationId());
}
}
} catch (Exception e) {
LOG.info("Problem in submitting renew tasks in token renewer "
+ "thread.", e);
}
// remove done and cancelled task
if (future.isDone() || future.isCancelled()) {
try {
futures.remove(evt);
LOG.info("Removed done or cancelled renew tasks of {} in token
renewer thread.", evt.getApplicationId());
} catch (Exception e) {
LOG.warn("Problem in removing done or cancelled renew tasks in
token renewer thread.", e);
}
}
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
