[
https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752897#comment-17752897
]
ASF GitHub Bot commented on HDFS-17148:
---------------------------------------
simbadzina commented on code in PR #5936:
URL: https://github.com/apache/hadoop/pull/5936#discussion_r1290501755
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/SQLDelegationTokenSecretManager.java:
##########
@@ -153,6 +163,39 @@ public synchronized TokenIdent
cancelToken(Token<TokenIdent> token,
return super.cancelToken(token, canceller);
}
+ /**
+ * Obtain a list of tokens that will be considered for cleanup, based on the
last
+ * time the token was updated in SQL. This list may include tokens that are
not
+ * expired and should not be deleted (e.g. if the token was last renewed
using a
+ * higher renewal interval).
+ * The number of results is limited to reduce performance impact. Some level
of
+ * contention is expected when multiple routers run cleanup simultaneously.
+ * @return Map of tokens that have not been updated in SQL after the token
renewal
+ * period.
+ */
+ @Override
+ protected Map<TokenIdent, DelegationTokenInformation> getTokensForCleanup() {
+ Map<TokenIdent, DelegationTokenInformation> tokens = new HashMap<>();
+ try {
+ // Query SQL for tokens that haven't been updated after
+ // the last token renewal period.
+ long maxModifiedTime = Time.now() - getTokenRenewInterval();
+ Map<byte[], byte[]> tokenInfoBytesList =
selectTokenInfos(maxModifiedTime,
+ this.maxTokenCleanupResults);
+
+ LOG.info("Found {} tokens for cleanup", tokenInfoBytesList.size());
+ for (Map.Entry<byte[], byte[]> tokenInfoBytes :
tokenInfoBytesList.entrySet()) {
+ TokenIdent tokenIdent = createTokenIdent(tokenInfoBytes.getKey());
+ DelegationTokenInformation tokenInfo =
createTokenInfo(tokenInfoBytes.getValue());
+ tokens.put(tokenIdent, tokenInfo);
+ }
+ } catch (IOException | SQLException e) {
+ LOG.error("Failed to get all tokens in SQL secret manager", e);
Review Comment:
This is not `all tokens` but a subset, filtered my maxModifiedTime.
> RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
> -----------------------------------------------------------------------
>
> Key: HDFS-17148
> URL: https://issues.apache.org/jira/browse/HDFS-17148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Hector Sandoval Chaverri
> Priority: Major
> Labels: pull-request-available
>
> The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them
> temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in
> AbstractDelegationTokenSecretManager runs periodically to cleanup any expired
> tokens from the cache, but most tokens have been evicted automatically per
> the TTL configuration. This leads to many expired tokens in the SQL database
> that should be cleaned up.
> The SQLDelegationTokenSecretManager should find expired tokens in SQL instead
> of in the memory cache when running the periodic cleanup.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
