[
https://issues.apache.org/jira/browse/HADOOP-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127621#comment-18127621
]
ASF GitHub Bot commented on HADOOP-19966:
-----------------------------------------
joseluisll commented on code in PR #8688:
URL: https://github.com/apache/hadoop/pull/8688#discussion_r4288047632
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:
##########
@@ -386,6 +433,24 @@ private void loadFromZKCache(final boolean isTokenCache) {
LOG.info("Loaded {} cache.", cacheName);
}
+ private void awaitCacheInitialized(CountDownLatch initialized,
+ String cacheName) throws IOException {
+ try {
+ if (cacheInitTimeoutMs > 0) {
+ if (!initialized.await(cacheInitTimeoutMs, TimeUnit.MILLISECONDS)) {
+ throw new IOException("Timed out after " + cacheInitTimeoutMs
+ + " ms waiting for " + cacheName + " cache initialization");
+ }
+ } else {
+ initialized.await();
Review Comment:
**Ask:** log a WARN periodically while waiting here, for example by calling
`await(timeout)` in a loop and logging on each pass.
**Why:** if ZooKeeper becomes unreachable, startup now hangs here with
nothing in the logs.
**Optional:** add a code comment noting that in Curator 5.2.0 a failed read
during the first load still counts as done, so the latch can release with the
cache only partly loaded.
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java:
##########
@@ -301,9 +334,11 @@ public void startThreads() throws IOException {
}
})
.forDeletes(childData -> processKeyRemoved(childData.getPath()))
+ .forInitialized(keyCacheInitialized::countDown)
Review Comment:
**Ask (nit, optional):** add `.afterInitialized()` to both listener builders.
**Why:** without it, these listeners also process every existing node during
the first load, and `loadFromZKCache` then decodes them all again. The latch
still works with `.afterInitialized()`.
> ZKDelegationTokenSecretManager startup cache load races with CuratorCache
> initialization
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-19966
> URL: https://issues.apache.org/jira/browse/HADOOP-19966
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Cheng Pan
> Priority: Major
> Labels: pull-request-available
>
> Since HADOOP-17835 (3.4.0), ZKDelegationTokenSecretManager starts its
> CuratorCache asynchronously and reads it immediately, so loadFromZKCache runs
> against an empty or partial cache. Token verify/renew/cancel are unaffected
> because they fall back to ZooKeeper, and the cache listener fills the
> in-memory map a few seconds later. The lasting effects are:
> * token owner stats (Router getTopTokenRealOwners) are computed from a
> partial map and never corrected for pre-restart tokens;
> * the root container znode, which carries Curator's default data (the local
> address), is parsed as a key/token on every startup and logs an ERROR;
> * testNodesLoadedAfterRestart is flaky in CI.
> Fix: wait for CuratorCache initialization (forInitialized) before loading
> each cache, skip the root container znodes when processing cache events and
> when streaming the cache, create the root znodes with explicit empty data,
> and release the caches, counters and Curator client when startThreads() fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
