[
https://issues.apache.org/jira/browse/HADOOP-19966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated HADOOP-19966:
-------------------------------
Description:
Since HADOOP-17835 (3.4.0), ZKDelegationTokenSecretManager starts its
CuratorCache asynchronously and reads it immediately, so loadFromZKCache runs
against an empty or partial cache. Token verify/renew/cancel are unaffected
because they fall back to ZooKeeper, and the cache listener fills the in-memory
map a few seconds later. The lasting effects are:
* token owner stats (Router getTopTokenRealOwners) are computed from a partial
map and never corrected for pre-restart tokens;
* the root container znode, which carries Curator's default data (the local
address), is parsed as a key/token on every startup and logs an ERROR;
* testNodesLoadedAfterRestart is flaky in CI.
Fix: wait for CuratorCache initialization (forInitialized) before loading each
cache, skip the root container znodes when processing cache events and when
streaming the cache, create the root znodes with explicit empty data, and
release the caches, counters and Curator client when startThreads() fails.
> ZKDelegationTokenSecretManager may fail to load tokens and keys from
> ZooKeeper on startup
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-19966
> URL: https://issues.apache.org/jira/browse/HADOOP-19966
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Cheng Pan
> Priority: Major
> Labels: pull-request-available
>
> Since HADOOP-17835 (3.4.0), ZKDelegationTokenSecretManager starts its
> CuratorCache asynchronously and reads it immediately, so loadFromZKCache runs
> against an empty or partial cache. Token verify/renew/cancel are unaffected
> because they fall back to ZooKeeper, and the cache listener fills the
> in-memory map a few seconds later. The lasting effects are:
> * token owner stats (Router getTopTokenRealOwners) are computed from a
> partial map and never corrected for pre-restart tokens;
> * the root container znode, which carries Curator's default data (the local
> address), is parsed as a key/token on every startup and logs an ERROR;
> * testNodesLoadedAfterRestart is flaky in CI.
> Fix: wait for CuratorCache initialization (forInitialized) before loading
> each cache, skip the root container znodes when processing cache events and
> when streaming the cache, create the root znodes with explicit empty data,
> and release the caches, counters and Curator client when startThreads() fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
