Wei-Chiu Chuang created HADOOP-20046:
----------------------------------------
Summary: Avoid holding the delegation-token secret manager API
write lock during ZooKeeper renewal, cancellation, and master-key operations
Key: HADOOP-20046
URL: https://issues.apache.org/jira/browse/HADOOP-20046
Project: Hadoop Common
Issue Type: Improvement
Components: common, kms
Reporter: Wei-Chiu Chuang
h3. Problem
HADOOP-18851 introduced a read-write lock in
AbstractDelegationTokenSecretManager to allow concurrent token verification.
HADOOP-20045 addresses token creation holding that lock during ZooKeeper
operations.
Other paths still hold {{apiLock.writeLock()}} while performing synchronous
ZooKeeper operations:
|| Write-lock holder || ZooKeeper operations under the lock ||
| renewToken() | Read signing key/token information on cache misses; check
existence and update/create token |
| cancelToken() | Read token information, check existence, and delete token |
| updateCurrentKey() | Check existence and persist the new master key |
| rollMasterKey() | Check existence and persist the retiring key’s updated
expiry |
| removeExpiredKeys() | Check existence and delete expired master keys,
potentially multiple keys per lock hold |
ZooKeeper latency or retries in these paths can block unrelated requests that
need the API read lock to verify cached tokens. This affects KMS and other
consumers of the shared ZKDelegationTokenSecretManager, including HDFS Router.
h3. Proposed improvement
Move ZooKeeper operations outside the global API write-lock sections. Retain
short critical sections for inspecting or publishing local state, with explicit
coordination to preserve operation ordering.
The design must ensure:
* Renewal cannot restore a token after cancellation or overwrite a newer
renewal.
* Cache publication and storage-failure handling remain consistent with the
operation’s outcome.
* A new master key is persisted before it becomes available for signing tokens.
* Retiring-key updates and expired-key deletion remain correctly ordered with
rotation and issuance.
* Existing storage-hook behavior for non-ZooKeeper implementations is preserved.
Per-token coordination, a separate key-rotation coordinator, and versioned
ZooKeeper operations are potential mechanisms to evaluate. Simply releasing the
global lock around existing storage calls is insufficient.
h3. Validation
Add deterministic tests that pause each relevant ZooKeeper operation and
confirm that verification of an unrelated cached token continues. Add
concurrency and failure tests covering renewal versus cancellation, concurrent
renewals, key rotation versus issuance, interrupted operations, and persistence
failures.
Extend the KMS HTTP benchmark from HADOOP-20045 to measure renewal and
cancellation contention. Exercise key rotation and expired-key cleanup
separately. Compare throughput and verification tail latency against the
original implementation, using real ZooKeeper operations without injected delay
for performance measurements.
h3. Scope
This issue specifically tracks ZooKeeper operations executed while holding
{{apiLock.writeLock()}}. Read-lock cache misses, Router cache rebuilding, and
CPU-only cleanup scans are outside its scope.
The lock paths have been identified through source inspection; their individual
performance impact has not yet been benchmarked.
h3. Related issues
* [HADOOP-18851|https://issues.apache.org/jira/browse/HADOOP-18851]
* [HADOOP-20045|https://issues.apache.org/jira/browse/HADOOP-20045]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
