[
https://issues.apache.org/jira/browse/HADOOP-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-20046:
-------------------------------------
Attachment: HADOOP-20046-maintenance-comparison.png
HADOOP-20046-expired-key-stall.png
HADOOP-20046-maintenance-benchmark.md
HADOOP-20046-maintenance-results.json
HADOOP-20046-write-locks.md
> Avoid holding the delegation-token secret manager API write lock during
> ZooKeeper renewal, cancellation, and master-key operations
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20046
> URL: https://issues.apache.org/jira/browse/HADOOP-20046
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments: HADOOP-20046-expired-key-stall.png,
> HADOOP-20046-maintenance-benchmark.md,
> HADOOP-20046-maintenance-comparison.png,
> HADOOP-20046-maintenance-results.json, HADOOP-20046-write-locks.md
>
>
> h3. Problem
> HADOOP-18851 introduced a read-write lock in
> AbstractDelegationTokenSecretManager to allow concurrent token verification.
> HADOOP-20045 addresses token creation holding that lock during ZooKeeper
> operations.
> Other paths still hold {{apiLock.writeLock()}} while performing synchronous
> ZooKeeper operations:
> || Write-lock holder || ZooKeeper operations under the lock ||
> | renewToken() | Read signing key/token information on cache misses; check
> existence and update/create token |
> | cancelToken() | Read token information, check existence, and delete token |
> | updateCurrentKey() | Check existence and persist the new master key |
> | rollMasterKey() | Check existence and persist the retiring key’s updated
> expiry |
> | removeExpiredKeys() | Check existence and delete expired master keys,
> potentially multiple keys per lock hold |
> ZooKeeper latency or retries in these paths can block unrelated requests that
> need the API read lock to verify cached tokens. This affects KMS and other
> consumers of the shared ZKDelegationTokenSecretManager, including HDFS Router.
> h3. Proposed improvement
> Move ZooKeeper operations outside the global API write-lock sections. Retain
> short critical sections for inspecting or publishing local state, with
> explicit coordination to preserve operation ordering.
> The design must ensure:
> * Renewal cannot restore a token after cancellation or overwrite a newer
> renewal.
> * Cache publication and storage-failure handling remain consistent with the
> operation’s outcome.
> * A new master key is persisted before it becomes available for signing
> tokens.
> * Retiring-key updates and expired-key deletion remain correctly ordered with
> rotation and issuance.
> * Existing storage-hook behavior for non-ZooKeeper implementations is
> preserved.
> Per-token coordination, a separate key-rotation coordinator, and versioned
> ZooKeeper operations are potential mechanisms to evaluate. Simply releasing
> the global lock around existing storage calls is insufficient.
> h3. Validation
> Add deterministic tests that pause each relevant ZooKeeper operation and
> confirm that verification of an unrelated cached token continues. Add
> concurrency and failure tests covering renewal versus cancellation,
> concurrent renewals, key rotation versus issuance, interrupted operations,
> and persistence failures.
> Extend the KMS HTTP benchmark from HADOOP-20045 to measure renewal and
> cancellation contention. Exercise key rotation and expired-key cleanup
> separately. Compare throughput and verification tail latency against the
> original implementation, using real ZooKeeper operations without injected
> delay for performance measurements.
> h3. Scope
> This issue specifically tracks ZooKeeper operations executed while holding
> {{apiLock.writeLock()}}. Read-lock cache misses, Router cache rebuilding, and
> CPU-only cleanup scans are outside its scope.
> The lock paths have been identified through source inspection; their
> individual performance impact has not yet been benchmarked.
> h3. Related issues
> * [HADOOP-18851|https://issues.apache.org/jira/browse/HADOOP-18851]
> * [HADOOP-20045|https://issues.apache.org/jira/browse/HADOOP-20045]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
