[
https://issues.apache.org/jira/browse/HADOOP-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-20046:
-------------------------------------
Description:
h3. Problem
HADOOP-18851 introduced a read-write lock in
AbstractDelegationTokenSecretManager to allow concurrent token verification.
HADOOP-20045 addresses token creation holding that lock during ZooKeeper
operations.
Other paths still hold {{apiLock.writeLock()}} while performing synchronous
ZooKeeper operations:
|| Write-lock holder || ZooKeeper operations under the lock ||
| renewToken() | Read signing key/token information on cache misses; check
existence and update/create token |
| cancelToken() | Read token information, check existence, and delete token |
| updateCurrentKey() | Check existence and persist the new master key |
| rollMasterKey() | Check existence and persist the retiring key’s updated
expiry |
| removeExpiredKeys() | Check existence and delete expired master keys,
potentially multiple keys per lock hold |
ZooKeeper latency or retries in these paths can block unrelated requests that
need the API read lock to verify cached tokens. This affects KMS and other
consumers of the shared ZKDelegationTokenSecretManager, including HDFS Router.
h3. Proposed improvement
Move ZooKeeper operations outside the global API write-lock sections. Retain
short critical sections for inspecting or publishing local state, with explicit
coordination to preserve operation ordering.
The design must ensure:
* Renewal cannot restore a token after cancellation or overwrite a newer
renewal.
* Cache publication and storage-failure handling remain consistent with the
operation’s outcome.
* A new master key is persisted before it becomes available for signing tokens.
* Retiring-key updates and expired-key deletion remain correctly ordered with
rotation and issuance.
* Existing storage-hook behavior for non-ZooKeeper implementations is preserved.
Per-token coordination, a separate key-rotation coordinator, and versioned
ZooKeeper operations are potential mechanisms to evaluate. Simply releasing the
global lock around existing storage calls is insufficient.
h3. Validation
Add deterministic tests that pause each relevant ZooKeeper operation and
confirm that verification of an unrelated cached token continues. Add
concurrency and failure tests covering renewal versus cancellation, concurrent
renewals, key rotation versus issuance, interrupted operations, and persistence
failures.
Extend the KMS HTTP benchmark from HADOOP-20045 to measure renewal and
cancellation contention. Exercise key rotation and expired-key cleanup
separately. Compare throughput and verification tail latency against the
original implementation, using real ZooKeeper operations without injected delay
for performance measurements.
h3. Scope
This issue specifically tracks ZooKeeper operations executed while holding
{{apiLock.writeLock()}}. Read-lock cache misses, Router cache rebuilding, and
CPU-only cleanup scans are outside its scope.
Implementation and real-ZooKeeper benchmarks are now available. The attached
design describes token lock stripes, versioned renewal/deletion, durable
master-key publication, and compatibility with non-ZooKeeper stores.
Measurements include renewal/cancellation HTTP workloads and 24 validated
background-maintenance cases (four scenarios, two builds, three repetitions).
See the implementation/benchmark comment and attached report, charts, and raw
results for methods, validation, and limitations.
h3. Related issues
* [HADOOP-18851|https://issues.apache.org/jira/browse/HADOOP-18851]
* [HADOOP-20045|https://issues.apache.org/jira/browse/HADOOP-20045]
was:
h3. Problem
HADOOP-18851 introduced a read-write lock in
AbstractDelegationTokenSecretManager to allow concurrent token verification.
HADOOP-20045 addresses token creation holding that lock during ZooKeeper
operations.
Other paths still hold {{apiLock.writeLock()}} while performing synchronous
ZooKeeper operations:
|| Write-lock holder || ZooKeeper operations under the lock ||
| renewToken() | Read signing key/token information on cache misses; check
existence and update/create token |
| cancelToken() | Read token information, check existence, and delete token |
| updateCurrentKey() | Check existence and persist the new master key |
| rollMasterKey() | Check existence and persist the retiring key’s updated
expiry |
| removeExpiredKeys() | Check existence and delete expired master keys,
potentially multiple keys per lock hold |
ZooKeeper latency or retries in these paths can block unrelated requests that
need the API read lock to verify cached tokens. This affects KMS and other
consumers of the shared ZKDelegationTokenSecretManager, including HDFS Router.
h3. Proposed improvement
Move ZooKeeper operations outside the global API write-lock sections. Retain
short critical sections for inspecting or publishing local state, with explicit
coordination to preserve operation ordering.
The design must ensure:
* Renewal cannot restore a token after cancellation or overwrite a newer
renewal.
* Cache publication and storage-failure handling remain consistent with the
operation’s outcome.
* A new master key is persisted before it becomes available for signing tokens.
* Retiring-key updates and expired-key deletion remain correctly ordered with
rotation and issuance.
* Existing storage-hook behavior for non-ZooKeeper implementations is preserved.
Per-token coordination, a separate key-rotation coordinator, and versioned
ZooKeeper operations are potential mechanisms to evaluate. Simply releasing the
global lock around existing storage calls is insufficient.
h3. Validation
Add deterministic tests that pause each relevant ZooKeeper operation and
confirm that verification of an unrelated cached token continues. Add
concurrency and failure tests covering renewal versus cancellation, concurrent
renewals, key rotation versus issuance, interrupted operations, and persistence
failures.
Extend the KMS HTTP benchmark from HADOOP-20045 to measure renewal and
cancellation contention. Exercise key rotation and expired-key cleanup
separately. Compare throughput and verification tail latency against the
original implementation, using real ZooKeeper operations without injected delay
for performance measurements.
h3. Scope
This issue specifically tracks ZooKeeper operations executed while holding
{{apiLock.writeLock()}}. Read-lock cache misses, Router cache rebuilding, and
CPU-only cleanup scans are outside its scope.
The lock paths have been identified through source inspection; their individual
performance impact has not yet been benchmarked.
h3. Related issues
* [HADOOP-18851|https://issues.apache.org/jira/browse/HADOOP-18851]
* [HADOOP-20045|https://issues.apache.org/jira/browse/HADOOP-20045]
> Avoid holding the delegation-token secret manager API write lock during
> ZooKeeper renewal, cancellation, and master-key operations
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-20046
> URL: https://issues.apache.org/jira/browse/HADOOP-20046
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments: HADOOP-20046-expired-key-stall.png,
> HADOOP-20046-maintenance-benchmark.md,
> HADOOP-20046-maintenance-comparison.png,
> HADOOP-20046-maintenance-results.json, HADOOP-20046-write-locks.md
>
>
> h3. Problem
> HADOOP-18851 introduced a read-write lock in
> AbstractDelegationTokenSecretManager to allow concurrent token verification.
> HADOOP-20045 addresses token creation holding that lock during ZooKeeper
> operations.
> Other paths still hold {{apiLock.writeLock()}} while performing synchronous
> ZooKeeper operations:
> || Write-lock holder || ZooKeeper operations under the lock ||
> | renewToken() | Read signing key/token information on cache misses; check
> existence and update/create token |
> | cancelToken() | Read token information, check existence, and delete token |
> | updateCurrentKey() | Check existence and persist the new master key |
> | rollMasterKey() | Check existence and persist the retiring key’s updated
> expiry |
> | removeExpiredKeys() | Check existence and delete expired master keys,
> potentially multiple keys per lock hold |
> ZooKeeper latency or retries in these paths can block unrelated requests that
> need the API read lock to verify cached tokens. This affects KMS and other
> consumers of the shared ZKDelegationTokenSecretManager, including HDFS Router.
> h3. Proposed improvement
> Move ZooKeeper operations outside the global API write-lock sections. Retain
> short critical sections for inspecting or publishing local state, with
> explicit coordination to preserve operation ordering.
> The design must ensure:
> * Renewal cannot restore a token after cancellation or overwrite a newer
> renewal.
> * Cache publication and storage-failure handling remain consistent with the
> operation’s outcome.
> * A new master key is persisted before it becomes available for signing
> tokens.
> * Retiring-key updates and expired-key deletion remain correctly ordered with
> rotation and issuance.
> * Existing storage-hook behavior for non-ZooKeeper implementations is
> preserved.
> Per-token coordination, a separate key-rotation coordinator, and versioned
> ZooKeeper operations are potential mechanisms to evaluate. Simply releasing
> the global lock around existing storage calls is insufficient.
> h3. Validation
> Add deterministic tests that pause each relevant ZooKeeper operation and
> confirm that verification of an unrelated cached token continues. Add
> concurrency and failure tests covering renewal versus cancellation,
> concurrent renewals, key rotation versus issuance, interrupted operations,
> and persistence failures.
> Extend the KMS HTTP benchmark from HADOOP-20045 to measure renewal and
> cancellation contention. Exercise key rotation and expired-key cleanup
> separately. Compare throughput and verification tail latency against the
> original implementation, using real ZooKeeper operations without injected
> delay for performance measurements.
> h3. Scope
> This issue specifically tracks ZooKeeper operations executed while holding
> {{apiLock.writeLock()}}. Read-lock cache misses, Router cache rebuilding, and
> CPU-only cleanup scans are outside its scope.
> Implementation and real-ZooKeeper benchmarks are now available. The attached
> design describes token lock stripes, versioned renewal/deletion, durable
> master-key publication, and compatibility with non-ZooKeeper stores.
> Measurements include renewal/cancellation HTTP workloads and 24 validated
> background-maintenance cases (four scenarios, two builds, three repetitions).
> See the implementation/benchmark comment and attached report, charts, and raw
> results for methods, validation, and limitations.
> h3. Related issues
> * [HADOOP-18851|https://issues.apache.org/jira/browse/HADOOP-18851]
> * [HADOOP-20045|https://issues.apache.org/jira/browse/HADOOP-20045]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
