[
https://issues.apache.org/jira/browse/HDDS-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17866:
----------------------------------
Labels: ozone-hbase-perf (was: )
> cache verified block tokens instead of decoding and HMAC-checking the token
> on every request
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-17866
> URL: https://issues.apache.org/jira/browse/HDDS-17866
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Security
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Minor
> Labels: ozone-hbase-perf
>
> h3. Problem
> In secure mode, \{{HddsDispatcher.validateToken}} runs the full verification
> for every container command that needs a token (every ReadChunk and GetBlock):
> {noformat}
> TokenVerifier.verify(cmd, encodedToken)
> Token.decodeFromUrlString Base64 decode + Token.readFields
> CompositeTokenVerifier -> ShortLivedTokenVerifier.verify
> OzoneBlockTokenIdentifier.readFromByteArray BlockTokenSecretProto.parseFrom
> verifyTokenPassword
> secretKeyClient.getSecretKey(id), expiry check
> ManagedSecretKey.isValidSignature(tokenId, password)
> tokenId.getBytes() re-serializes the identifier that was just parsed
> sign(...) HMAC (javax.crypto.Mac.doFinal)
> isExpired, service check, BlockTokenVerifier access-mode check
> {noformat}
> A client reading a block sends the same encoded token on every request for
> that block. The client encodes the token once per request
> (\{{Token.encodeToUrlString}}), but the resulting string is identical each
> time. So the datanode decodes, parses and HMAC-checks byte-identical tokens
> again and again.
> h3. Evidence
> async-profiler CPU profile of a datanode serving HBase YCSB workload C
> (gets), EC rs-3-2, secure cluster:
> ||Frame||Share of \{{HddsDispatcher.dispatch}} CPU||
> |\{{HddsDispatcher.validateToken}} |14.0%|
> |\{{Token.decodeFromUrlString}} (Base64 + readFields)|3.7%|
> |\{{OzoneBlockTokenIdentifier.readFromByteArray}} |2.5%|
> |\{{ManagedSecretKey.isValidSignature}} (HMAC; includes \{{getBytes}}
> re-serialization 1.2%)|3.6%|
> That is about 4% of total datanode CPU, spent on requests that carry an
> already verified token. HDDS-10054 reduced the allocation cost of this path;
> this change removes most of the repeated work.
> h3. Proposed fix
> Add a small bounded cache of successfully verified tokens to the datanode's
> token verifier:
> * *Key:* the exact encoded token string from the request. A hit means a
> byte-identical token (identifier, password, kind, service) has already passed
> the HMAC check.
> * *Value:* the parsed \{{OzoneBlockTokenIdentifier}} (or the container token
> identifier).
> * *On a hit*, skip decode, parse and HMAC, but still run every cheap
> per-request check:
> ** the signing secret key is still known to \{{secretKeyClient}} and not
> expired;
> ** the token is not expired;
> ** the token's service matches the command's block or container ID (including
> the container check from HDDS-17440);
> ** the token's access modes allow the command's access mode.
> * *On a miss*, run the full verification as today and insert the entry only
> if it succeeds. Failures are never cached, to avoid stale negative answers
> like HDDS-16957.
> * *Bounds:* entry count limited by a new config (for example a few thousand
> entries), and each entry expires no later than its token's expiry.
> * Only the secure-mode verifiers are affected. \{{NoopTokenVerifier}} is
> unchanged.
> Optional, separate: \{{ManagedSecretKey.isValidSignature(TokenIdentifier,
> byte[])}} re-serializes the identifier with \{{getBytes()}} although
> \{{Token.getIdentifier()}} already holds the signed bytes. Signing the raw
> identifier bytes is equivalent only if serialization is canonical, so this
> needs care and should not be part of the cache change.
> h3. Security considerations
> * A cache hit requires an exact match of the full encoded token, so no token
> that has not passed the HMAC check is ever accepted.
> * Block tokens are not individually revocable today. Expiry and secret key
> rotation are the controls, and both are checked on every hit.
> * Memory is bounded by the entry limit. An attacker can only cause misses,
> which cost the same as today.
> h3. Testing
> * Unit tests for the cached verifier:
> ** a hit still rejects an expired token;
> ** a hit still rejects a token whose secret key was removed or expired;
> ** a hit still rejects a service mismatch or a wrong access mode;
> ** a different password with the same identifier misses and fails;
> ** verification failures are not cached;
> ** the size bound holds.
> * Re-profile the datanode under YCSB C. \{{validateToken}} should drop from
> about 14% of dispatch CPU to close to the cost of one hash lookup.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]