[ 
https://issues.apache.org/jira/browse/HDDS-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17866:
----------------------------------
    Labels: ozone-hbase-perf  (was: )

> cache verified block tokens instead of decoding and HMAC-checking the token 
> on every request
> --------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17866
>                 URL: https://issues.apache.org/jira/browse/HDDS-17866
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Security
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Minor
>              Labels: ozone-hbase-perf
>
> h3. Problem
> In secure mode, \{{HddsDispatcher.validateToken}} runs the full verification 
> for every container command that needs a token (every ReadChunk and GetBlock):
> {noformat}
> TokenVerifier.verify(cmd, encodedToken)
> Token.decodeFromUrlString Base64 decode + Token.readFields
> CompositeTokenVerifier -> ShortLivedTokenVerifier.verify
> OzoneBlockTokenIdentifier.readFromByteArray BlockTokenSecretProto.parseFrom
> verifyTokenPassword
> secretKeyClient.getSecretKey(id), expiry check
> ManagedSecretKey.isValidSignature(tokenId, password)
> tokenId.getBytes() re-serializes the identifier that was just parsed
> sign(...) HMAC (javax.crypto.Mac.doFinal)
> isExpired, service check, BlockTokenVerifier access-mode check
> {noformat}
> A client reading a block sends the same encoded token on every request for 
> that block. The client encodes the token once per request 
> (\{{Token.encodeToUrlString}}), but the resulting string is identical each 
> time. So the datanode decodes, parses and HMAC-checks byte-identical tokens 
> again and again.
> h3. Evidence
> async-profiler CPU profile of a datanode serving HBase YCSB workload C 
> (gets), EC rs-3-2, secure cluster:
> ||Frame||Share of \{{HddsDispatcher.dispatch}} CPU||
> |\{{HddsDispatcher.validateToken}} |14.0%|
> |\{{Token.decodeFromUrlString}} (Base64 + readFields)|3.7%|
> |\{{OzoneBlockTokenIdentifier.readFromByteArray}} |2.5%|
> |\{{ManagedSecretKey.isValidSignature}} (HMAC; includes \{{getBytes}} 
> re-serialization 1.2%)|3.6%|
> That is about 4% of total datanode CPU, spent on requests that carry an 
> already verified token. HDDS-10054 reduced the allocation cost of this path; 
> this change removes most of the repeated work.
> h3. Proposed fix
> Add a small bounded cache of successfully verified tokens to the datanode's 
> token verifier:
> * *Key:* the exact encoded token string from the request. A hit means a 
> byte-identical token (identifier, password, kind, service) has already passed 
> the HMAC check.
> * *Value:* the parsed \{{OzoneBlockTokenIdentifier}} (or the container token 
> identifier).
> * *On a hit*, skip decode, parse and HMAC, but still run every cheap 
> per-request check:
> ** the signing secret key is still known to \{{secretKeyClient}} and not 
> expired;
> ** the token is not expired;
> ** the token's service matches the command's block or container ID (including 
> the container check from HDDS-17440);
> ** the token's access modes allow the command's access mode.
> * *On a miss*, run the full verification as today and insert the entry only 
> if it succeeds. Failures are never cached, to avoid stale negative answers 
> like HDDS-16957.
> * *Bounds:* entry count limited by a new config (for example a few thousand 
> entries), and each entry expires no later than its token's expiry.
> * Only the secure-mode verifiers are affected. \{{NoopTokenVerifier}} is 
> unchanged.
> Optional, separate: \{{ManagedSecretKey.isValidSignature(TokenIdentifier, 
> byte[])}} re-serializes the identifier with \{{getBytes()}} although 
> \{{Token.getIdentifier()}} already holds the signed bytes. Signing the raw 
> identifier bytes is equivalent only if serialization is canonical, so this 
> needs care and should not be part of the cache change.
> h3. Security considerations
> * A cache hit requires an exact match of the full encoded token, so no token 
> that has not passed the HMAC check is ever accepted.
> * Block tokens are not individually revocable today. Expiry and secret key 
> rotation are the controls, and both are checked on every hit.
> * Memory is bounded by the entry limit. An attacker can only cause misses, 
> which cost the same as today.
> h3. Testing
> * Unit tests for the cached verifier:
> ** a hit still rejects an expired token;
> ** a hit still rejects a token whose secret key was removed or expired;
> ** a hit still rejects a service mismatch or a wrong access mode;
> ** a different password with the same identifier misses and fails;
> ** verification failures are not cached;
> ** the size bound holds.
> * Re-profile the datanode under YCSB C. \{{validateToken}} should drop from 
> about 14% of dispatch CPU to close to the cost of one hash lookup.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to