Siyao Meng created HDDS-16962:
---------------------------------
Summary: Secret key verifier fetches every key id it does not hold
from SCM on the verifying thread; refresh the key set or bound the lookup
Key: HDDS-16962
URL: https://issues.apache.org/jira/browse/HDDS-16962
Project: Apache Ozone
Issue Type: Improvement
Reporter: Siyao Meng
h3. Current behaviour
{{ShortLivedTokenVerifier.verifyTokenPassword}} asks
{{SecretKeyVerifierClient.getSecretKey}} for the key named by the token's
secret key id and then checks the token's signature with it.
{{DefaultSecretKeyVerifierClient}} keeps a cache sized for the valid keys and,
on a miss, fetches that one key with a synchronous
{{SecretKeyProtocol.getSecretKey}} call to SCM, made on the thread that is
verifying the token. {{SecretKeyProtocol.getAllSecretKeys}}, which SCM already
serves, is not used by the verifier client. The same client class is used by
the OM token verifiers.
h3. Proposed improvement
Two directions, either of which a maintainer could take.
# Keep the verifier's key set locally refreshed through
{{SecretKeyProtocol.getAllSecretKeys}} at a bounded rate and answer lookups
from that set, so that an id which is not in the set is answered locally and no
per id SCM call is made. Trade off for a healthy cluster: a key that SCM
created after the last refresh is not known until the next refresh, so a token
signed with a brand new key would be rejected during that window. The on demand
fetch by id exists to avoid exactly that, so the refresh interval would need to
stay well below {{hdds.secret.key.rotate.duration}}, or a single fetch by id
could be kept as a fallback for ids that are newer than the last refresh.
# Bound how long one lookup may take on the verifying thread, for example a
short wait or a small dedicated pool for the SCM call, and treat a lookup that
does not complete within the bound as "key not available", so the token is
rejected and the thread continues. Trade off for a healthy cluster: nothing
changes while SCM answers promptly; during an SCM failover a lookup for a key
that is not cached yet could be rejected instead of completing after the
failover, and the bound value is a configuration choice that needs an entry in
{{ozone-default.xml}}.
The first direction removes the per id call altogether; the second keeps
today's fetch model and only limits a single lookup. They can also be combined.
h3. Notes
HDDS-16957 touches the same class: it changes what the client keeps when SCM
answers that it has no such key, and is independent of this issue. This issue
has no patch and no standalone reproduction attached; it is a proposal for a
maintainer to pick up.
Found by code review of the SCM secret key rotation and distribution paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16957 (same class, different change), HDDS-7735 and HDDS-7830
(introduced the fetch by id and {{getAllSecretKeys}}). Generated with Specula
(Claude Opus 5.5, Claude Fable 5.1).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
