Siyao Meng created HDDS-16959:
---------------------------------
Summary: OM drops persisted delegation tokens from memory when the
secret key fetch from SCM fails during token load
Key: HDDS-16959
URL: https://issues.apache.org/jira/browse/HDDS-16959
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch,
TestBugCR4OmRestartDuringScmKeyInit.java
h3. Mechanism
The OM DB stores only the identifier and the renew date of a delegation token.
When {{OzoneDelegationTokenSecretManager}} is created, {{loadTokenSecretState}}
calls {{addPersistedDelegationToken}} for every stored token, which fetches the
token's secret key from SCM with {{SecretKeyClient.getSecretKey}} and
recomputes the password. {{loadTokenSecretState}} catches every exception from
that call, logs "exception while loading delegation token from DB... ignored to
continue startup" and goes on to the next token.
HDDS-13234 added that catch for a secret key that SCM has already removed, in
which case {{getSecretKey}} returns null and the token can never be verified
again. A fetch that fails ({{SCMSecurityException}} "Error fetching secret key
...") ends in the same catch. Before HDDS-13234 the same failure aborted the OM
start. That change is in 2.1.0 and later releases (its fix versions are 2.0.1
and 2.1.0). The token is then missing from {{currentTokens}} although it is
unexpired, still in the DB, and its key still exists on SCM. Nothing loads it
later, so the OM rejects it until the OM process is restarted.
Two things make the failed fetch reachable during a normal start:
* The token load runs in the {{OzoneManager}} constructor
({{instantiateServices}}), before {{startSecretManager}} starts the signer
client. Only the signer client's initial fetch waits for SCM to leave
{{SECRET_KEY_NOT_INITIALIZED}}.
* {{SecretKeyProtocolClientSideTranslatorPB.handleError}} turns the error
status into an exception after the RPC has returned, so the SCM failover retry
policy does not retry it.
h3. Trigger
# Secure cluster, at least one delegation token issued.
# An SCM is restarted and becomes leader. Until {{SecretKeyManagerService}} has
run {{SecretKeyManager.checkAndInitialize}} on it, the secret key RPC answers
{{SECRET_KEY_NOT_INITIALIZED}}.
# An OM starts in that window. The window is normally short: SCM initializes
the keys as soon as it is leader ready, and if that attempt fails the next one
comes with the next rotation check ({{hdds.secret.key.rotate.check.duration}},
default 10m). Its natural width was not measured.
# The token is renewed or used for authentication on that OM.
Any other failure of the fetch at load time has the same effect, for example
SCM not reachable within the client retries (same catch, not run).
{{OzoneManager.reloadOMState}} (Ratis snapshot installation on a follower) and
{{OzoneManager.restart}} build the secret manager the same way, so they are
exposed too (read from the code, not run).
h3. Impact
* Every token loaded while the fetch fails is rejected by that OM: renewal
fails with "Delegation token renewal failed" and the OM logs "token ... can't
be found in cache", {{retrievePassword}} throws {{InvalidToken}} with the same
message, so token authentication fails. Jobs that hold delegation tokens fail
although their tokens are valid.
* The state lasts until that OM is restarted while SCM can serve the key. The
only trace at load time is one ERROR line per token.
* In OM HA every OM loads its tokens on its own, and tokens are only checked on
the leader. An OM that lost its tokens as a follower shows it when it becomes
leader (read from the code, the reproduction uses one OM).
h3. Reproduction
PASS in about one minute, unmodified source.
[^TestBugCR4OmRestartDuringScmKeyInit.java] ({{hadoop-ozone/integration-test}})
runs a Kerberos mini cluster with three SCMs and one OM. A client gets a
delegation token, a follower SCM is restarted, leadership is transferred to it
and the OM is restarted before that SCM has initialized its secret keys. The
test holds the monitor of the new leader's {{SecretKeyManager}} during the OM
restart, which only delays the synchronized {{checkAndInitialize}} so that the
window is hit on every run. Afterwards the token is unexpired, present in the
OM DB and its key is served by SCM, but {{renewDelegationToken}} fails and
{{retrievePassword}} throws {{InvalidToken}}. After a second OM restart the
same token renews. A passing test means the defect is present.
h3. Patch
[^CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch],
against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at
a65a32e7c4b.
{{addPersistedDelegationToken}} now tells the two cases apart. If the fetch
throws {{SCMSecurityException}}, the token is kept in {{currentTokens}} as an
entry of its own type ({{TokenInfoWithoutPassword}}) and one WARN line without
a stack trace is logged. {{validateToken}} computes the password of such an
entry from its secret key when the token is used, and throws {{InvalidToken}}
if the key still cannot be fetched or SCM reports it as gone. Only the load
creates such an entry, and every later update of the token replaces it with a
regular one. The value {{validateToken}} computes is the one the existing
signature check requires the stored password to have, so the accepted tokens
are the same as after a successful load. The HDDS-13234 behaviour for a key
that SCM no longer has (null result) is unchanged, and OM startup does not wait
for SCM any longer than today.
Retrying the fetch during the load was considered. It would hold the OM
constructor (and a snapshot installation) for as long as SCM does not answer,
and would still need a decision on what to do when the retries run out.
Covered by {{testReloadTokenWhenSecretKeyFetchFails}} in the existing
{{TestOzoneDelegationTokenSecretManager}}: the fetch fails during the reload
and on first use, then SCM reports the key as missing, then serves it. Without
the change the test fails because the token "can't be found in cache". With the
change {{TestOzoneDelegationTokenSecretManager}} (17 tests, including
{{testExpiredSecretKey}} from HDDS-13234), the token identifier, delegation
token request and response and double buffer unit tests (19 tests),
{{TestDelegationToken}} (2 tests), {{TestSecureOzoneCluster}} (9 tests) and
{{TestSecretKeySnapshot}} pass and checkstyle is clean. With the patch the
reproduction fails at the assertion that expects the renewal to fail, because
the renewal succeeds.
Found by code review of the SCM secret key rotation and distribution paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
