[
https://issues.apache.org/jira/browse/HDDS-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16959:
------------------------------
Attachment:
CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch
> OM drops persisted delegation tokens from memory when the secret key fetch
> from SCM fails during token load
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16959
> URL: https://issues.apache.org/jira/browse/HDDS-16959
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch,
> TestBugCR4OmRestartDuringScmKeyInit.java
>
>
> h3. Mechanism
> The OM DB stores only the identifier and the renew date of a delegation
> token. When {{OzoneDelegationTokenSecretManager}} is created,
> {{loadTokenSecretState}} calls {{addPersistedDelegationToken}} for every
> stored token, which fetches the token's secret key from SCM with
> {{SecretKeyClient.getSecretKey}} and recomputes the password.
> {{loadTokenSecretState}} catches every exception from that call, logs
> "exception while loading delegation token from DB... ignored to continue
> startup" and goes on to the next token.
> HDDS-13234 added that catch for a secret key that SCM has already removed, in
> which case {{getSecretKey}} returns null and the token can never be verified
> again. A fetch that fails ({{SCMSecurityException}} "Error fetching secret
> key ...") ends in the same catch. Before HDDS-13234 the same failure aborted
> the OM start. That change is in 2.1.0 and later releases (its fix versions
> are 2.0.1 and 2.1.0). The token is then missing from {{currentTokens}}
> although it is unexpired, still in the DB, and its key still exists on SCM.
> Nothing loads it later, so the OM rejects it until the OM process is
> restarted.
> Two things make the failed fetch reachable during a normal start:
> * The token load runs in the {{OzoneManager}} constructor
> ({{instantiateServices}}), before {{startSecretManager}} starts the signer
> client. Only the signer client's initial fetch waits for SCM to leave
> {{SECRET_KEY_NOT_INITIALIZED}}.
> * {{SecretKeyProtocolClientSideTranslatorPB.handleError}} turns the error
> status into an exception after the RPC has returned, so the SCM failover
> retry policy does not retry it.
> h3. Trigger
> # Secure cluster, at least one delegation token issued.
> # An SCM is restarted and becomes leader. Until {{SecretKeyManagerService}}
> has run {{SecretKeyManager.checkAndInitialize}} on it, the secret key RPC
> answers {{SECRET_KEY_NOT_INITIALIZED}}.
> # An OM starts in that window. The window is normally short: SCM initializes
> the keys as soon as it is leader ready, and if that attempt fails the next
> one comes with the next rotation check
> ({{hdds.secret.key.rotate.check.duration}}, default 10m). Its natural width
> was not measured.
> # The token is renewed or used for authentication on that OM.
> Any other failure of the fetch at load time has the same effect, for example
> SCM not reachable within the client retries (same catch, not run).
> {{OzoneManager.reloadOMState}} (Ratis snapshot installation on a follower)
> and {{OzoneManager.restart}} build the secret manager the same way, so they
> are exposed too (read from the code, not run).
> h3. Impact
> * Every token loaded while the fetch fails is rejected by that OM: renewal
> fails with "Delegation token renewal failed" and the OM logs "token ... can't
> be found in cache", {{retrievePassword}} throws {{InvalidToken}} with the
> same message, so token authentication fails. Jobs that hold delegation tokens
> fail although their tokens are valid.
> * The state lasts until that OM is restarted while SCM can serve the key. The
> only trace at load time is one ERROR line per token.
> * In OM HA every OM loads its tokens on its own, and tokens are only checked
> on the leader. An OM that lost its tokens as a follower shows it when it
> becomes leader (read from the code, the reproduction uses one OM).
> h3. Reproduction
> PASS in about one minute, unmodified source.
> [^TestBugCR4OmRestartDuringScmKeyInit.java]
> ({{hadoop-ozone/integration-test}}) runs a Kerberos mini cluster with three
> SCMs and one OM. A client gets a delegation token, a follower SCM is
> restarted, leadership is transferred to it and the OM is restarted before
> that SCM has initialized its secret keys. The test holds the monitor of the
> new leader's {{SecretKeyManager}} during the OM restart, which only delays
> the synchronized {{checkAndInitialize}} so that the window is hit on every
> run. Afterwards the token is unexpired, present in the OM DB and its key is
> served by SCM, but {{renewDelegationToken}} fails and {{retrievePassword}}
> throws {{InvalidToken}}. After a second OM restart the same token renews. A
> passing test means the defect is present.
> h3. Patch
> [^CR-4-keep-persisted-delegation-tokens-when-secret-key-fetch-fails.patch],
> against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master
> at a65a32e7c4b.
> {{addPersistedDelegationToken}} now tells the two cases apart. If the fetch
> throws {{SCMSecurityException}}, the token is kept in {{currentTokens}} as an
> entry of its own type ({{TokenInfoWithoutPassword}}) and one WARN line
> without a stack trace is logged. {{validateToken}} computes the password of
> such an entry from its secret key when the token is used, and throws
> {{InvalidToken}} if the key still cannot be fetched or SCM reports it as
> gone. Only the load creates such an entry, and every later update of the
> token replaces it with a regular one. The value {{validateToken}} computes is
> the one the existing signature check requires the stored password to have, so
> the accepted tokens are the same as after a successful load. The HDDS-13234
> behaviour for a key that SCM no longer has (null result) is unchanged, and OM
> startup does not wait for SCM any longer than today.
> Retrying the fetch during the load was considered. It would hold the OM
> constructor (and a snapshot installation) for as long as SCM does not answer,
> and would still need a decision on what to do when the retries run out.
> Covered by {{testReloadTokenWhenSecretKeyFetchFails}} in the existing
> {{TestOzoneDelegationTokenSecretManager}}: the fetch fails during the reload
> and on first use, then SCM reports the key as missing, then serves it.
> Without the change the test fails because the token "can't be found in
> cache". With the change {{TestOzoneDelegationTokenSecretManager}} (17 tests,
> including {{testExpiredSecretKey}} from HDDS-13234), the token identifier,
> delegation token request and response and double buffer unit tests (19
> tests), {{TestDelegationToken}} (2 tests), {{TestSecureOzoneCluster}} (9
> tests) and {{TestSecretKeySnapshot}} pass and checkstyle is clean. With the
> patch the reproduction fails at the assertion that expects the renewal to
> fail, because the renewal succeeds.
> Found by code review of the SCM secret key rotation and distribution paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
