[
https://issues.apache.org/jira/browse/HDDS-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16961:
------------------------------
Attachment: CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch
> ozone admin om fetch-key does not make OM fetch the current secret key from
> SCM, it prints the key OM already has
> -----------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16961
> URL: https://issues.apache.org/jira/browse/HDDS-16961
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch,
> TestBugCR11FetchKeyDoesNotReachSigner.java
>
>
> h3. Mechanism
> {{ozone admin om fetch-key}} is described as "CLI command to force OM to
> fetch the latest secret key from SCM". {{OzoneManager.refetchSecretKey}}
> serves it with two calls on its secret key client: {{refetchSecretKey()}},
> then {{getCurrentSecretKey().getId()}}, which the command prints as "Current
> Secret Key ID".
> Since HDDS-8829 that client is a {{DefaultSecretKeyClient}}, the composition
> of a signer client and a verifier client. It passes {{getCurrentSecretKey}},
> {{start}} and {{stop}} on to {{DefaultSecretKeySignerClient}}, but not
> {{refetchSecretKey}}. So the empty default of
> {{SecretKeySignerClient.refetchSecretKey}} runs, SCM is not asked, and the
> command reports the ID of the key OM already holds, with exit code 0.
> When HDDS-8677 added the command, OM held a {{DefaultSecretKeySignerClient}}
> directly, whose {{refetchSecretKey}} does ask SCM. The command worked in
> 1.4.x and has had no effect since 2.0.0.
> h3. Trigger
> # Secure cluster.
> # SCM gets a new current secret key that OM does not have yet, for example
> through {{ozone admin scm rotate --force}}.
> # {{ozone admin om fetch-key}} is run.
> h3. Impact
> * The command reports success and a key ID, but OM keeps signing with its old
> key until its own poller fetches the new one, which happens once the old key
> is older than {{hdds.secret.key.rotate.duration}} (1d). OM cannot be made to
> switch earlier, which is what the command is for, short of restarting it.
> * The printed ID is not SCM's current key, without any hint.
> * Nothing else is affected. The old key stays valid on SCM until it expires,
> so tokens that OM issues in the meantime are accepted.
> h3. Reproduction
> PASS, deterministic, unmodified source, under 1 second.
> [^TestBugCR11FetchKeyDoesNotReachSigner.java] ({{hadoop-hdds/framework}})
> creates the client the way OM does, with a mock SCM that has a new current
> key after the first fetch, and makes the two calls of
> {{OzoneManager.refetchSecretKey}}. The key is unchanged and SCM was called
> once (by {{start}}). A second test is the control: the same sequence on
> {{DefaultSecretKeySignerClient}} alone returns the new key. A passing test
> means the defect is present.
> On a secure mini cluster the command itself was run through
> {{TestBlockTokensCLI.testFetchKeyOMAdminCommand}} with a forced rotation on
> SCM added before it (this is the regression test of the patch): without the
> fix it fails with "expected: <3f7f03d9-...> but was: <e13b7dfc-...>", the
> first being SCM's current key and the second the ID that the command printed.
> h3. Patch
> [^CR-11-let-om-fetch-key-reach-secret-key-signer-client.patch], against
> 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at
> 4389803780f.
> {{DefaultSecretKeyClient.refetchSecretKey}} now passes the call on to the
> signer client, as it does for the other signer methods. The verifier side is
> not touched. {{OzoneManager.refetchSecretKey}} rethrows the cause of the
> {{UncheckedIOException}} of the signer client, so that a failed fetch from
> SCM is returned to the caller as an error.
> {{TestBlockTokensCLI.testFetchKeyOMAdminCommand}} now rotates the key on SCM
> first, so that the printed ID can only match if OM really fetched it. The
> existing test compared the printed ID with SCM's current key without a
> rotation in between, which is why it kept passing.
> {{TestOMAdminRpcAuthorization}} gets a test in which the fetch fails and the
> caller has to receive the error.
> With the patch {{TestBlockTokensCLI}} (3 tests),
> {{TestOMAdminRpcAuthorization}} (5 tests) and {{TestBlockTokens}} (5 tests)
> pass on a mini cluster, the secret key and token unit suites of
> {{hadoop-hdds/framework}} (38 tests) pass, and checkstyle is clean. Without
> the two main changes the changed test and the new test fail. With the patch
> the reproduction fails at its assertion that the key is unchanged.
> Found by code review of the SCM secret key rotation and distribution paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
