[
https://issues.apache.org/jira/browse/HDDS-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16958:
------------------------------
Attachment:
CR-1-include-rotation-check-interval-in-delegation-token-guard.patch
> OM delegation token and secret key duration guard ignores
> hdds.secret.key.rotate.check.duration
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16958
> URL: https://issues.apache.org/jira/browse/HDDS-16958
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-1-include-rotation-check-interval-in-delegation-token-guard.patch,
> TestBugCR1TokenOutlivesSecretKey.java
>
>
> h3. Mechanism
> HDDS-13343 added a startup guard in
> {{OzoneManager.createDelegationTokenSecretManager}}: OM refuses to start
> unless {{hdds.secret.key.rotate.duration}} +
> {{ozone.manager.delegation.token.max-lifetime}} +
> {{ozone.manager.delegation.remover.scan.interval}} <=
> {{hdds.secret.key.expiry.duration}}. The formula assumes that a secret key
> signs tokens for exactly the rotate duration. It can sign for up to two
> {{hdds.secret.key.rotate.check.duration}} longer:
> * SCM rotates only when {{SecretKeyManagerService}} runs, every rotation
> check interval, so the rotation is up to one interval late.
> * {{DefaultSecretKeySignerClient}} on OM polls SCM for a new key at the
> rotate duration of its current key and then every rotation check interval, so
> OM keeps signing with the old key for up to one more interval.
> With R = rotate duration, C = rotation check duration, L = token max lifetime
> and E = key expiry duration, the last token signed with a key can have a max
> date of up to key creation + R + 2C + L, while SCM may remove the key at the
> first rotation after key creation + E. If E < R + 2C + L, the key can be gone
> before the token's max date. An OM that has not cached the key (after a
> failover, or because it never verified a token of that key) then rejects the
> token: {{verifySignature}} logs "Secret verify key ... not found" and the
> client gets "Tampered/Invalid token.". An OM restarted after the removal
> skips the token when it loads its tokens.
> h3. Trigger
> The guard already forces E >= R + L + remover scan interval, so this needs
> both of the following:
> # The remover scan interval is shorter than two rotation check intervals.
> # The expiry duration is within two rotation check intervals of R + L.
> The default settings (R = 1d, C = 10m, E = 9d, L = 7d, scan interval 1h) are
> not affected, and neither is any expiry duration that the guard accepts as
> long as the rotation check interval is at most half of the remover scan
> interval. The two conditions are necessary, not sufficient: SCM removes an
> expired key only at a rotation, so whether a key is hit depends on how the
> two timers line up. The check in the patch is the worst case bound. The
> reproduction below uses a rotation check interval close to the rotate
> duration.
> h3. Impact
> A delegation token signed shortly before OM switches to a new key is rejected
> at most two rotation check intervals before its max date, on an OM that does
> not have the key cached. Only configurations whose expiry duration is within
> two rotation check intervals of R + L are affected, which the guard allows
> only when the remover scan interval is shorter than two rotation check
> intervals. Block and container tokens were not examined.
> h3. Reproduction
> PASS in about two minutes, unmodified source.
> [^TestBugCR1TokenOutlivesSecretKey.java] ({{hadoop-ozone/integration-test}})
> starts a Kerberos secured SCM and OM with R = 18s, C = 15s, E = 89s, L = 70s
> and a scan interval of 1s, which the guard accepts (18 + 70 + 1 = 89). It
> requests tokens until OM has switched keys twice, keeps the last token of the
> previous key and waits until SCM has removed that key. In the run SCM removed
> the key 18.3s before the max date of the token, and the token was then
> rejected with "Tampered/Invalid token." 18.2s before its max date, while a
> token of the current key was accepted. A passing test means the defect is
> present.
> h3. Patch
> [^CR-1-include-rotation-check-interval-in-delegation-token-guard.patch],
> against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master
> at a65a32e7c4b.
> The guard gets a second condition, R + L + 2C <= E, with its own message, and
> the descriptions of {{ozone.manager.delegation.token.max-lifetime}} and
> {{hdds.secret.key.expiry.duration}} in {{ozone-default.xml}} name it. The
> existing condition is unchanged. The default settings and the compose and
> test configurations in the tree pass, except {{TestSecretKeySnapshot}}, which
> sits exactly on the old minimum (R = 30s, C = 1s, E = 61s, L = 30s, scan
> interval 1s) and gets a token lifetime that is one check interval shorter.
> Compatibility: OM refuses to start on a configuration that violates the new
> condition, like the HDDS-13343 check next to it. The configurations that
> start today and are refused after the change are those with R + L + scan
> interval <= E < R + L + 2C, for example the default settings with C = 1h and
> E = 8d1h. The new condition is a worst case bound, sufficient but not
> necessary, so it also refuses configurations in which the timers never line
> up. Logging a warning instead of refusing to start is the alternative if
> reviewers prefer it. Adding 2C to the existing sum (R + L + scan interval +
> 2C <= E) was considered and not chosen, because it would also refuse
> configurations in which no unexpired token can lose its key.
> Covered by a second case in the existing guard check of
> {{TestSecureOzoneCluster.testSecureScmAndOmStartupAndAccessControl}}: E =
> 8d1h with C = 1h passes the old condition and is now refused. Without the
> change the test fails because OM starts. With it {{TestSecureOzoneCluster}}
> (9 tests), {{TestSecretKeySnapshot}}, {{TestDelegationToken}} and
> {{TestBlockTokens}} pass and checkstyle is clean. With the patch the
> reproduction fails when it creates the OM, with the new message.
> Found by code review of the SCM secret key rotation and distribution paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
