Siyao Meng created HDDS-16958:
---------------------------------
Summary: OM delegation token and secret key duration guard ignores
hdds.secret.key.rotate.check.duration
Key: HDDS-16958
URL: https://issues.apache.org/jira/browse/HDDS-16958
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-1-include-rotation-check-interval-in-delegation-token-guard.patch,
TestBugCR1TokenOutlivesSecretKey.java
h3. Mechanism
HDDS-13343 added a startup guard in
{{OzoneManager.createDelegationTokenSecretManager}}: OM refuses to start unless
{{hdds.secret.key.rotate.duration}} +
{{ozone.manager.delegation.token.max-lifetime}} +
{{ozone.manager.delegation.remover.scan.interval}} <=
{{hdds.secret.key.expiry.duration}}. The formula assumes that a secret key
signs tokens for exactly the rotate duration. It can sign for up to two
{{hdds.secret.key.rotate.check.duration}} longer:
* SCM rotates only when {{SecretKeyManagerService}} runs, every rotation check
interval, so the rotation is up to one interval late.
* {{DefaultSecretKeySignerClient}} on OM polls SCM for a new key at the rotate
duration of its current key and then every rotation check interval, so OM keeps
signing with the old key for up to one more interval.
With R = rotate duration, C = rotation check duration, L = token max lifetime
and E = key expiry duration, the last token signed with a key can have a max
date of up to key creation + R + 2C + L, while SCM may remove the key at the
first rotation after key creation + E. If E < R + 2C + L, the key can be gone
before the token's max date. An OM that has not cached the key (after a
failover, or because it never verified a token of that key) then rejects the
token: {{verifySignature}} logs "Secret verify key ... not found" and the
client gets "Tampered/Invalid token.". An OM restarted after the removal skips
the token when it loads its tokens.
h3. Trigger
The guard already forces E >= R + L + remover scan interval, so this needs both
of the following:
# The remover scan interval is shorter than two rotation check intervals.
# The expiry duration is within two rotation check intervals of R + L.
The default settings (R = 1d, C = 10m, E = 9d, L = 7d, scan interval 1h) are
not affected, and neither is any expiry duration that the guard accepts as long
as the rotation check interval is at most half of the remover scan interval.
The two conditions are necessary, not sufficient: SCM removes an expired key
only at a rotation, so whether a key is hit depends on how the two timers line
up. The check in the patch is the worst case bound. The reproduction below uses
a rotation check interval close to the rotate duration.
h3. Impact
A delegation token signed shortly before OM switches to a new key is rejected
at most two rotation check intervals before its max date, on an OM that does
not have the key cached. Only configurations whose expiry duration is within
two rotation check intervals of R + L are affected, which the guard allows only
when the remover scan interval is shorter than two rotation check intervals.
Block and container tokens were not examined.
h3. Reproduction
PASS in about two minutes, unmodified source.
[^TestBugCR1TokenOutlivesSecretKey.java] ({{hadoop-ozone/integration-test}})
starts a Kerberos secured SCM and OM with R = 18s, C = 15s, E = 89s, L = 70s
and a scan interval of 1s, which the guard accepts (18 + 70 + 1 = 89). It
requests tokens until OM has switched keys twice, keeps the last token of the
previous key and waits until SCM has removed that key. In the run SCM removed
the key 18.3s before the max date of the token, and the token was then rejected
with "Tampered/Invalid token." 18.2s before its max date, while a token of the
current key was accepted. A passing test means the defect is present.
h3. Patch
[^CR-1-include-rotation-check-interval-in-delegation-token-guard.patch],
against 025266b52a6dcaf39508607cad736799c40907e8. It also applies to master at
a65a32e7c4b.
The guard gets a second condition, R + L + 2C <= E, with its own message, and
the descriptions of {{ozone.manager.delegation.token.max-lifetime}} and
{{hdds.secret.key.expiry.duration}} in {{ozone-default.xml}} name it. The
existing condition is unchanged. The default settings and the compose and test
configurations in the tree pass, except {{TestSecretKeySnapshot}}, which sits
exactly on the old minimum (R = 30s, C = 1s, E = 61s, L = 30s, scan interval
1s) and gets a token lifetime that is one check interval shorter.
Compatibility: OM refuses to start on a configuration that violates the new
condition, like the HDDS-13343 check next to it. The configurations that start
today and are refused after the change are those with R + L + scan interval <=
E < R + L + 2C, for example the default settings with C = 1h and E = 8d1h. The
new condition is a worst case bound, sufficient but not necessary, so it also
refuses configurations in which the timers never line up. Logging a warning
instead of refusing to start is the alternative if reviewers prefer it. Adding
2C to the existing sum (R + L + scan interval + 2C <= E) was considered and not
chosen, because it would also refuse configurations in which no unexpired token
can lose its key.
Covered by a second case in the existing guard check of
{{TestSecureOzoneCluster.testSecureScmAndOmStartupAndAccessControl}}: E = 8d1h
with C = 1h passes the old condition and is now refused. Without the change the
test fails because OM starts. With it {{TestSecureOzoneCluster}} (9 tests),
{{TestSecretKeySnapshot}}, {{TestDelegationToken}} and {{TestBlockTokens}} pass
and checkstyle is clean. With the patch the reproduction fails when it creates
the OM, with the new message.
Found by code review of the SCM secret key rotation and distribution paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
025266b52a6dcaf39508607cad736799c40907e8. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
