[
https://issues.apache.org/jira/browse/HDDS-16923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16923:
------------------------------
Attachment: TestBugMC3SecretKeyFileLagsInstalledDb.java
> SCM snapshot install replaces scm.db before saving the leader's secret keys,
> so a failure in between later drops unexpired secret keys on all SCMs
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16923
> URL: https://issues.apache.org/jira/browse/HDDS-16923
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-3-save-secret-keys-before-replacing-scm-db.patch,
> TestBugMC3SecretKeyFileLagsInstalledDb.java
>
>
> h3. Mechanism
> SCM keeps its symmetric secret keys only in memory and in a local file
> ({{LocalSecretKeyStore}}). They are not in scm.db. When a follower installs a
> Ratis snapshot, {{SCMStateMachine.reinitialize}} first calls
> {{SCMHAManagerImpl.installCheckpoint}}, which moves the leader's checkpoint
> into place as scm.db and reloads all managers from it. Only after that does
> it call {{SecretKeyManager.reinitialize}} with the keys fetched from the
> leader, which updates memory and saves the file.
> If the SCM stops between the two steps, scm.db is at the checkpoint index
> while the key file is from before the install, or missing. The key rotations
> in between are already purged from the leader's log, so they are never
> replayed, and a follower never loads the key file at startup, so its in
> memory key state stays empty. When this SCM later becomes leader,
> {{SecretKeyManager.checkAndInitialize}} loads the stale file (or, if nothing
> unexpired is left, generates one new key) and replicates it through the
> {{@Replicate}} method {{SecretKeyState.updateKeys}}. Every SCM replaces its
> key list with it and rewrites its own file, so keys that have not expired are
> gone from the whole cluster. Later rotations build on that list, so they
> never come back.
> h3. Trigger
> # A follower SCM is behind the leader's purged log (a newly bootstrapped SCM,
> or one that was down while the leader rotated a key and moved on), so it
> receives an install snapshot notification.
> # Between the DB swap and the key save, the SCM stops. This happens without a
> crash if {{LocalSecretKeyStore.save}} throws, for example on an I/O error, or
> on a full disk when the key file is on another volume than scm.db
> ({{hdds.metadata.dir}} versus {{ozone.scm.db.dirs}}): {{reinitialize}}
> throws, Ratis closes the state machine and SCM terminates. (If both share one
> full volume, reopening the new DB is likely to fail first, and the install
> rolls back to the old DB.) A process crash (kill -9, OOM kill, power loss) at
> that point should leave the same files; this was argued from code, not run. A
> graceful stop waits for {{reinitialize}} to finish and is not a trigger.
> # The SCM restarts and becomes leader (election or {{ozone admin scm
> transfer}}) before a further key rotation is replicated to it
> ({{hdds.secret.key.rotate.duration}}, default 1d).
> The window covers the reload of all SCM managers from the new DB, the
> deletion of the old DB backup and the transaction buffer init. It was about
> 60 ms on a small test DB and grows with the size of the SCM tables (not
> measured on a large DB).
> h3. Impact
> * Unexpired secret keys are removed from every SCM.
> {{SecretKeyProtocol.getSecretKey}} returns null for them, so datanodes reject
> block and container tokens signed with them ({{ShortLivedTokenVerifier}}:
> "Can't find the signing secret key"), and OM cannot verify delegation and STS
> tokens signed with them. This was run on a secure mini cluster up to the
> {{getSecretKey}} lookup. The token level consequences below are from code
> reading.
> * OM keeps signing new tokens with its cached current key, which may be one
> of the dropped keys. It switches only after that key passes the rotation
> duration, or after {{ozone admin om fetch-key}}. Until then, retries that get
> a new block token from OM fail the same way. A datanode or OM that had
> already cached the key before it was dropped keeps accepting it.
> * Delegation tokens signed with a dropped key fail until they are replaced or
> expire, which can take up to their 7 day maximum lifetime
> ({{ozone.manager.delegation.token.max-lifetime}}); read from code, not run.
> * It fails closed: no token is accepted that should be rejected. It ends once
> OM signs with the new key and the tokens signed with the dropped keys have
> been replaced or have expired.
> h3. Reproduction
> PASS on an unmodified checkout, no hook or instrumentation.
> [^TestBugMC3SecretKeyFileLagsInstalledDb.java]
> ({{hadoop-ozone/integration-test}}) uses the setup of
> {{TestSecretKeySnapshot}}: a secure mini cluster with 3 SCMs of which 2
> start. The leader rotates from K1 to K2 and its log is purged past index 200.
> A directory is created at the inactive SCM's key file path, so the save fails
> with an I/O error, and the SCM is started. It installs the snapshot and
> terminates, with scm.db at index 202 and no key file. The directory is
> removed, the SCM is restarted and leadership is transferred to it. All three
> SCMs then hold only one newly generated key, and {{getSecretKey(K2)}} returns
> null although K2 expires hours later. A passing test means the defect is
> present. The class Javadoc explains two harness steps needed only for the in
> JVM restart.
> h3. Patch
> [^MC-3-save-secret-keys-before-replacing-scm-db.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> {{SCMStateMachine.reinitialize}} now saves the keys fetched from the leader
> before it calls {{installCheckpoint}}. The keys are fetched after the
> checkpoint download, so they include every unexpired key at the checkpoint
> index. If the SCM stops after the save, it restarts on the old DB, installs
> again and saves again. If the save fails, the DB is not touched.
> Covered by two tests in the existing {{TestSCMStateMachine}}: the keys are
> saved before the DB is replaced, and a failing save means
> {{installCheckpoint}} is never called. Both fail without the change. With it,
> {{TestSCMStateMachine}} and the neighbouring SCM HA unit suites (27 tests),
> {{TestSecretKeySnapshot}}, {{TestSecretKeysApi}}, {{TestSCMInstallSnapshot}}
> and {{TestSCMInstallSnapshotWithHA}} pass, and checkstyle is clean. With the
> patch, the reproduction fails at its first assertion because the follower's
> DB is never replaced. With that assertion removed, the restarted SCM installs
> again and keeps K1 and K2 after it becomes leader.
> A related patch for the same {{reinitialize}} method (a transaction buffer
> init failure there is swallowed) is filed separately. The two conflict
> textually in {{SCMStateMachine.reinitialize}} and {{TestSCMStateMachine}}.
> This one is meant to go in first; the other will be rebased on it and reuse
> its test helper.
> {{LocalSecretKeyStore.save}} still rewrites the file in place without fsync.
> Writing to a temporary file and renaming it would close the remaining gap for
> a crash during the save itself or a power loss; that is not part of this
> patch.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
