[
https://issues.apache.org/jira/browse/HDDS-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16917:
------------------------------
Attachment: TestBugMC4CrashBetweenCommitMoves.java
> SCM that crashes while it applies a root CA rotation commit cannot start
> again, the sub CA directory switch is not completed on start
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16917
> URL: https://issues.apache.org/jira/browse/HDDS-16917
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-4-complete-interrupted-rotation-commit-on-scm-start.patch,
> TestBugMC4CrashBetweenCommitMoves.java
>
>
> h3. Mechanism
> {{RootCARotationHandlerImpl.rotationCommit}} switches to the new sub CA
> certificate in three separate steps: move {{scm/sub-ca}} to
> {{scm/sub-ca-previous}}, move {{scm/sub-ca-next}} to {{scm/sub-ca}}, write
> the new certificate ID to the VERSION file. SCM start reads only the VERSION
> file and {{scm/sub-ca}} and has no step that completes or reverts a switch
> that stopped in between.
> * Stopped after the first move: {{scm/sub-ca}} does not exist.
> {{DefaultCAServer.init}} for the sub CA throws {{IllegalStateException:
> INTERMEDIARY_CA Should not be in Initialize State during startup.}} in the
> {{StorageContainerManager}} constructor, on the primary SCM and on any other
> SCM.
> * Stopped after the second move: {{scm/sub-ca}} holds the new certificate,
> the VERSION file names the old one. {{SCMCertificateClient.getCertificate}}
> returns null. On the primary SCM the constructor fails with a
> {{NullPointerException}} in {{persistPrimarySCMCerts}}. On any other SCM the
> constructor returns without a certificate, and
> {{RootCARotationManager.shouldSkipRootCert}}, the first call of the commit
> when it is applied again, throws a {{NullPointerException}} on the missing
> trust chain. Inside {{SCMStateMachine.applyTransaction}} that terminates the
> SCM (the replay itself is argued, not run).
> The commit can never be applied again to finish the switch, because the SCM
> fails before or while it replays the RAFT log.
> The same three steps are in the release tags ozone-1.4.0, 1.4.1, 2.0.0,
> 2.1.0, 2.1.1, 2.1.2, 2.2.0 and 2.2.1, and SCM start does not look at
> {{scm/sub-ca-previous}} in any of them (read with git grep, not run).
> h3. Trigger
> Secure SCM HA with {{hdds.x509.ca.rotation.enabled=true}} (default false). An
> SCM process dies (kill, OOM, power loss) between the first directory move and
> the VERSION write of {{rotationCommit}}. The window is two renames and one
> small file write, once per rotation and SCM. With the default
> {{hdds.x509.max.duration}} of P1865D a rotation runs once in about five years.
> The second state is also reached without a process death in that window. An
> SCM that is restarted between its prepare ack and the commit terminates after
> both moves and before the VERSION write (HDDS-16913). An {{IOException}} from
> the VERSION write makes the handler call {{scm.shutDown}} on the thread that
> applies the log entry, which leaves the SCM hung until it is killed
> (HDDS-16911), and the next start then finds the second state (argued, not
> run).
> h3. Impact
> The SCM fails on every later start until an operator moves the directories or
> edits the VERSION file by hand. No key or certificate is lost: the old pair
> is in {{scm/sub-ca-previous}} and the new pair in {{scm/sub-ca-next}} or
> {{scm/sub-ca}}. {{ozone repair scm cert recover}} does not handle these
> directories.
> If the SCM was a follower, the other SCMs finish the rotation and the ring
> runs with one member less. If it was the leader, the other SCMs still apply
> the commit, but the committed request, which only the leader sends and only
> right after the commit, is not sent (read, not run). The rotation is then
> also left unfinished on the other SCMs, see HDDS-16912.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC4CrashBetweenCommitMoves.java] ({{hadoop-ozone/integration-test}})
> runs {{scm --init}} of a secure SCM with a MiniKdc, stages a new sub CA
> certificate with the real certificate client and a new root CA (the leader
> that signs the request is a mock backed by that CA), performs the directory
> moves of {{rotationCommit}} up to the crash point and then constructs the
> {{StorageContainerManager}} again. Both crash points fail as described above.
> A passing test means the defect is present.
> The test SCM is the primary. For an SCM that is not the primary the same
> states were given to the real constructor after the primary SCM ID in the
> VERSION file was changed (run, not part of the attached reproduction): after
> the first move the constructor throws the same {{IllegalStateException}},
> after the second move it returns with {{getCertificate()}} null and
> {{shouldSkipRootCert}} throws the {{NullPointerException}}.
> h3. Patch
> [^MC-4-complete-interrupted-rotation-commit-on-scm-start.patch], against
> b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
> 5108acff480. The patch is made against that commit alone, other fixes
> proposed for the same code may need a manual merge.
> Before the SCM certificate client is created,
> {{StorageContainerManager.initializeCertificateClient}} calls the new
> {{RootCARotationManager.completeInterruptedCommit}}. It acts only when the
> certificate named in the VERSION file is found in {{scm/sub-ca-previous}} and
> is not the one in {{scm/sub-ca}}, which only an unfinished commit leaves
> behind. It then moves {{scm/sub-ca-next}} to {{scm/sub-ca}} if that has not
> happened, writes the ID of the certificate in {{scm/sub-ca}} to the VERSION
> file and deletes {{scm/sub-ca-previous}}. The switch is completed and not
> reverted because the commit entry is already committed in the RAFT log. When
> the commit and committed entries are applied again they are skipped, since
> the SCM already trusts the new root certificate. If {{scm/sub-ca}} is missing
> and nothing is staged in {{scm/sub-ca-next}} the directories are left as they
> are and the start fails as before (run). A certificate that is staged but was
> never committed is not touched.
> Covered by four cases of {{testScmStartCompletesInterruptedRotationCommit}}
> in the existing {{TestSecureOzoneCluster}}: the two crash points, a
> certificate that is staged and not committed, and a {{scm/sub-ca-previous}}
> that is a copy of the directory in use. In the last two the SCM must start on
> its old certificate and leave the directories alone. Without the change the
> two crash point cases fail with the two exceptions above. With it the new
> test, {{testSecureScmStartupFailure}} of the same class,
> {{TestRootCARotationManager}} and {{TestScmInvokerCodeGenerator}} pass (18
> tests) and checkstyle is clean. The other tests of {{TestSecureOzoneCluster}}
> were not run. With the patch both cases of the reproduction fail, because the
> SCM starts.
> Found by TLA+ model checking and code review of the SCM root CA rotation
> paths under HDDS-15926, on commit b9b33851b20fa11baf4b07982de9afb8ea44f85a.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
