[
https://issues.apache.org/jira/browse/HDDS-16928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16928:
------------------------------
Attachment:
MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch
> SCM never catches up after a checkpoint install that fails or is interrupted
> once Ratis has acknowledged it, even after a restart
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16928
> URL: https://issues.apache.org/jira/browse/HDDS-16928
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch,
> TestBugMC1FailedInstallWedgesFollower.java
>
>
> h3. Mechanism
> {{SCMStateMachine.notifyInstallSnapshotFromLeader}} completes its future with
> the checkpoint TermIndex as soon as the leader's checkpoint is downloaded and
> verified. The DB swap is deferred to {{SCMStateMachine.reinitialize}}, which
> Ratis runs later on the {{StateMachineUpdater}} thread. On the next install
> notification Ratis (3.3.1) pauses the state machine and signals the
> {{StateMachineUpdater}} thread to run {{reinitialize}}, but does not wait for
> it: in the same call it purges the follower's Raft log up to the checkpoint
> index and replies SNAPSHOT_INSTALLED, and the leader continues with the
> entries after that index. {{reinitialize}} then calls
> {{SCMHAManagerImpl.installCheckpoint}}.
> If that install fails, every failure path keeps the old DB (failed backup
> move, failed copy with restore, failed reload with rollback), and SCM shuts
> down. The same holds if the process is killed after the acknowledgement and
> before {{HAUtils.replaceDBWithCheckpoint}} moves the old DB aside (argued,
> not run); a kill during the copy leaves the {{dbInconsistentMarker}} file,
> and the SCM then refuses to start. The pending checkpoint was only held in
> memory. After a restart the applied index is the old DB's (for a new SCM, -1)
> while the Raft log starts after the checkpoint index, so
> {{StateMachineUpdater}} never finds the next entry to apply. The leader does
> not send another install notification, because the follower's next index is
> already inside the leader's log.
> h3. Trigger
> # An SCM installs the leader's checkpoint through the Ratis install
> notification. At default settings
> ({{ozone.scm.ha.ratis.log.purge.enabled=false}}) this is in practice an SCM
> that is bootstrapping (new or replaced node); with purge enabled, also a
> follower that fell behind.
> # The leader commits at least one more entry that reaches this SCM before the
> install completes (any SCM write traffic).
> # {{installCheckpoint}} fails with the old DB kept, or the process is killed
> after the acknowledgement and before the old DB is moved aside (a kill during
> the copy leaves the {{dbInconsistentMarker}}, and the SCM then refuses to
> start). Realistic causes are a full disk during the copy of the whole DB, an
> I/O error, or a failure while reloading the SCM services from the new DB. In
> the reproduction the failure is forced with a leftover
> {{dbInconsistentMarker}} file in the DB directory, which makes
> {{HAUtils.replaceDBWithCheckpoint}} fail and restore the old DB. The kill
> case was not run. For a bootstrapping SCM this needs the leader to have added
> it to the Raft group, which happens within moments of the acknowledgement,
> long before a copy of a real DB ends; when the injected immediate failure
> came first, the bootstrap did not complete instead (not followed further).
> # The SCM is restarted.
> h3. Impact
> * The restarted SCM never applies another transaction and never starts its
> datanode server, with no error beyond the periodic "Deferred datanode-server
> start is still waiting for follower catch-up" warning. Nothing recovers it
> automatically.
> * A graceful stop of that SCM never completes ({{StateMachineUpdater}} waits
> to reach the commit index), and its updater thread spins on one core.
> * Its Raft log is complete, so it can become leader. After a leadership
> transfer to it (admin {{transferLeadership}}), it was the Ratis leader and no
> SCM reported itself as a ready leader for the 20 s observed, so the SCM
> service was unavailable. I did not run an election that picks it without the
> transfer.
> * Recovery needs an operator: kill it, remove its SCM DB and Ratis storage,
> and bootstrap it again (argued, not run).
> h3. Reproduction
> PASS on unmodified source, default configuration.
> [^TestBugMC1FailedInstallWedgesFollower.java]
> ({{hadoop-ozone/integration-test}}) starts three SCMs with one inactive,
> places the marker file in the inactive SCM's DB directory, starts it while
> the leader keeps allocating containers, waits until it shuts down after the
> failed install, removes the file and restarts it. Result: SNAPSHOT_INSTALLED
> and the leader moving to the next index logged before the install ran; after
> the restart applied index -1 before and after, log start 90, commit 192
> against leader applied 191, 0 install notifications, datanode server not
> started, 0 of 80 containers, about 9.7 s of updater CPU in 10 s; after a
> leadership transfer to it, it was the Ratis leader and no SCM was a ready
> leader; a graceful stop still running after 20 s. In this JVM the restarted
> SCM gets new datanode and gRPC ports, because the stopped instance keeps them
> bound; a process exit releases them. If the injected failure comes before the
> leader has added the SCM to the Raft group, the test aborts and has to be
> rerun. A passing test means the defect is present.
> h3. Suggested fix
> SCM cannot undo the purge and the acknowledgement from {{reinitialize}}.
> Recommended: install the checkpoint inside the notification future, as
> {{OzoneManagerStateMachine.notifyInstallSnapshotFromLeader}} does, and fail
> the future on any error, so Ratis neither purges nor acknowledges a failed
> install. This reverses the deferral chosen in HDDS-5173 and needs the updater
> to be quiesced during the swap. It should not copy OM's handling of a failed
> install, which completes the future with null (HDDS-11470). As a safety net,
> SCM should check at startup whether its applied index is below the start of
> its Raft log and fail with a clear message; that also catches the same state
> reached through other paths, such as a transaction buffer init failure in
> {{reinitialize}} or a reload failure that rolls back to the old DB.
> Alternatives: change Ratis to purge and reply SNAPSHOT_INSTALLED only after
> {{reinitialize}} has succeeded, or persist the pending checkpoint before
> acknowledging and install it at startup.
> h3. Patch
> [^MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch],
> against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63, test only. It also applies
> to master at 52812559dad.
> It adds {{testSCMCatchesUpAfterFailedCheckpointInstall}} to
> {{TestStorageContainerManagerHA}}. A follower SCM that is a voting member
> falls behind the start of the leader's log (log purge enabled, one snapshot
> requested on the leader), and when it restarts its install fails once in
> {{installCheckpoint}} with the old DB in place (its metadata store, set
> through {{SCMConfigurator.setMetadataStore}}, refuses to stop once). After
> another restart the follower must catch up with the leader. On the current
> code it fails with "Timed out waiting for condition": the follower stays at
> applied index 42 while the leader is at 1475 and the follower's log starts at
> 1261, and the teardown then takes several minutes (about 5) because the
> wedged SCM's stop does not complete. It is meant to land with the fix. A
> member follower is used instead of the bootstrap case of the reproduction
> because an injected failure is immediate, and for a bootstrapping SCM it can
> come before the leader has added the SCM to the Raft group.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS
> issues and apache/ozone pull requests for duplicates before filing. The
> attached patch is a test proposal for review. Generated with Specula (Claude
> Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
