Siyao Meng created HDDS-16928:
---------------------------------
Summary: SCM never catches up after a checkpoint install that
fails or is interrupted once Ratis has acknowledged it, even after a restart
Key: HDDS-16928
URL: https://issues.apache.org/jira/browse/HDDS-16928
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch,
TestBugMC1FailedInstallWedgesFollower.java
h3. Mechanism
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} completes its future with
the checkpoint TermIndex as soon as the leader's checkpoint is downloaded and
verified. The DB swap is deferred to {{SCMStateMachine.reinitialize}}, which
Ratis runs later on the {{StateMachineUpdater}} thread. On the next install
notification Ratis (3.3.1) pauses the state machine and signals the
{{StateMachineUpdater}} thread to run {{reinitialize}}, but does not wait for
it: in the same call it purges the follower's Raft log up to the checkpoint
index and replies SNAPSHOT_INSTALLED, and the leader continues with the entries
after that index. {{reinitialize}} then calls
{{SCMHAManagerImpl.installCheckpoint}}.
If that install fails, every failure path keeps the old DB (failed backup move,
failed copy with restore, failed reload with rollback), and SCM shuts down. The
same holds if the process is killed after the acknowledgement and before
{{HAUtils.replaceDBWithCheckpoint}} moves the old DB aside (argued, not run); a
kill during the copy leaves the {{dbInconsistentMarker}} file, and the SCM then
refuses to start. The pending checkpoint was only held in memory. After a
restart the applied index is the old DB's (for a new SCM, -1) while the Raft
log starts after the checkpoint index, so {{StateMachineUpdater}} never finds
the next entry to apply. The leader does not send another install notification,
because the follower's next index is already inside the leader's log.
h3. Trigger
# An SCM installs the leader's checkpoint through the Ratis install
notification. At default settings
({{ozone.scm.ha.ratis.log.purge.enabled=false}}) this is in practice an SCM
that is bootstrapping (new or replaced node); with purge enabled, also a
follower that fell behind.
# The leader commits at least one more entry that reaches this SCM before the
install completes (any SCM write traffic).
# {{installCheckpoint}} fails with the old DB kept, or the process is killed
after the acknowledgement and before the old DB is moved aside (a kill during
the copy leaves the {{dbInconsistentMarker}}, and the SCM then refuses to
start). Realistic causes are a full disk during the copy of the whole DB, an
I/O error, or a failure while reloading the SCM services from the new DB. In
the reproduction the failure is forced with a leftover {{dbInconsistentMarker}}
file in the DB directory, which makes {{HAUtils.replaceDBWithCheckpoint}} fail
and restore the old DB. The kill case was not run. For a bootstrapping SCM this
needs the leader to have added it to the Raft group, which happens within
moments of the acknowledgement, long before a copy of a real DB ends; when the
injected immediate failure came first, the bootstrap did not complete instead
(not followed further).
# The SCM is restarted.
h3. Impact
* The restarted SCM never applies another transaction and never starts its
datanode server, with no error beyond the periodic "Deferred datanode-server
start is still waiting for follower catch-up" warning. Nothing recovers it
automatically.
* A graceful stop of that SCM never completes ({{StateMachineUpdater}} waits to
reach the commit index), and its updater thread spins on one core.
* Its Raft log is complete, so it can become leader. After a leadership
transfer to it (admin {{transferLeadership}}), it was the Ratis leader and no
SCM reported itself as a ready leader for the 20 s observed, so the SCM service
was unavailable. I did not run an election that picks it without the transfer.
* Recovery needs an operator: kill it, remove its SCM DB and Ratis storage, and
bootstrap it again (argued, not run).
h3. Reproduction
PASS on unmodified source, default configuration.
[^TestBugMC1FailedInstallWedgesFollower.java]
({{hadoop-ozone/integration-test}}) starts three SCMs with one inactive, places
the marker file in the inactive SCM's DB directory, starts it while the leader
keeps allocating containers, waits until it shuts down after the failed
install, removes the file and restarts it. Result: SNAPSHOT_INSTALLED and the
leader moving to the next index logged before the install ran; after the
restart applied index -1 before and after, log start 90, commit 192 against
leader applied 191, 0 install notifications, datanode server not started, 0 of
80 containers, about 9.7 s of updater CPU in 10 s; after a leadership transfer
to it, it was the Ratis leader and no SCM was a ready leader; a graceful stop
still running after 20 s. In this JVM the restarted SCM gets new datanode and
gRPC ports, because the stopped instance keeps them bound; a process exit
releases them. If the injected failure comes before the leader has added the
SCM to the Raft group, the test aborts and has to be rerun. A passing test
means the defect is present.
h3. Suggested fix
SCM cannot undo the purge and the acknowledgement from {{reinitialize}}.
Recommended: install the checkpoint inside the notification future, as
{{OzoneManagerStateMachine.notifyInstallSnapshotFromLeader}} does, and fail the
future on any error, so Ratis neither purges nor acknowledges a failed install.
This reverses the deferral chosen in HDDS-5173 and needs the updater to be
quiesced during the swap. It should not copy OM's handling of a failed install,
which completes the future with null (HDDS-11470). As a safety net, SCM should
check at startup whether its applied index is below the start of its Raft log
and fail with a clear message; that also catches the same state reached through
other paths, such as a transaction buffer init failure in {{reinitialize}} or a
reload failure that rolls back to the old DB.
Alternatives: change Ratis to purge and reply SNAPSHOT_INSTALLED only after
{{reinitialize}} has succeeded, or persist the pending checkpoint before
acknowledging and install it at startup.
h3. Patch
[^MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch],
against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63, test only. It also applies to
master at 52812559dad.
It adds {{testSCMCatchesUpAfterFailedCheckpointInstall}} to
{{TestStorageContainerManagerHA}}. A follower SCM that is a voting member falls
behind the start of the leader's log (log purge enabled, one snapshot requested
on the leader), and when it restarts its install fails once in
{{installCheckpoint}} with the old DB in place (its metadata store, set through
{{SCMConfigurator.setMetadataStore}}, refuses to stop once). After another
restart the follower must catch up with the leader. On the current code it
fails with "Timed out waiting for condition": the follower stays at applied
index 42 while the leader is at 1475 and the follower's log starts at 1261, and
the teardown then takes several minutes (about 5) because the wedged SCM's stop
does not complete. It is meant to land with the fix. A member follower is used
instead of the bootstrap case of the reproduction because an injected failure
is immediate, and for a bootstrapping SCM it can come before the leader has
added the SCM to the Raft group.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a test proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
