Siyao Meng created HDDS-16929:
---------------------------------
Summary: SCM follower stops applying the Raft log, while reporting
RUNNING, if the transaction buffer init fails after a checkpoint install
Key: HDDS-16929
URL: https://issues.apache.org/jira/browse/HDDS-16929
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-5-fail-reinitialize-when-transaction-buffer-init-fails.patch,
TestBugMC5BufferInitFailureStaleApplied.java
h3. Mechanism
{{SCMStateMachine.reinitialize}} installs the downloaded checkpoint and then
calls {{transactionBuffer.init()}} and {{setLastAppliedTermIndex}} inside a try
block whose catch only logs "Failed to unpause". If {{init()}} throws, the
state machine still resumes with the transaction info and latest snapshot from
before the install. Ratis {{StateMachineUpdater.reload}} then takes that old
snapshot index (-1 for a new SCM) as the applied index. Ratis purged the
follower's log up to the checkpoint index when it acknowledged the install, so
the entry after the applied index no longer exists and the updater never
applies anything again. The leader does not send another install notification,
because the follower's log is consistent with its own.
h3. Trigger
# A follower SCM installs the leader's checkpoint through the normal Ratis
install notification (for example a new SCM bootstrapping, or a lagging
follower with {{ozone.scm.ha.ratis.log.purge.enabled}}).
# {{SCMHADBTransactionBufferImpl.init}} fails during {{reinitialize}}, for
example a read error on the transaction info key of the newly installed DB.
The second step is unlikely: {{installCheckpoint}} reads and decodes the same
key from the checkpoint moments earlier, so it needs a transient read error at
that instant. In the reproduction it is injected.
h3. Impact
* The follower reports RUNNING but never applies another entry and never starts
its datanode server. It is missing every container created after the install.
* A graceful stop of that SCM never completes, because {{StateMachineUpdater}}
waits to reach the commit index; the updater thread also spins on one core. It
has to be killed. Argued, not run: after a restart it opens the installed DB
and catches up, unless it was elected leader in the meantime, because the flush
on becoming leader ({{SCMStateMachine.notifyLeaderChanged}}) writes the stale
transaction info into that DB.
* Argued, not run for this trigger: its log is complete, so it can be elected
leader, and then no SCM is a ready leader. I ran that outcome for the same
applied index below log start state reached through a failed checkpoint copy.
h3. Reproduction
PASS on unmodified source. [^TestBugMC5BufferInitFailureStaleApplied.java]
({{hadoop-ozone/integration-test}}) uses the cluster shape of
{{TestSCMInstallSnapshotWithHA}} (three SCMs, one started late, so it installs
the leader's checkpoint through Ratis; log purge enabled, purge gap 5, snapshot
threshold 5) and injects one {{RocksDatabaseException}} into the transaction
info read that {{SCMHADBTransactionBufferImpl.init}} does inside
{{reinitialize}}, with a Mockito spy on the follower's {{SCMMetadataStore}}.
Result: lifecycle RUNNING, DB transaction index 202, Ratis applied index -1,
log start 203, commit 238, 16 of 16 later containers missing, datanode server
not started, about 4.9 s of updater CPU in 5 s, graceful stop still running
after 20 s. A passing test means the defect is present.
h3. Patch
[^MC-5-fail-reinitialize-when-transaction-buffer-init-fails.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad. It conflicts textually with the patch attached to HDDS-16923 in
{{SCMStateMachine}} and {{TestSCMStateMachine}}; whichever lands second needs a
rebase.
The patch removes the catch, so the failure propagates from {{reinitialize}}
like a failure of {{installCheckpoint}} already does. Ratis then closes the
server and SCM shuts down; on restart it opens the installed DB, whose
transaction info is the checkpoint index, and catches up. With the patch the
reproduction fails at its first assertion because the follower SCM stops, and a
restart of that follower applied up to the leader's index and started its
datanode server (local variant of the reproduction, not attached).
Covered by {{testReinitializeFailsWhenTransactionBufferInitFails}} and
{{testReinitializeResumesAtInstalledCheckpoint}} in the existing
{{TestSCMStateMachine}}. Without the change the first fails with "Expected
java.io.IOException to be thrown, but nothing was thrown." With it
{{TestSCMStateMachine}}, {{TestSCMHAManagerImpl}} and
{{TestSCMHADBTransactionBufferImpl}} (31 tests) and the mini cluster suites
{{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests) pass,
checkstyle is clean.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
