[
https://issues.apache.org/jira/browse/HDDS-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Parvatikar reassigned HDDS-16929:
----------------------------------------
Assignee: Anurag Parvatikar
> SCM follower stops applying the Raft log, while reporting RUNNING, if the
> transaction buffer init fails after a checkpoint install
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16929
> URL: https://issues.apache.org/jira/browse/HDDS-16929
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Anurag Parvatikar
> Priority: Minor
> Attachments:
> MC-5-fail-reinitialize-when-transaction-buffer-init-fails.patch,
> TestBugMC5BufferInitFailureStaleApplied.java
>
>
> h3. Mechanism
> {{SCMStateMachine.reinitialize}} installs the downloaded checkpoint and then
> calls {{transactionBuffer.init()}} and {{setLastAppliedTermIndex}} inside a
> try block whose catch only logs "Failed to unpause". If {{init()}} throws,
> the state machine still resumes with the transaction info and latest snapshot
> from before the install. Ratis {{StateMachineUpdater.reload}} then takes that
> old snapshot index (-1 for a new SCM) as the applied index. Ratis purged the
> follower's log up to the checkpoint index when it acknowledged the install,
> so the entry after the applied index no longer exists and the updater never
> applies anything again. The leader does not send another install
> notification, because the follower's log is consistent with its own.
> h3. Trigger
> # A follower SCM installs the leader's checkpoint through the normal Ratis
> install notification (for example a new SCM bootstrapping, or a lagging
> follower with {{ozone.scm.ha.ratis.log.purge.enabled}}).
> # {{SCMHADBTransactionBufferImpl.init}} fails during {{reinitialize}}, for
> example a read error on the transaction info key of the newly installed DB.
> The second step is unlikely: {{installCheckpoint}} reads and decodes the same
> key from the checkpoint moments earlier, so it needs a transient read error
> at that instant. In the reproduction it is injected.
> h3. Impact
> * The follower reports RUNNING but never applies another entry and never
> starts its datanode server. It is missing every container created after the
> install.
> * A graceful stop of that SCM never completes, because
> {{StateMachineUpdater}} waits to reach the commit index; the updater thread
> also spins on one core. It has to be killed. Argued, not run: after a restart
> it opens the installed DB and catches up, unless it was elected leader in the
> meantime, because the flush on becoming leader
> ({{SCMStateMachine.notifyLeaderChanged}}) writes the stale transaction info
> into that DB.
> * Argued, not run for this trigger: its log is complete, so it can be elected
> leader, and then no SCM is a ready leader. I ran that outcome for the same
> applied index below log start state reached through a failed checkpoint copy.
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC5BufferInitFailureStaleApplied.java]
> ({{hadoop-ozone/integration-test}}) uses the cluster shape of
> {{TestSCMInstallSnapshotWithHA}} (three SCMs, one started late, so it
> installs the leader's checkpoint through Ratis; log purge enabled, purge gap
> 5, snapshot threshold 5) and injects one {{RocksDatabaseException}} into the
> transaction info read that {{SCMHADBTransactionBufferImpl.init}} does inside
> {{reinitialize}}, with a Mockito spy on the follower's {{SCMMetadataStore}}.
> Result: lifecycle RUNNING, DB transaction index 202, Ratis applied index -1,
> log start 203, commit 238, 16 of 16 later containers missing, datanode server
> not started, about 4.9 s of updater CPU in 5 s, graceful stop still running
> after 20 s. A passing test means the defect is present.
> h3. Patch
> [^MC-5-fail-reinitialize-when-transaction-buffer-init-fails.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad. It conflicts textually with the patch attached to HDDS-16923 in
> {{SCMStateMachine}} and {{TestSCMStateMachine}}; whichever lands second needs
> a rebase.
> The patch removes the catch, so the failure propagates from {{reinitialize}}
> like a failure of {{installCheckpoint}} already does. Ratis then closes the
> server and SCM shuts down; on restart it opens the installed DB, whose
> transaction info is the checkpoint index, and catches up. With the patch the
> reproduction fails at its first assertion because the follower SCM stops, and
> a restart of that follower applied up to the leader's index and started its
> datanode server (local variant of the reproduction, not attached).
> Covered by {{testReinitializeFailsWhenTransactionBufferInitFails}} and
> {{testReinitializeResumesAtInstalledCheckpoint}} in the existing
> {{TestSCMStateMachine}}. Without the change the first fails with "Expected
> java.io.IOException to be thrown, but nothing was thrown." With it
> {{TestSCMStateMachine}}, {{TestSCMHAManagerImpl}} and
> {{TestSCMHADBTransactionBufferImpl}} (31 tests) and the mini cluster suites
> {{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests)
> pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS
> issues and apache/ozone pull requests for duplicates before filing. The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
