[
https://issues.apache.org/jira/browse/HDDS-16930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16930:
------------------------------
Attachment:
CR-6-flush-scm-transaction-buffer-before-checkpoint-install.patch
> Follower SCM exits when it installs a Ratis snapshot after applying a delete
> transaction removal that its buffer has not flushed yet
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16930
> URL: https://issues.apache.org/jira/browse/HDDS-16930
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-6-flush-scm-transaction-buffer-before-checkpoint-install.patch,
> TestBugCR6InstallAfterDeleteTxRemoval.java
>
>
> h3. Mechanism
> {{DeletedBlockLogStateManagerImpl.removeTransactionsFromDB}} adds the removed
> transaction IDs to the in memory {{deletingTxIDs}} and queues the deletes in
> the SCM transaction buffer. Only {{onFlush()}}, called when the buffer is
> flushed, clears the set. {{DeletedBlockLogStateManagerImpl.reinitialize}}
> runs {{Preconditions.checkArgument(deletingTxIDs.isEmpty())}}, and its
> comment says that a flush is called before the reload. No flush happens on
> the install path: Ratis calls {{SCMStateMachine.pause}}, which only changes
> the lifecycle state, and then {{SCMStateMachine.reinitialize}}, and
> {{SCMHAManagerImpl.installCheckpoint}} stops the store, replaces the DB and
> calls {{startServices}}, which calls {{DeletedBlockLog.reinitialize}}. If the
> set is not empty the check throws. The rollback puts the old DB back and
> reloads again, the set is still not empty, the same check throws, and SCM
> calls {{exitSystem(1)}} ("Failed to reload SCM state and instantiate
> services.").
> On a follower the buffer is flushed by a Ratis snapshot (every
> {{ozone.scm.ha.ratis.snapshot.threshold}} entries, default 1000), at the
> pending operation limit (default 10000), when it becomes leader, when it
> serves a checkpoint download ({{InterSCMGrpcService.download}}) and when it
> applies a replicated {{StatefulServiceStateManager.saveConfiguration}}. The
> periodic flush ({{ozone.scm.ha.dbtransactionbuffer.flush.interval}}) runs on
> the leader only. So after a follower applies a removal (sent for every delete
> acknowledgement from datanodes and by the scanner cleanup), the set stays non
> empty until the next of these flushes, which for a follower that stops
> applying may never come.
> h3. Trigger
> # {{ozone.scm.ha.ratis.log.purge.enabled}} is true (default false) and the
> leader has purged its log past the follower's position. The leader purges
> only after {{ozone.scm.ha.ratis.log.purge.gap}} entries (default 1000000)
> past its last purge.
> # The follower applied a delete transaction removal since its last buffer
> flush, then fell behind the purged log while running (for example paused or
> cut off from the leader).
> # Ratis asks the follower to install a snapshot.
> With default settings Ratis purges only below every peer's commit index, so a
> running follower is not sent a snapshot, and a newly bootstrapped SCM has an
> empty set. I did not find a default configuration route.
> h3. Impact
> * The follower SCM process exits during the install (run with the exit
> replaced by a recording exit manager; the process exit itself not run).
> * A restart does not recover it. In a run with the exit replaced, stopping
> and restarting that SCM right after the failed install failed: the stop could
> not flush the transaction buffer ({{RocksDatabaseException: Rocks Database is
> closed}} from {{SCMHADBTransactionBufferImpl.flush}}), and the restarted SCM
> could not open the restored {{scm.db}} (RocksDB reported a missing {{.sst}}
> file and a possibly corrupted MANIFEST). The rollback restores the old DB
> while the store opened by the failed reload is still open, which is
> HDDS-16921: with the patch of that issue applied, the same restart opened the
> DB (2 of 2 runs). A real process exit was not run.
> * Through the real Ratis path the failure comes after Ratis has already
> purged the follower's log up to the snapshot index and replied
> SNAPSHOT_INSTALLED to the leader ({{SnapshotInstallationHandler}} does both
> before {{reinitialize}} runs on the state machine updater thread). So even
> with a DB that opens, the restarted follower is left in the wedge reported in
> HDDS-16928 (code reading for this trigger; the wedge itself is run there). HA
> redundancy stays reduced until that SCM is repaired.
> h3. Reproduction
> PASS on the unmodified source, default configuration.
> [^TestBugCR6InstallAfterDeleteTxRemoval.java]
> ({{hadoop-ozone/integration-test}}) starts three SCMs, has the leader add one
> delete transaction and remove it again ({{removeTransactions}}, the request a
> datanode acknowledgement sends), and checks that both followers applied the
> removal without a flush. Both followers then install the leader's checkpoint
> through the state machine calls Ratis makes ({{pause}},
> {{setInstallingSnapshotData}}, {{reinitialize}}); the log purge notification
> itself is not exercised. The control follower flushes its buffer first and
> installs fine. The other one fails at
> {{DeletedBlockLogStateManagerImpl.reinitialize}} in the reload and in the
> rollback reload, and the exit is requested with the
> {{IllegalArgumentException}}. A passing test means the defect is present.
> With the patch both installs succeed and the test fails at its
> {{assertThatThrownBy}}.
> h3. Patch
> [^CR-6-flush-scm-transaction-buffer-before-checkpoint-install.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> {{SCMHAManagerImpl.installCheckpoint}} flushes the transaction buffer before
> it stops the store. This clears {{deletingTxIDs}} through {{onFlush()}} as
> the comments in {{DeletedBlockLogStateManagerImpl.reinitialize}} and
> {{SCMDeletedBlockTransactionStatusManager.reinitialize}} already assume, and
> it leaves the old DB, which a rollback restores, consistent with the
> follower's applied index. Ratis calls {{reinitialize}} on its state machine
> updater thread, so no transaction is applied during the flush. Resetting the
> set in {{reinitialize}} instead would also avoid the crash, but it drops the
> check and leaves a restored old DB behind the applied index.
> Covered by {{testInstallCheckpointAfterDeleteTransactionRemoval}} in the
> existing {{TestSCMInstallSnapshotWithHA}}: the follower applies a removal
> that is not flushed yet and then installs the leader's checkpoint. Without
> the change the install fails with the {{IllegalArgumentException}} above in
> both reloads. With it {{TestSCMInstallSnapshotWithHA}},
> {{TestSCMInstallSnapshot}}, {{TestSCMSnapshot}}, {{TestScmHAFinalization}},
> {{TestSecretKeySnapshot}} and the server-scm {{TestDeletedBlockLog}},
> {{TestSCMHADBTransactionBufferImpl}}, {{TestSCMHAManagerImpl}} and
> {{TestSCMStateMachine}} pass, and checkstyle is clean.
> {{TestSCMInstallSnapshotWithHA}} is tagged flaky (HDDS-5631), so the new test
> does not run in the regular CI checks.
> Found by code review of the SCM Ratis snapshot and checkpoint install paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
