Siyao Meng created HDDS-16930:
---------------------------------
Summary: Follower SCM exits when it installs a Ratis snapshot
after applying a delete transaction removal that its buffer has not flushed yet
Key: HDDS-16930
URL: https://issues.apache.org/jira/browse/HDDS-16930
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-6-flush-scm-transaction-buffer-before-checkpoint-install.patch,
TestBugCR6InstallAfterDeleteTxRemoval.java
h3. Mechanism
{{DeletedBlockLogStateManagerImpl.removeTransactionsFromDB}} adds the removed
transaction IDs to the in memory {{deletingTxIDs}} and queues the deletes in
the SCM transaction buffer. Only {{onFlush()}}, called when the buffer is
flushed, clears the set. {{DeletedBlockLogStateManagerImpl.reinitialize}} runs
{{Preconditions.checkArgument(deletingTxIDs.isEmpty())}}, and its comment says
that a flush is called before the reload. No flush happens on the install path:
Ratis calls {{SCMStateMachine.pause}}, which only changes the lifecycle state,
and then {{SCMStateMachine.reinitialize}}, and
{{SCMHAManagerImpl.installCheckpoint}} stops the store, replaces the DB and
calls {{startServices}}, which calls {{DeletedBlockLog.reinitialize}}. If the
set is not empty the check throws. The rollback puts the old DB back and
reloads again, the set is still not empty, the same check throws, and SCM calls
{{exitSystem(1)}} ("Failed to reload SCM state and instantiate services.").
On a follower the buffer is flushed by a Ratis snapshot (every
{{ozone.scm.ha.ratis.snapshot.threshold}} entries, default 1000), at the
pending operation limit (default 10000), when it becomes leader, when it serves
a checkpoint download ({{InterSCMGrpcService.download}}) and when it applies a
replicated {{StatefulServiceStateManager.saveConfiguration}}. The periodic
flush ({{ozone.scm.ha.dbtransactionbuffer.flush.interval}}) runs on the leader
only. So after a follower applies a removal (sent for every delete
acknowledgement from datanodes and by the scanner cleanup), the set stays non
empty until the next of these flushes, which for a follower that stops applying
may never come.
h3. Trigger
# {{ozone.scm.ha.ratis.log.purge.enabled}} is true (default false) and the
leader has purged its log past the follower's position. The leader purges only
after {{ozone.scm.ha.ratis.log.purge.gap}} entries (default 1000000) past its
last purge.
# The follower applied a delete transaction removal since its last buffer
flush, then fell behind the purged log while running (for example paused or cut
off from the leader).
# Ratis asks the follower to install a snapshot.
With default settings Ratis purges only below every peer's commit index, so a
running follower is not sent a snapshot, and a newly bootstrapped SCM has an
empty set. I did not find a default configuration route.
h3. Impact
* The follower SCM process exits during the install (run with the exit replaced
by a recording exit manager; the process exit itself not run).
* A restart does not recover it. In a run with the exit replaced, stopping and
restarting that SCM right after the failed install failed: the stop could not
flush the transaction buffer ({{RocksDatabaseException: Rocks Database is
closed}} from {{SCMHADBTransactionBufferImpl.flush}}), and the restarted SCM
could not open the restored {{scm.db}} (RocksDB reported a missing {{.sst}}
file and a possibly corrupted MANIFEST). The rollback restores the old DB while
the store opened by the failed reload is still open, which is HDDS-16921: with
the patch of that issue applied, the same restart opened the DB (2 of 2 runs).
A real process exit was not run.
* Through the real Ratis path the failure comes after Ratis has already purged
the follower's log up to the snapshot index and replied SNAPSHOT_INSTALLED to
the leader ({{SnapshotInstallationHandler}} does both before {{reinitialize}}
runs on the state machine updater thread). So even with a DB that opens, the
restarted follower is left in the wedge reported in HDDS-16928 (code reading
for this trigger; the wedge itself is run there). HA redundancy stays reduced
until that SCM is repaired.
h3. Reproduction
PASS on the unmodified source, default configuration.
[^TestBugCR6InstallAfterDeleteTxRemoval.java]
({{hadoop-ozone/integration-test}}) starts three SCMs, has the leader add one
delete transaction and remove it again ({{removeTransactions}}, the request a
datanode acknowledgement sends), and checks that both followers applied the
removal without a flush. Both followers then install the leader's checkpoint
through the state machine calls Ratis makes ({{pause}},
{{setInstallingSnapshotData}}, {{reinitialize}}); the log purge notification
itself is not exercised. The control follower flushes its buffer first and
installs fine. The other one fails at
{{DeletedBlockLogStateManagerImpl.reinitialize}} in the reload and in the
rollback reload, and the exit is requested with the
{{IllegalArgumentException}}. A passing test means the defect is present. With
the patch both installs succeed and the test fails at its
{{assertThatThrownBy}}.
h3. Patch
[^CR-6-flush-scm-transaction-buffer-before-checkpoint-install.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
{{SCMHAManagerImpl.installCheckpoint}} flushes the transaction buffer before it
stops the store. This clears {{deletingTxIDs}} through {{onFlush()}} as the
comments in {{DeletedBlockLogStateManagerImpl.reinitialize}} and
{{SCMDeletedBlockTransactionStatusManager.reinitialize}} already assume, and it
leaves the old DB, which a rollback restores, consistent with the follower's
applied index. Ratis calls {{reinitialize}} on its state machine updater
thread, so no transaction is applied during the flush. Resetting the set in
{{reinitialize}} instead would also avoid the crash, but it drops the check and
leaves a restored old DB behind the applied index.
Covered by {{testInstallCheckpointAfterDeleteTransactionRemoval}} in the
existing {{TestSCMInstallSnapshotWithHA}}: the follower applies a removal that
is not flushed yet and then installs the leader's checkpoint. Without the
change the install fails with the {{IllegalArgumentException}} above in both
reloads. With it {{TestSCMInstallSnapshotWithHA}}, {{TestSCMInstallSnapshot}},
{{TestSCMSnapshot}}, {{TestScmHAFinalization}}, {{TestSecretKeySnapshot}} and
the server-scm {{TestDeletedBlockLog}}, {{TestSCMHADBTransactionBufferImpl}},
{{TestSCMHAManagerImpl}} and {{TestSCMStateMachine}} pass, and checkstyle is
clean. {{TestSCMInstallSnapshotWithHA}} is tagged flaky (HDDS-5631), so the new
test does not run in the regular CI checks.
Found by code review of the SCM Ratis snapshot and checkpoint install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
