Siyao Meng created HDDS-16932:
---------------------------------
Summary: SCM follower rejects every append from a new leader when
leadership moves while it downloads a checkpoint that then succeeds
Key: HDDS-16932
URL: https://issues.apache.org/jira/browse/HDDS-16932
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
When the leader notifies a follower SCM to install a snapshot,
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} downloads the leader's
checkpoint and completes its future with the checkpoint TermIndex. Ratis
(3.3.1) {{SnapshotInstallationHandler}} then keeps its install in progress
index set; the install is applied and the index cleared only when the next
install notification arrives. While the index is set,
{{RaftServerImpl.checkInconsistentAppendEntries}} rejects every append ("Failed
appendEntries as snapshot (...) installation is in progress"). Ratis never
clears or applies a pending install on a leader change. A new leader sends an
install notification to a follower that is already a voting member only when
the follower's next index is below the start of its own log
({{LogAppender.shouldInstallSnapshot}}). So if leadership moves during the
download to an SCM whose log still contains the follower's next entry, and the
download then succeeds, the follower rejects all of the new leader's appends
indefinitely.
HDDS-16899 reports the case where the download fails and the future completes
with null; its patch fails the future instead, which makes Ratis clear the
index at once. It does not cover a successful download. The null results of
{{SCMHAManagerImpl.verifyCheckpointFromLeader}} (a checkpoint that is not ahead
of the follower, or an error while reading it) leave the index set the same way
and are not covered either (the second is listed under "Not covered" in
HDDS-16899).
h3. Trigger
# {{ozone.scm.ha.ratis.log.purge.enabled=true}} (default false). Follower F is
behind; leader L1 has purged its log past F's next index, while another SCM L2
has not (for example after a snapshot request to L1 only, or different snapshot
timing).
# F gets an install notification from L1 and downloads the checkpoint. With a
large DB this takes minutes.
# L1 stays up but loses leadership to L2 during the download: an admin
leadership transfer, or an election after a GC pause of L1 or a partition
between L1 and L2. The leader side of the download
({{InterSCMGrpcService.download}}) does not check leadership, so the download
succeeds.
h3. Impact
* F stays at its old applied index and rejects every append from L2. It
recovers on a restart, or (argued) when a leader whose log is purged past F's
next index takes over or L2 purges past it.
* With L1 up, L1 and L2 form a majority, so the SCM service stays available.
The ring has no failure margin left: if L1 or L2 then fails, the remaining two
SCMs cannot commit (argued).
h3. Reproduction
The reproduction is not attached. It is a mini cluster test with three SCMs,
purge enabled with gap 5 and automatic snapshots off. F stops, containers are
allocated, and only L1 takes a snapshot, so L1's log starts after F's next
index and L2's does not. F's download is routed through a proxy that holds the
connection; while it is held, leadership is transferred from L1 to L2 with
{{StorageContainerLocationClient.transferLeadership}}; then the proxy is
released and the download succeeds (F logs "StateMachine successfully installed
snapshot index 352" after L2 became leader at term 3). Results 30 s after the
transfer:
* On b0aa6475b7856d2c7f467dd3db12b9a1dd116f63 (PASS, the defect is present,
92.7 s): L2 leader and ready, L1 running, leader applied index 395, F applied
index 42, 3258 rejected appends logged on F when the download ended and 3275 30
s later, 1 install notification to F.
* With the patch attached to HDDS-16899 applied (PASS, 96.2 s): the same,
leader applied index 396, F applied index 42, 2358 and 2374 rejected appends, 1
install notification.
h3. Suggested fix
In SCM: after the download, fail the future instead of completing it when the
leader changed while the download ran (the leader in the notification's role
info is no longer the current leader). Ratis then clears the index at once, F
accepts L2's appends from its next index, and L2 notifies F if its log no
longer covers that index. This also needs the {{verifyCheckpointFromLeader}}
null results to fail the future. Because Ratis never clears or applies a
pending install on a leader change, a Ratis change that does so would close the
case fully, for SCM and for OM, which uses the same notification mechanism
(argued, not run for OM).
No patch is attached. The code involved is unchanged on master at 52812559dad.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS issues
and apache/ozone pull requests for duplicates before filing. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
