Siyao Meng created HDDS-16899:
---------------------------------
Summary: SCM bootstrap falls back to Raft log replay when the
checkpoint download fails, so the new SCM lacks DB state that is not in the log
Key: HDDS-16899
URL: https://issues.apache.org/jira/browse/HDDS-16899
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-retry-failed-scm-bootstrap-checkpoint-download.patch,
TestBugMC3BootstrapReplayAfterFailedCheckpoint.java
h3. Mechanism
A bootstrapping SCM is meant to get its DB from the leader's checkpoint
(HDDS-5016). Ratis (3.3.1) keeps notifying a bootstrapping follower to install
a snapshot only until it gets a reply other than {{IN_PROGRESS}}. When the
download fails, {{SCMHAManagerImpl.downloadCheckpointFromLeader}} logs the
exception and returns null, and
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} returns null as well. It
does the same when {{getSecretKeysFromLeader}} fails. Ratis reports a null
result to the leader as {{SNAPSHOT_UNAVAILABLE}}. If the leader's Raft log
still starts at index 0, the leader then appends the whole log, and the new SCM
applies it over its own freshly initialized DB and joins the ring.
Whatever the leader's DB holds that the log does not explain is then missing on
the new SCM for good. For {{sequenceIdTable}} this is visible directly. Every
SCM start runs {{SequenceIdGenerator.upgradeToSequenceId}}, which writes absent
rows from local data, and each replicated {{allocateBatch}} is a compare and
set that every replica judges against its own row. If the new SCM's row differs
from what the leader's row was at the start of the log, the two SCMs accept
different requests and keep different rows. Only a WARN ("Failed to allocate a
batch for ...") is logged.
h3. Trigger
# The leader's Raft log still starts at index 0. That is the case for the SCM
the cluster was initialized on until its first log purge
({{ozone.scm.ha.ratis.log.purge.gap}} defaults to 1000000 entries). A leader
that was bootstrapped itself has a log that starts after the checkpoint it
installed, and it keeps notifying the new SCM instead (run).
# The leader's DB holds state that is not in the Raft log. Two cases were
looked at: (a) a cluster upgraded from SCM without Ratis (HDDS-12031
initializes Ratis on the existing DB), where everything written before the
upgrade is in the DB only, and (b) the {{localId}} row, whose seed depends on
the year of the SCM's first start and, since HDDS-14337, on the release.
# A new SCM is bootstrapped and its first checkpoint download fails. This need
not be a transient error: when the leader's inter SCM gRPC service
({{ozone.scm.grpc.port}}) cannot be reached while its Ratis port can, every
bootstrap joins by replay after one failed download (run with that service
stopped on the leader: the new SCM was added within a second of the failure).
The other fault used in the runs is a Ratis snapshot directory on the new SCM
that is not writable.
h3. Impact
* Case (a), run with a {{containerId}} row written directly into the DB of the
existing SCMs, not with a real upgrade: the new SCM ended with a lower
{{containerId}} row than the leader (1000 against 1005). After leadership was
transferred to it, {{allocateContainer}} for another owner returned an existing
container, because the ID it drew was already in use. The rows were still
different after leadership went back. On a really upgraded cluster the new SCM
would also lack every container, pipeline and deleted block transaction from
before Ratis was enabled (argued from the code, not run).
* Case (b), run: the {{localId}} rows stay different for good. No ID was issued
twice. The two seeds are at least one day (5.66e12 IDs) apart and each SCM
allocates from its own row.
* On a cluster that never ran without Ratis, the {{containerId}}, {{delTxnId}}
and {{CertificateId}} rows were equal after the replay (run on an unsecured
cluster, a secure cluster was not run).
h3. Reproduction
PASS in each of my three runs, unmodified source, default Ratis settings.
[^TestBugMC3BootstrapReplayAfterFailedCheckpoint.java]
({{hadoop-ozone/integration-test}}) starts an SCM HA mini cluster with one SCM
left inactive, makes that SCM's Ratis snapshot directory read only until its
first download attempt has failed, and then starts it. The first test writes a
{{containerId}} row of 5 into the existing SCMs before the first container ID
is allocated and shows the lower row on the new SCM and the existing container
returned by {{allocateContainer}} after a leadership transfer. The second test
gives the leader the {{localId}} seed of an older release and shows equal
{{containerId}}, {{delTxnId}} and {{CertificateId}} rows, different {{localId}}
rows and disjoint local IDs. Both tests need the SCM the cluster was
initialized on to lead when the new SCM joins and fail with a message saying so
if it does not (that did not happen). A passing test means the defect is
present.
h3. Suggested fix
Fail the install snapshot attempt when the checkpoint cannot be obtained, so
that the leader asks again and the log is not replayed (the attached patch).
The patch retries without a bound of its own, as an SCM behind a purged leader
log already does. The alternative for a bootstrapping SCM is a bounded number
of attempts that ends the bootstrap with an error naming the failed checkpoint
download. That needs a policy and probably a configuration key, so it is not in
the patch.
h3. Patch
[^MC-3-retry-failed-scm-bootstrap-checkpoint-download.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
f650a22f3e3.
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} now fails its future when
the checkpoint download or the secret key fetch fails, instead of returning
null. Ratis answers a failed future with {{IN_PROGRESS}} and the leader
notifies the follower again, so the download is retried and the log is not
replayed. The failure is reported after a pause of one second, because the
leader notifies again at once, and the downloaded checkpoint is removed when
the secret key fetch fails. {{SCMSnapshotProvider.getSCMDBSnapshot}} no longer
sets the interrupt flag of the calling thread for a failed download that was
not an interrupt. With the flag set the pause was skipped (2048 failed
downloads in 10 s, against 10 with the change). Null is still returned when the
downloaded checkpoint is not ahead of the follower. Only the follower's install
path changes, not how a logged request is applied.
Behaviour change, as run on a mini cluster with the leader's inter SCM gRPC
service stopped and {{hdds.scmclient.rpc.timeout}} lowered to 20s: a bootstrap
whose download keeps failing no longer joins by log replay. The new SCM retries
once per second and logs five ERROR lines per attempt, one of which says that
it cannot join without the leader's checkpoint. The start does not wait
forever. When its {{addSCM}} call times out, the client retries 15 times at
intervals of 2 s, the leader refuses each retry because the first
reconfiguration is still in progress, and the start fails with
{{ReconfigurationInProgressException}} (after 50 s in the run). With the
default timeout of 15m that is about 15.5 minutes (from the configuration
defaults, not run). The {{ozone scm}} process then exits (read from
{{StorageContainerManagerStarter}}, not run) and has to be started again once
the download can work.
Until then the leader stays in the Ratis staging state. In the run it notified
the new SCM about 200 times per second, each notification logged at INFO on
both SCMs, and 16 s after the new SCM had been stopped it gave up the pending
reconfiguration with {{ReconfigurationTimeoutException}}. Read from the Ratis
code and not run: a leadership transfer is refused in that state, and the
leader checks the progress of the new peer in place of its periodic leadership
check. The repeated notifications are not new, a leader whose log does not
start at index 0 already sends them while the download fails (seen on the
unfixed code with a leader that had been bootstrapped itself). For a leader
whose log starts at index 0 they replace a join that took one attempt. An SCM
behind a purged leader log already retried a failing download without end, now
it does so once per second. SCMs that have already joined by replay are not
repaired.
Not covered: {{SCMHAManagerImpl.verifyCheckpointFromLeader}} returns null both
when the downloaded checkpoint is not ahead of this SCM and when reading it
throws, and the second case still ends in the replay. It is left alone because
the existing {{TestSCMHAManagerImpl.testAddSCM}} adds its second SCM through
exactly that exit (its mocked checkpoint cannot be read), so closing it means
reworking that test.
Covered by {{testBootstrapRetriesFailedCheckpointDownload}} in the existing
{{TestStorageContainerManagerHA}}, by
{{testFailedCheckpointDownloadFailsInstallSnapshotAttemptAfterPause}},
{{testSecretKeyFailureFailsInstallSnapshotAttempt}} and
{{testCheckpointNotAheadOfFollowerIsReportedAsUnavailable}} in
{{TestSCMStateMachine}}, and by
{{testFailedDownloadDoesNotInterruptCallingThread}} in
{{TestSCMSnapshotProvider}}. Without the change the first fails with "expected:
1005L but was: 1000L", the second and third because the future completes with
null, and the last because the thread is left interrupted. The fourth is the
negative case and passes either way. With the change the SCM HA unit suites
around the state machine, snapshot provider, sequence ID generator and inter
SCM gRPC service (40 tests) and {{TestStorageContainerManagerHA}},
{{TestSCMInstallSnapshot}} and {{TestSCMInstallSnapshotWithHA}} (9 tests) pass,
checkstyle is clean. {{TestSecretKeySnapshot}} could not be run in my
environment (its secure mini cluster does not start, with or without the
change). With the patch both tests of the reproduction fail, because the new
SCM then has the leader's rows.
Found by TLA+ model checking and code review of the SCM sequence ID generator
and its startup and checkpoint paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. HDDS-5016 states the
requirement that a bootstrapping SCM always downloads the checkpoint, this
issue is the case where it does not. The attached patch is a proposal for
review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
