Siyao Meng created HDDS-16924:
---------------------------------
Summary: SCM that installed a Ratis snapshot in process keeps
serving checkpoints from the closed DB, so downloads from it fail once it is
leader
Key: HDDS-16924
URL: https://issues.apache.org/jira/browse/HDDS-16924
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-serve-scm-checkpoints-from-current-db-store.patch,
TestBugMC4CheckpointServiceClosedStore.java
h3. Mechanism
{{InterSCMGrpcService}} takes the transaction info table and an
{{SCMDBCheckpointProvider}} from {{scm.getScmMetadataStore().getStore()}} once,
in its constructor, and the service is created once per process in the
{{SCMHAManagerImpl}} constructor. {{SCMHAManagerImpl.installCheckpoint}} closes
that store ({{stopServices}}, which runs {{SCMMetadataStoreImpl.stop}}) and
{{startServices}} opens a new one and points the managers at it, but the gRPC
service is not updated. From then on every {{InterSCMGrpcService.download}}
reads the closed RocksDB and fails with {{RocksDatabaseException: Rocks
Database is closed}}. Only a restart of that SCM clears it.
The javadoc of {{SCMHAManagerImpl.reloadSCMState}} says that all classes which
use or store the metadata manager should be updated with the new instance; this
one is missed.
h3. Trigger
# An SCM installs a checkpoint in process. With default settings this happens
on every SCM added with {{--bootstrap}}: Ratis (3.3.1) sends a snapshot
notification to a bootstrapping follower once, independent of log purging, and
the new SCM downloads and installs the leader's checkpoint.
# That SCM becomes leader before its next restart (failover or
{{transferLeadership}}).
# Something downloads a checkpoint from it.
h3. Impact
* Every checkpoint download from that leader fails with gRPC {{UNKNOWN}}, and
the leader logs "Error streaming SCM DB checkpoint" with "Rocks Database is
closed" (run).
* Consumers of that download: an SCM being bootstrapped while this SCM is
leader (see HDDS-16899 for how a bootstrap behaves when the download fails),
Recon's SCM DB snapshot sync
({{StorageContainerServiceProviderImpl.getSCMDBSnapshot}} downloads through
{{InterSCMGrpcClient}} from the leader), and, with
{{ozone.scm.ha.ratis.log.purge.enabled}}, any follower that lags behind the
purged log. These are code reading; the run shows the failing download by the
call a follower makes in {{SCMStateMachine.notifyInstallSnapshotFromLeader}}.
* It lasts until that SCM restarts. Losing leadership does not clear it:
downloads then go to another SCM, and they fail again whenever this SCM is
leader before a restart.
* The SCM's own data is not affected. An SCM bootstrapped while this one is
leader is subject to HDDS-16899 and can end up without DB state that is not in
the Raft log.
{{SCMDBCheckpointServlet}} ({{/dbCheckpoint}}) stores the store the same way,
but Jetty initializes it on its first request, so it is affected only if it
served a request before the install, and no Ozone component uses SCM's
{{/dbCheckpoint}}. It is not changed by the patch (code reading only). The same
pattern in the OM checkpoint servlets is HDDS-16878.
{{SCMDBCheckpointServlet.init}} passes the store to the same
{{DBCheckpointServlet.initialize}}, so the SCM servlet can follow the fix made
there.
h3. Reproduction
PASS on the unmodified source, default configuration.
[^TestBugMC4CheckpointServiceClosedStore.java]
({{hadoop-ozone/integration-test}}) starts three SCMs with one held back,
starts the held back SCM C (it bootstraps and installs the leader's checkpoint
in process, and its original store is closed), checks that a download from
leader A works, transfers leadership to C and downloads from C through the
follower's {{SCMSnapshotProvider}}. The download from C fails and C logs "Rocks
Database is closed". A passing test means the defect is present. With the patch
the download from C succeeds and the test fails at its {{assertThatThrownBy}}.
h3. Patch
[^MC-4-serve-scm-checkpoints-from-current-db-store.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
{{InterSCMGrpcService.download}} looks up the current store on each request and
builds the transaction info table and the {{SCMDBCheckpointProvider}} from it,
so the service holds no reference to a store that an install can close. Nothing
changes for an SCM that never installed a checkpoint.
Covered by an assertion added to
{{TestSCMInstallSnapshot.testInstallCheckPoint}}: after the in process install
the SCM serves a checkpoint, and that checkpoint carries the installed
transaction index. Without the change the download fails with the error above.
With it {{TestSCMInstallSnapshot}}, {{TestSCMInstallSnapshotWithHA}},
{{TestSCMSnapshot}}, {{TestScmHAFinalization}}, {{TestSecretKeySnapshot}} and
the server-scm {{TestInterSCMGrpcProtocolService}}, {{TestInterSCMGrpcClient}}
and {{TestSCMSnapshotProvider}} pass, and checkstyle is clean.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
