[
https://issues.apache.org/jira/browse/HDDS-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16924:
------------------------------
Attachment: TestBugMC4CheckpointServiceClosedStore.java
> SCM that installed a Ratis snapshot in process keeps serving checkpoints from
> the closed DB, so downloads from it fail once it is leader
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16924
> URL: https://issues.apache.org/jira/browse/HDDS-16924
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-4-serve-scm-checkpoints-from-current-db-store.patch,
> TestBugMC4CheckpointServiceClosedStore.java
>
>
> h3. Mechanism
> {{InterSCMGrpcService}} takes the transaction info table and an
> {{SCMDBCheckpointProvider}} from {{scm.getScmMetadataStore().getStore()}}
> once, in its constructor, and the service is created once per process in the
> {{SCMHAManagerImpl}} constructor. {{SCMHAManagerImpl.installCheckpoint}}
> closes that store ({{stopServices}}, which runs
> {{SCMMetadataStoreImpl.stop}}) and {{startServices}} opens a new one and
> points the managers at it, but the gRPC service is not updated. From then on
> every {{InterSCMGrpcService.download}} reads the closed RocksDB and fails
> with {{RocksDatabaseException: Rocks Database is closed}}. Only a restart of
> that SCM clears it.
> The javadoc of {{SCMHAManagerImpl.reloadSCMState}} says that all classes
> which use or store the metadata manager should be updated with the new
> instance; this one is missed.
> h3. Trigger
> # An SCM installs a checkpoint in process. With default settings this happens
> on every SCM added with {{--bootstrap}}: Ratis (3.3.1) sends a snapshot
> notification to a bootstrapping follower once, independent of log purging,
> and the new SCM downloads and installs the leader's checkpoint.
> # That SCM becomes leader before its next restart (failover or
> {{transferLeadership}}).
> # Something downloads a checkpoint from it.
> h3. Impact
> * Every checkpoint download from that leader fails with gRPC {{UNKNOWN}}, and
> the leader logs "Error streaming SCM DB checkpoint" with "Rocks Database is
> closed" (run).
> * Consumers of that download: an SCM being bootstrapped while this SCM is
> leader (see HDDS-16899 for how a bootstrap behaves when the download fails),
> Recon's SCM DB snapshot sync
> ({{StorageContainerServiceProviderImpl.getSCMDBSnapshot}} downloads through
> {{InterSCMGrpcClient}} from the leader), and, with
> {{ozone.scm.ha.ratis.log.purge.enabled}}, any follower that lags behind the
> purged log. These are code reading; the run shows the failing download by the
> call a follower makes in {{SCMStateMachine.notifyInstallSnapshotFromLeader}}.
> * It lasts until that SCM restarts. Losing leadership does not clear it:
> downloads then go to another SCM, and they fail again whenever this SCM is
> leader before a restart.
> * The SCM's own data is not affected. An SCM bootstrapped while this one is
> leader is subject to HDDS-16899 and can end up without DB state that is not
> in the Raft log.
> {{SCMDBCheckpointServlet}} ({{/dbCheckpoint}}) stores the store the same way,
> but Jetty initializes it on its first request, so it is affected only if it
> served a request before the install, and no Ozone component uses SCM's
> {{/dbCheckpoint}}. It is not changed by the patch (code reading only). The
> same pattern in the OM checkpoint servlets is HDDS-16878.
> {{SCMDBCheckpointServlet.init}} passes the store to the same
> {{DBCheckpointServlet.initialize}}, so the SCM servlet can follow the fix
> made there.
> h3. Reproduction
> PASS on the unmodified source, default configuration.
> [^TestBugMC4CheckpointServiceClosedStore.java]
> ({{hadoop-ozone/integration-test}}) starts three SCMs with one held back,
> starts the held back SCM C (it bootstraps and installs the leader's
> checkpoint in process, and its original store is closed), checks that a
> download from leader A works, transfers leadership to C and downloads from C
> through the follower's {{SCMSnapshotProvider}}. The download from C fails and
> C logs "Rocks Database is closed". A passing test means the defect is
> present. With the patch the download from C succeeds and the test fails at
> its {{assertThatThrownBy}}.
> h3. Patch
> [^MC-4-serve-scm-checkpoints-from-current-db-store.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> {{InterSCMGrpcService.download}} looks up the current store on each request
> and builds the transaction info table and the {{SCMDBCheckpointProvider}}
> from it, so the service holds no reference to a store that an install can
> close. Nothing changes for an SCM that never installed a checkpoint.
> Covered by an assertion added to
> {{TestSCMInstallSnapshot.testInstallCheckPoint}}: after the in process
> install the SCM serves a checkpoint, and that checkpoint carries the
> installed transaction index. Without the change the download fails with the
> error above. With it {{TestSCMInstallSnapshot}},
> {{TestSCMInstallSnapshotWithHA}}, {{TestSCMSnapshot}},
> {{TestScmHAFinalization}}, {{TestSecretKeySnapshot}} and the server-scm
> {{TestInterSCMGrpcProtocolService}}, {{TestInterSCMGrpcClient}} and
> {{TestSCMSnapshotProvider}} pass, and checkstyle is clean.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
