[
https://issues.apache.org/jira/browse/HDDS-16878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16878:
------------------------------
Attachment: CR-1-checkpoint-servlets-read-current-db-store.patch
> OM checkpoint servlets keep using the DB store closed by a checkpoint install
> and return HTTP 500 once that OM is leader
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16878
> URL: https://issues.apache.org/jira/browse/HDDS-16878
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-1-checkpoint-servlets-read-current-db-store.patch,
> TestBugCR1CheckpointServletClosedStore.java
>
>
> h3. Mechanism
> {{DBCheckpointServlet.initialize}} stores the {{DBStore}} it is given, and
> {{OMDBCheckpointServlet.init}} and
> {{OMDBCheckpointServletInodeBasedXfer.init}} pass
> {{om.getMetadataManager().getStore()}}. Both servlets then take every
> checkpoint, the DB location and the {{RocksDBCheckpointDiffer}} from
> {{getDbStore()}}, which returns that stored field.
> {{OzoneManager.installCheckpoint}} stops the old metadata manager, which
> closes its store, and {{reloadOMState}} creates a new
> {{OmMetadataManagerImpl}}. The HTTP server and its servlets are created once
> in {{OzoneManager.start}} and are never initialized again, so after the
> install the servlets still use the closed store.
> Jetty initializes each servlet separately, on its own first request. Each
> endpoint is affected if it served at least one request before the install in
> the same process lifetime: for example while that OM was leader earlier (a
> follower install, an OM bootstrap or a Recon full snapshot), or a request
> that reached it as a follower and got 503. Followers and bootstrapping OMs
> download from {{/v2/dbCheckpoint}}.
> The same code is in every release since 2.0.0 (code reading of the release
> tags).
> h3. Trigger
> # OM X serves a request on a checkpoint endpoint (as leader, or 503 as
> follower). Only the endpoints that served a request are affected.
> # X falls behind far enough that the leader sends a snapshot, and X installs
> the checkpoint through {{installSnapshotFromLeader}}.
> # X becomes leader (election or leadership transfer).
> h3. Impact
> * While X is leader, every request to {{/dbCheckpoint}} and
> {{/v2/dbCheckpoint}} returns HTTP 500 and the OM logs
> {{RocksDatabaseException: Rocks Database is closed}} from
> {{RDBStore.getCheckpoint}}. This lasts until X restarts.
> * A follower that needs a checkpoint from X cannot get one:
> {{OmRatisSnapshotProvider.downloadDBSnapshotFromLeader}} fails with
> "ErrorCode: 500" (run). A bootstrapping OM and a Recon full snapshot use the
> same endpoints and would fail the same way (code reading, not run). Recon
> incremental sync through {{getDBUpdates}} reads the current store and is not
> affected.
> * No client data is affected.
> h3. Reproduction
> PASS on the unmodified source, deterministic.
> [^TestBugCR1CheckpointServletClosedStore.java]
> ({{hadoop-ozone/integration-test}}) starts three OMs with one held back,
> advances the leader past the Ratis log purge gap, and starts the late OM with
> the existing {{OzoneManager.setTestInstallSnapshot}} hook set only until both
> servlets have answered one request (503). Ratis then installs the checkpoint
> through the real path, leadership is transferred to that OM, and both
> endpoints plus a real follower download are tried. Output:
> {noformat}
> Before install, omNode-3 answered /dbCheckpoint -> 503, /v2/dbCheckpoint ->
> 503. After install: store replaced = true, old store closed = true
> After omNode-3 became leader: /dbCheckpoint -> 500, /v2/dbCheckpoint -> 500,
> follower download failed: Unexpected exception when trying to reach OM to
> download latest checkpoint. Checkpoint URL:
> http://127.0.0.1:15017/v2/dbCheckpoint?includeSnapshotData=true&flushBeforeCheckpoint=true.
> ErrorCode: 500
> {noformat}
> A passing test means the defect is present. With the patch both endpoints
> return 200, the download succeeds and the test fails at its first defect
> assertion.
> h3. Patch
> [^CR-1-checkpoint-servlets-read-current-db-store.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> 6818707698c.
> Both OM servlets override {{getDbStore()}} to return the store of the OM's
> current metadata manager, the same way their bootstrap lock and
> {{getSnapshotDir()}} already read the OM's current state. Nothing changes for
> an OM that never installed a checkpoint. {{DBCheckpointServlet}} and the SCM
> servlet are not changed. Four places in the existing servlet tests injected a
> spy store through {{initialize}}; they now also stub {{getDbStore()}} with it.
> Covered by the new
> {{TestOMRatisSnapshots#testDbCheckpointServletsServeInstalledDbAfterLeaderChange}},
> which fails without the change with {{[/dbCheckpoint] expected: 200 but was:
> 500}}. With the change {{TestOMRatisSnapshots}},
> {{TestOMDbCheckpointServlet}}, {{TestOMDbCheckpointServletInodeBasedXfer}},
> {{TestGetDBUpdatesAuthorization}} and
> {{TestOMDBCheckpointServletInodeBasedXferNonLeader}} pass (37 tests) and
> checkstyle is clean.
> {{SCMDBCheckpointServlet}} caches the SCM store the same way, and an SCM
> checkpoint install also creates a new store (code reading, not run). Recon
> fetches the SCM DB over gRPC, so only direct use of the SCM endpoint is
> affected; it is left for a separate change. Other holders of the old metadata
> manager, such as the snapshot directory metrics and the multi tenant Ranger
> sync service (HDDS-16853), are out of scope here.
> Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
