[
https://issues.apache.org/jira/browse/HDDS-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16862:
------------------------------
Attachment: MC-4-bound-snapshot-finalization-to-snapshot-version.patch
> Finalization from a Ratis snapshot should stop at the version recorded in the
> snapshot (OM, SCM)
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16862
> URL: https://issues.apache.org/jira/browse/HDDS-16862
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-4-bound-snapshot-finalization-to-snapshot-version.patch
>
>
> h3. Mechanism
> When an OM installs a Ratis snapshot, {{OzoneManager.instantiateServices}}
> calls {{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}. SCM calls
> the same method from {{FinalizationStateManagerImpl.reinitialize}}. If the
> {{#LAYOUTVERSION}} in the installed DB is ahead of the local apparent
> version, the method logs "Attempting to finalize to that version" and then
> calls {{ComponentVersionManager.finalizeUpgrade()}}. That call does not stop
> at the snapshot's version. It follows {{nextVersion()}} up to the local
> software version, runs the upgrade action of every version on the way, and
> the method then writes the local software version into the installed DB.
> So whenever the snapshot records a version below the local software version,
> the node finalizes further than the leader it took the snapshot from. The pre
> ZDU code added by HDDS-4908 behaved the same way: it finalized to the full
> software layout version and only checked afterwards that the result was not
> below the DB version.
> h3. Trigger
> # An OM (or SCM) is down while its peers finalize to some version V.
> # The cluster is then upgraded to a release whose software version is above V
> and is not finalized yet. The down node is started on that release.
> # It installs a snapshot from the leader whose DB records V. Instead of
> stopping at V, it finalizes to its own software version.
> This works with the current version catalogs. For OM, V can be any pre ZDU
> layout feature, and the node then finalizes to {{ZDU (100)}}.
> h3. Impact
> Downgrade after finalization is not supported by design. The problem is that
> one node ends up finalized further than its peers:
> * Its VERSION file and {{#LAYOUTVERSION}} are ahead of its peers, and nothing
> lowers them again. It has run the upgrade actions of versions its peers have
> not finalized, for OM including the {{ZDU}} action.
> * That OM alone reports the newer version to clients and allows the features
> gated on it. While it is leader, those features are in use although the other
> OMs have not finalized them (argued, not run).
> * The newer release is not finalized on its peers, so the design still allows
> downgrading it, but this node holds a version the older release does not know
> and cannot start on it (argued, not run).
> h3. Patch
> [^MC-4-bound-snapshot-finalization-to-snapshot-version.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{ComponentVersionManager}} gets {{finalizeUpgrade(ComponentVersion
> lastVersion)}}, which runs the existing loop and stops after {{lastVersion}}.
> {{finalizeUpgrade()}} delegates with the software version, so its behaviour
> is unchanged. {{finalizeFromSnapshotIfRequired}} passes the version read from
> the snapshot DB, so OM and SCM finalize from a snapshot exactly to the
> version the snapshot records, and run only the upgrade actions up to it. A
> snapshot from a fully finalized leader of the same release records the
> software version, so that case behaves as before.
> Regression tests in the patch:
> {{testFinalizeFromSnapshotStopsAtSnapshotVersion}} in the existing
> {{TestOMVersionManager}} and {{TestScmVersionManager}}, using the real
> version catalogs and VERSION files. Each test covers three cases:
> * A snapshot between the node's apparent and software versions: the node
> stops at the snapshot's version and stays pre-finalized. Its upgrade action
> runs and the {{ZDU}} action does not.
> * A snapshot at the software version: the node finalizes to it.
> * A snapshot that is not ahead, or a DB without a version: nothing changes.
> Without the change they fail with "expected: <HBASE_SUPPORT (7)> but was:
> <ZDU (100)>" (OM) and "expected: <ERASURE_CODED_STORAGE_SUPPORT (3)> but was:
> <ZDU (100)>" (SCM). With it, the following pass and checkstyle is clean:
> * {{TestOMVersionManager}}, {{TestScmVersionManager}} and
> {{TestDatanodeVersionManager}}.
> * The OM finalize request and service unit tests.
> * {{TestOMUpgradeFinalization}} and {{TestScmHAFinalization}} on a mini
> cluster, including their snapshot finalization tests.
> Found by TLA+ model checking and code review of the OM upgrade finalization
> paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
