[
https://issues.apache.org/jira/browse/HDDS-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16863:
------------------------------
Attachment: MC-1-finalize-om-from-db-on-startup.patch
> OM that stops during snapshot install after the DB swap restarts
> pre-finalized although its DB is finalized
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16863
> URL: https://issues.apache.org/jira/browse/HDDS-16863
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-1-finalize-om-from-db-on-startup.patch,
> TestBugMC1PreFinalizedAfterInterruptedInstall.java
>
>
> h3. Mechanism
> The OM keeps its apparent version in two places: the VERSION file
> ({{OMStorage}}) and {{#LAYOUTVERSION}} in the OM DB meta table. On startup
> {{OMVersionManager}} is built from the VERSION file only.
> {{OzoneManager.instantiateServices}} then calls
> {{RatisBasedVersionManager.validateDBVersion}}, which only logs at INFO when
> the DB differs ("That is expected if this component has never been finalized
> to a newer version.").
> {{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}, the only code
> that brings the VERSION file up to the DB, runs only when
> {{instantiateServices}} is called from {{reloadOMState}} during a snapshot
> install.
> {{OzoneManager.installCheckpoint}} first replaces the DB with the leader's
> checkpoint and only then writes the VERSION file, once per version, from
> {{finalizeFromSnapshotIfRequired}}. If the OM process stops in between, the
> DB is finalized and the VERSION file is not. That happens on a crash, or on
> any exception that sends {{installCheckpoint}} to {{exitSystem}}, for example
> a failed VERSION file write. The window runs from the end of the checkpoint
> file move (the transient DB marker is gone, so the DB opens normally on
> restart) through opening the new DB and building the services, to the last
> VERSION write.
> After a restart nothing repairs this:
> * Startup only logs the mismatch.
> * The Raft log is replayed from the DB's transaction index, which is the
> checkpoint index, so the {{CompleteFinalizeUpgrade}} entry below it is never
> applied on this OM.
> * {{OMUpgradeFinalizeService}} only acts when the finalization in progress
> marker is set, and the finalized checkpoint does not have it.
> The OM runs pre-finalized until a later snapshot install on that node
> finalizes it by chance, or until finalize is run again and that OM later
> leads (see Impact).
> h3. Trigger
> # An OM is down or behind while the cluster is finalized, and the leader
> purges its log past it.
> # The OM installs a snapshot from the leader. The checkpoint carries the
> finalized version.
> # The OM process stops after the DB swap and before the VERSION file is
> written: a crash (kill, OOM, host failure) or an error such as a full or
> failing metadata disk, which leads to {{exitSystem}}.
> # The OM is restarted and rejoins without another snapshot install. With
> default settings ({{ozone.om.ratis.log.purge.gap}} 1,000,000) it catches up
> from the Raft log.
> I ran the VERSION write failure variant (read-only {{om/current}}) on a mini
> cluster. The crash variant is from code, not run: it leaves the same files on
> disk.
> The same state also arises without a snapshot install when {{om.db}} is
> replaced by hand with a checkpoint from a finalized peer, the recovery that
> the comment next to the inconsistent DB marker error in
> {{OmMetadataManagerImpl}} recommends. The gap exists in every release since
> 1.2.0, where HDDS-4908 added the snapshot-only finalize under the same
> {{withNewSnapshot}} condition, so an OM left in this state by an older
> release is carried into a newer one (from code, not run).
> SCM has the same gap for a crash during its snapshot install. A reload
> failure there restores the DB backup, so only the crash variant applies. This
> patch does not cover SCM.
> h3. Impact
> * The OM silently runs pre-finalized on top of a finalized DB.
> {{getVersionForClient}}, which is advertised to clients in {{ServiceInfo}},
> reports the last pre-ZDU client version instead of the software version (run).
> * While that OM is leader, the version-gated request checks and the version
> stamped into requests are evaluated against the old version, and
> {{queryUpgradeStatus}} reports the OM as UNFINALIZED and the cluster as
> IN_PROGRESS (from code, not run).
> * Running finalize again sets the finalization marker only on the stuck OM,
> because each replica checks {{needsFinalization}} when it applies the
> request. The stuck OM then finalizes once it is next leader. Until then its
> meta table differs from its peers by that marker (from code, not run).
> * No data loss. Followers apply the requests as the leader stamped them.
> h3. Reproduction
> PASS on the unmodified commit, 3 of 3 runs, about 35 s.
> [^TestBugMC1PreFinalizedAfterInterruptedInstall.java]
> ({{hadoop-ozone/integration-test}}) uses a three OM HA mini cluster with one
> OM down, set up as in
> {{TestOMUpgradeFinalization.testFinalizationFromSnapshot}}. It finalizes the
> two running OMs, purges the log and makes the down OM's {{om/current}} read
> only. When that OM starts it installs the snapshot, the VERSION write fails
> after the DB swap and {{installCheckpoint}} calls {{exitSystem}}. A test exit
> manager parks the exiting thread and the test stops the OM. The directory is
> made writable again and the OM is restarted as a new {{OzoneManager}} on the
> same storage.
> Result: {{apparent version INITIAL_VERSION (0), VERSION file 0, DB version
> 100, finalization marker null, DB transaction 1#106, applied index (t:1,
> i:106), leader index 106}}, one snapshot install in total, and the startup
> log line "Version file has different apparent version (INITIAL_VERSION (0))
> than DB (ZDU (100)). That is expected if this component has never been
> finalized to a newer version." A passing test means the defect is present.
> The test does not write after the restart. With its 5 entry purge gap the
> leader would purge past the OM again, and the resulting second snapshot
> install finalizes it.
> h3. Patch
> [^MC-1-finalize-om-from-db-on-startup.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> On startup (the {{withNewSnapshot}} false branch of {{instantiateServices}})
> the OM now calls a new
> {{RatisBasedVersionManager.finalizeToDBVersionIfRequired}}. It finalizes only
> up to the version recorded in the DB, never to the software version of the
> binary, and logs a warning when it does. It is a no-op unless the DB records
> a version the VERSION file does not allow. On a normal restart the DB is at
> or below the VERSION file, since {{OMCompleteFinalizeUpgradeRequest}} writes
> the VERSION file before its DB batch. It runs before the Ratis server is
> created, so there is no concurrent apply and no log replay yet.
> The bound uses {{ComponentVersionManager.finalizeUpgrade(lastVersion)}}.
> HDDS-16862 adds the same helper, with an identical hunk, to bound the
> snapshot install path, while this patch bounds the startup path. The two
> patches merge cleanly in either order ({{git merge}}, or {{git apply --3way}}
> for the patch files).
> {{QuotaRepairUpgradeAction}} now skips the repair, with a warning like the
> follower branch, when the OM has no Ratis server yet. Without that, a VERSION
> file below {{QUOTA}} makes the startup finalize fail with an NPE in
> {{checkLeaderStatus}} and the OM does not start. A follower already skips
> this repair. If the pending mark from HDDS-16861 lands, this startup skip
> should set it too. The patches for HDDS-16860 and HDDS-16861 change the same
> few lines of {{QuotaRepairUpgradeAction.execute}} as this one, so whichever
> lands later needs a trivial rebase.
> Covered by {{testFinalizationFromDBOnRestart}} in the existing
> {{TestOMUpgradeFinalization}}. On one OM it checks four cases:
> * An OM without a DB version stays pre-finalized.
> * A DB at {{HBASE_SUPPORT}} over a VERSION file at {{INITIAL_VERSION}}
> finalizes the OM to {{HBASE_SUPPORT}} and no further.
> * A DB behind the VERSION file changes nothing.
> * A DB at the software version finalizes the OM fully.
> Failures:
> * Without the change: {{expected: <HBASE_SUPPORT (7)> but was:
> <INITIAL_VERSION (0)>}}.
> * With an unbounded startup finalize: {{expected: <HBASE_SUPPORT (7)> but
> was: <ZDU (100)>}}.
> * Without the quota repair change: the NPE above.
> With the patch, these pass and checkstyle is clean:
> * {{TestOMUpgradeFinalization}} (4 tests)
> * the OM upgrade, state machine and Ratis server unit suites, plus suites
> that start an OM through {{OmTestManagers}} (171 tests)
> * the SCM and datanode version manager suites (40 tests)
> * {{TestOzoneManagerRestart}}, {{TestUpgradeFinalizationWithOldClients}},
> {{TestOMRatisSnapshots}}, {{TestOMBucketLayoutUpgrade}},
> {{TestOMInstallSnapshotDuringBootstrapping}} and
> {{TestOmStartupInvalidApparentVersion}} (22 tests)
> With the patch the reproduction fails, because the restarted OM finalizes in
> its constructor ({{expected: INITIAL_VERSION (0) but was: ZDU (100)}}).
> Found by TLA+ model checking and code review of the OM upgrade finalization
> paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
