Siyao Meng created HDDS-16863:
---------------------------------
Summary: OM that stops during snapshot install after the DB swap
restarts pre-finalized although its DB is finalized
Key: HDDS-16863
URL: https://issues.apache.org/jira/browse/HDDS-16863
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-1-finalize-om-from-db-on-startup.patch,
TestBugMC1PreFinalizedAfterInterruptedInstall.java
h3. Mechanism
The OM keeps its apparent version in two places: the VERSION file
({{OMStorage}}) and {{#LAYOUTVERSION}} in the OM DB meta table. On startup
{{OMVersionManager}} is built from the VERSION file only.
{{OzoneManager.instantiateServices}} then calls
{{RatisBasedVersionManager.validateDBVersion}}, which only logs at INFO when
the DB differs ("That is expected if this component has never been finalized to
a newer version.").
{{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}, the only code that
brings the VERSION file up to the DB, runs only when {{instantiateServices}} is
called from {{reloadOMState}} during a snapshot install.
{{OzoneManager.installCheckpoint}} first replaces the DB with the leader's
checkpoint and only then writes the VERSION file, once per version, from
{{finalizeFromSnapshotIfRequired}}. If the OM process stops in between, the DB
is finalized and the VERSION file is not. That happens on a crash, or on any
exception that sends {{installCheckpoint}} to {{exitSystem}}, for example a
failed VERSION file write. The window runs from the end of the checkpoint file
move (the transient DB marker is gone, so the DB opens normally on restart)
through opening the new DB and building the services, to the last VERSION write.
After a restart nothing repairs this:
* Startup only logs the mismatch.
* The Raft log is replayed from the DB's transaction index, which is the
checkpoint index, so the {{CompleteFinalizeUpgrade}} entry below it is never
applied on this OM.
* {{OMUpgradeFinalizeService}} only acts when the finalization in progress
marker is set, and the finalized checkpoint does not have it.
The OM runs pre-finalized until a later snapshot install on that node finalizes
it by chance, or until finalize is run again and that OM later leads (see
Impact).
h3. Trigger
# An OM is down or behind while the cluster is finalized, and the leader purges
its log past it.
# The OM installs a snapshot from the leader. The checkpoint carries the
finalized version.
# The OM process stops after the DB swap and before the VERSION file is
written: a crash (kill, OOM, host failure) or an error such as a full or
failing metadata disk, which leads to {{exitSystem}}.
# The OM is restarted and rejoins without another snapshot install. With
default settings ({{ozone.om.ratis.log.purge.gap}} 1,000,000) it catches up
from the Raft log.
I ran the VERSION write failure variant (read-only {{om/current}}) on a mini
cluster. The crash variant is from code, not run: it leaves the same files on
disk.
The same state also arises without a snapshot install when {{om.db}} is
replaced by hand with a checkpoint from a finalized peer, the recovery that the
comment next to the inconsistent DB marker error in {{OmMetadataManagerImpl}}
recommends. The gap exists in every release since 1.2.0, where HDDS-4908 added
the snapshot-only finalize under the same {{withNewSnapshot}} condition, so an
OM left in this state by an older release is carried into a newer one (from
code, not run).
SCM has the same gap for a crash during its snapshot install. A reload failure
there restores the DB backup, so only the crash variant applies. This patch
does not cover SCM.
h3. Impact
* The OM silently runs pre-finalized on top of a finalized DB.
{{getVersionForClient}}, which is advertised to clients in {{ServiceInfo}},
reports the last pre-ZDU client version instead of the software version (run).
* While that OM is leader, the version-gated request checks and the version
stamped into requests are evaluated against the old version, and
{{queryUpgradeStatus}} reports the OM as UNFINALIZED and the cluster as
IN_PROGRESS (from code, not run).
* Running finalize again sets the finalization marker only on the stuck OM,
because each replica checks {{needsFinalization}} when it applies the request.
The stuck OM then finalizes once it is next leader. Until then its meta table
differs from its peers by that marker (from code, not run).
* No data loss. Followers apply the requests as the leader stamped them.
h3. Reproduction
PASS on the unmodified commit, 3 of 3 runs, about 35 s.
[^TestBugMC1PreFinalizedAfterInterruptedInstall.java]
({{hadoop-ozone/integration-test}}) uses a three OM HA mini cluster with one OM
down, set up as in {{TestOMUpgradeFinalization.testFinalizationFromSnapshot}}.
It finalizes the two running OMs, purges the log and makes the down OM's
{{om/current}} read only. When that OM starts it installs the snapshot, the
VERSION write fails after the DB swap and {{installCheckpoint}} calls
{{exitSystem}}. A test exit manager parks the exiting thread and the test stops
the OM. The directory is made writable again and the OM is restarted as a new
{{OzoneManager}} on the same storage.
Result: {{apparent version INITIAL_VERSION (0), VERSION file 0, DB version 100,
finalization marker null, DB transaction 1#106, applied index (t:1, i:106),
leader index 106}}, one snapshot install in total, and the startup log line
"Version file has different apparent version (INITIAL_VERSION (0)) than DB (ZDU
(100)). That is expected if this component has never been finalized to a newer
version." A passing test means the defect is present.
The test does not write after the restart. With its 5 entry purge gap the
leader would purge past the OM again, and the resulting second snapshot install
finalizes it.
h3. Patch
[^MC-1-finalize-om-from-db-on-startup.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
On startup (the {{withNewSnapshot}} false branch of {{instantiateServices}})
the OM now calls a new
{{RatisBasedVersionManager.finalizeToDBVersionIfRequired}}. It finalizes only
up to the version recorded in the DB, never to the software version of the
binary, and logs a warning when it does. It is a no-op unless the DB records a
version the VERSION file does not allow. On a normal restart the DB is at or
below the VERSION file, since {{OMCompleteFinalizeUpgradeRequest}} writes the
VERSION file before its DB batch. It runs before the Ratis server is created,
so there is no concurrent apply and no log replay yet.
The bound uses {{ComponentVersionManager.finalizeUpgrade(lastVersion)}}.
HDDS-16862 adds the same helper, with an identical hunk, to bound the snapshot
install path, while this patch bounds the startup path. The two patches merge
cleanly in either order ({{git merge}}, or {{git apply --3way}} for the patch
files).
{{QuotaRepairUpgradeAction}} now skips the repair, with a warning like the
follower branch, when the OM has no Ratis server yet. Without that, a VERSION
file below {{QUOTA}} makes the startup finalize fail with an NPE in
{{checkLeaderStatus}} and the OM does not start. A follower already skips this
repair. If the pending mark from HDDS-16861 lands, this startup skip should set
it too. The patches for HDDS-16860 and HDDS-16861 change the same few lines of
{{QuotaRepairUpgradeAction.execute}} as this one, so whichever lands later
needs a trivial rebase.
Covered by {{testFinalizationFromDBOnRestart}} in the existing
{{TestOMUpgradeFinalization}}. On one OM it checks four cases:
* An OM without a DB version stays pre-finalized.
* A DB at {{HBASE_SUPPORT}} over a VERSION file at {{INITIAL_VERSION}}
finalizes the OM to {{HBASE_SUPPORT}} and no further.
* A DB behind the VERSION file changes nothing.
* A DB at the software version finalizes the OM fully.
Failures:
* Without the change: {{expected: <HBASE_SUPPORT (7)> but was: <INITIAL_VERSION
(0)>}}.
* With an unbounded startup finalize: {{expected: <HBASE_SUPPORT (7)> but was:
<ZDU (100)>}}.
* Without the quota repair change: the NPE above.
With the patch, these pass and checkstyle is clean:
* {{TestOMUpgradeFinalization}} (4 tests)
* the OM upgrade, state machine and Ratis server unit suites, plus suites that
start an OM through {{OmTestManagers}} (171 tests)
* the SCM and datanode version manager suites (40 tests)
* {{TestOzoneManagerRestart}}, {{TestUpgradeFinalizationWithOldClients}},
{{TestOMRatisSnapshots}}, {{TestOMBucketLayoutUpgrade}},
{{TestOMInstallSnapshotDuringBootstrapping}} and
{{TestOmStartupInvalidApparentVersion}} (22 tests)
With the patch the reproduction fails, because the restarted OM finalizes in
its constructor ({{expected: INITIAL_VERSION (0) but was: ZDU (100)}}).
Found by TLA+ model checking and code review of the OM upgrade finalization
paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
