Siyao Meng created HDDS-16931:
---------------------------------
Summary: SCM checks only the TransactionInfo of a downloaded
checkpoint before acknowledging it, so a checkpoint with corrupt SST blocks is
installed
Key: HDDS-16931
URL: https://issues.apache.org/jira/browse/HDDS-16931
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-7-verify-scm-checkpoint-checksums-before-install.patch,
TestBugMC7CorruptCheckpointAcceptedAndInstalled.java
h3. Mechanism
Before a follower SCM acknowledges a downloaded checkpoint to Ratis,
{{SCMHAManagerImpl.verifyCheckpointFromLeader}} opens it read only and reads
only the TransactionInfo. A checkpoint with a damaged data block in another
table passes. {{installCheckpoint}} then calls
{{HAUtils.replaceDBWithCheckpoint}}, which moves {{scm.db}} to a backup, copies
the checkpoint in and deletes {{dbInconsistentMarker}} before any service has
read the new DB. Only then does {{reloadSCMState}} read it; its failure
triggers a rollback to the old DB. If the process stops before that rollback
(kill, OOM, power loss during the reload, which iterates every container and
pipeline), {{scm.db}} is the corrupt checkpoint, there is no marker, and the
old DB is only in {{scm.db.backup.*}}. Every start then fails on the first read
of the damaged block, and nothing restores the backup.
h3. Trigger
# The checkpoint a follower downloads has a corrupt SST block outside the
TransactionInfo, for example media corruption of a rarely read SST on the
leader (the checkpoint hard links the leader's files) or a write error on the
follower while unpacking it.
# The follower SCM stops after {{replaceDBWithCheckpoint}} returned and before
the rollback in {{installCheckpoint}} completed. A failed copy during the
rollback (for example out of disk space) reaches the same state without a stop:
its reset branch moves the corrupt checkpoint back into {{scm.db}} and deletes
the marker (by code reading, not run).
Both steps are uncommon, and a stop alone (with a good checkpoint) is harmless
today: the restart opens the installed DB and catches up.
h3. Impact
* If the damaged block is in a table the reload iterates (containers,
pipelines), the scan ends early without an error, because
{{RDBStoreAbstractIterator.hasNext}} checks only {{isValid()}} and not
{{status()}}. The reload can then report success with part of the table loaded,
and the backup of the old DB is deleted (by code reading, not run). The
iterator behaviour affects every table scan and is worth its own issue.
* If the damaged block is read by key during the reload and the SCM stops
before the rollback: the SCM does not start until an operator repairs its DB.
There is no marker, so the error is a RocksDB corruption, not the "inconsistent
state" message that points at the cause.
* No data is lost cluster wide; the other SCMs have the state and the old DB is
intact in the backup. Moving the backup back makes the DB open, but the
follower acknowledged the install and its Raft log now starts at the snapshot
index, so the old DB alone does not let it rejoin (argued, see HDDS-16928).
Replacing the DB from another SCM does.
h3. Reproduction
PASS on the unmodified source.
[^TestBugMC7CorruptCheckpointAcceptedAndInstalled.java]
({{hadoop-ozone/integration-test}}) runs a one SCM mini cluster. It copies a
clean checkpoint of the SCM DB to a separate directory as the follower's DB,
and makes a second checkpoint with a raised TransactionInfo and the first 16
bytes of its meta table SST overwritten. {{verifyCheckpointFromLeader}} on the
SCM accepts it: {{(t:1, i:1004)}}. {{HAUtils.replaceDBWithCheckpoint}} on the
follower's directory then returns with no marker and the old DB in
{{scm.db.backup.4_...}}. Opening that directory with {{SCMMetadataStoreImpl}}
and reading the apparent layout version, as SCM start does, fails with
{{Corruption: ... Failed to get #LAYOUTVERSION from ColumnFamily-meta}}. The
stop itself is not simulated with a kill: {{replaceDBWithCheckpoint}} has
returned and nothing else writes to the directory before the reload, so the
directory is in the state a stop at that point leaves. A passing test means the
defect is present.
h3. Patch
[^MC-7-verify-scm-checkpoint-checksums-before-install.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
The patch makes {{verifyCheckpointFromLeader}} also run RocksDB
{{verifyChecksum}} on the read only checkpoint (a new
{{RocksDatabase.verifyChecksum}} and {{HAUtils.verifyCheckpointChecksums}}). A
damaged checkpoint is then handled like a failed download: the follower does
not acknowledge it, its DB is not touched, and Ratis retries the install. This
removes the precondition instead of narrowing the stop window, and it also
keeps checkpoints with damaged blocks in iterated tables out of the install.
The check is unconditional, with no switch, because it runs only during an
install, on the install thread, at a cost of the same order as the download
just before it, and installing a checkpoint with damaged blocks is never the
right outcome. The cost is one read of the downloaded checkpoint per install;
nothing changes for SCMs that do not install a checkpoint, and the shared
{{HAUtils.getTrxnInfoFromCheckpoint}} used by OM is unchanged. Keeping the
marker until the reload succeeds was not chosen: it would also refuse the start
after a stop during the reload of a good checkpoint, which recovers on its own
today. With the patch the reproduction fails at its first assertion:
{{verifyCheckpointFromLeader}} returns null and logs {{Corruption: ... Failed
to verifyChecksum}}.
Covered by {{testVerifyCheckpointFromLeaderRejectsCorruptCheckpoint}} in the
existing {{TestSCMHAManagerImpl}}: a clean checkpoint of a real SCM DB is
accepted, and the same checkpoint with damaged SST blocks (except the one with
the TransactionInfo) is rejected. Without the change it fails with "expected:
null but was: (t:1, i:1000)". With it the SCM HA unit suites
({{TestSCMHAManagerImpl}}, {{TestSCMStateMachine}},
{{TestSCMSnapshotProvider}}, 22 tests) and {{TestRDBStore}} and
{{TestRDBSnapshotProvider}} (18 tests) pass. The mini cluster suites
{{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests,
including an install through Ratis) pass and checkstyle is clean.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
