[
https://issues.apache.org/jira/browse/HDDS-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16931:
------------------------------
Attachment: TestBugMC7CorruptCheckpointAcceptedAndInstalled.java
> SCM checks only the TransactionInfo of a downloaded checkpoint before
> acknowledging it, so a checkpoint with corrupt SST blocks is installed
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16931
> URL: https://issues.apache.org/jira/browse/HDDS-16931
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-7-verify-scm-checkpoint-checksums-before-install.patch,
> TestBugMC7CorruptCheckpointAcceptedAndInstalled.java
>
>
> h3. Mechanism
> Before a follower SCM acknowledges a downloaded checkpoint to Ratis,
> {{SCMHAManagerImpl.verifyCheckpointFromLeader}} opens it read only and reads
> only the TransactionInfo. A checkpoint with a damaged data block in another
> table passes. {{installCheckpoint}} then calls
> {{HAUtils.replaceDBWithCheckpoint}}, which moves {{scm.db}} to a backup,
> copies the checkpoint in and deletes {{dbInconsistentMarker}} before any
> service has read the new DB. Only then does {{reloadSCMState}} read it; its
> failure triggers a rollback to the old DB. If the process stops before that
> rollback (kill, OOM, power loss during the reload, which iterates every
> container and pipeline), {{scm.db}} is the corrupt checkpoint, there is no
> marker, and the old DB is only in {{scm.db.backup.*}}. Every start then fails
> on the first read of the damaged block, and nothing restores the backup.
> h3. Trigger
> # The checkpoint a follower downloads has a corrupt SST block outside the
> TransactionInfo, for example media corruption of a rarely read SST on the
> leader (the checkpoint hard links the leader's files) or a write error on the
> follower while unpacking it.
> # The follower SCM stops after {{replaceDBWithCheckpoint}} returned and
> before the rollback in {{installCheckpoint}} completed. A failed copy during
> the rollback (for example out of disk space) reaches the same state without a
> stop: its reset branch moves the corrupt checkpoint back into {{scm.db}} and
> deletes the marker (by code reading, not run).
> Both steps are uncommon, and a stop alone (with a good checkpoint) is
> harmless today: the restart opens the installed DB and catches up.
> h3. Impact
> * If the damaged block is in a table the reload iterates (containers,
> pipelines), the scan ends early without an error, because
> {{RDBStoreAbstractIterator.hasNext}} checks only {{isValid()}} and not
> {{status()}}. The reload can then report success with part of the table
> loaded, and the backup of the old DB is deleted (by code reading, not run).
> The iterator behaviour affects every table scan and is worth its own issue.
> * If the damaged block is read by key during the reload and the SCM stops
> before the rollback: the SCM does not start until an operator repairs its DB.
> There is no marker, so the error is a RocksDB corruption, not the
> "inconsistent state" message that points at the cause.
> * No data is lost cluster wide; the other SCMs have the state and the old DB
> is intact in the backup. Moving the backup back makes the DB open, but the
> follower acknowledged the install and its Raft log now starts at the snapshot
> index, so the old DB alone does not let it rejoin (argued, see HDDS-16928).
> Replacing the DB from another SCM does.
> h3. Reproduction
> PASS on the unmodified source.
> [^TestBugMC7CorruptCheckpointAcceptedAndInstalled.java]
> ({{hadoop-ozone/integration-test}}) runs a one SCM mini cluster. It copies a
> clean checkpoint of the SCM DB to a separate directory as the follower's DB,
> and makes a second checkpoint with a raised TransactionInfo and the first 16
> bytes of its meta table SST overwritten. {{verifyCheckpointFromLeader}} on
> the SCM accepts it: {{(t:1, i:1004)}}. {{HAUtils.replaceDBWithCheckpoint}} on
> the follower's directory then returns with no marker and the old DB in
> {{scm.db.backup.4_...}}. Opening that directory with {{SCMMetadataStoreImpl}}
> and reading the apparent layout version, as SCM start does, fails with
> {{Corruption: ... Failed to get #LAYOUTVERSION from ColumnFamily-meta}}. The
> stop itself is not simulated with a kill: {{replaceDBWithCheckpoint}} has
> returned and nothing else writes to the directory before the reload, so the
> directory is in the state a stop at that point leaves. A passing test means
> the defect is present.
> h3. Patch
> [^MC-7-verify-scm-checkpoint-checksums-before-install.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> The patch makes {{verifyCheckpointFromLeader}} also run RocksDB
> {{verifyChecksum}} on the read only checkpoint (a new
> {{RocksDatabase.verifyChecksum}} and {{HAUtils.verifyCheckpointChecksums}}).
> A damaged checkpoint is then handled like a failed download: the follower
> does not acknowledge it, its DB is not touched, and Ratis retries the
> install. This removes the precondition instead of narrowing the stop window,
> and it also keeps checkpoints with damaged blocks in iterated tables out of
> the install. The check is unconditional, with no switch, because it runs only
> during an install, on the install thread, at a cost of the same order as the
> download just before it, and installing a checkpoint with damaged blocks is
> never the right outcome. The cost is one read of the downloaded checkpoint
> per install; nothing changes for SCMs that do not install a checkpoint, and
> the shared {{HAUtils.getTrxnInfoFromCheckpoint}} used by OM is unchanged.
> Keeping the marker until the reload succeeds was not chosen: it would also
> refuse the start after a stop during the reload of a good checkpoint, which
> recovers on its own today. With the patch the reproduction fails at its first
> assertion: {{verifyCheckpointFromLeader}} returns null and logs {{Corruption:
> ... Failed to verifyChecksum}}.
> Covered by {{testVerifyCheckpointFromLeaderRejectsCorruptCheckpoint}} in the
> existing {{TestSCMHAManagerImpl}}: a clean checkpoint of a real SCM DB is
> accepted, and the same checkpoint with damaged SST blocks (except the one
> with the TransactionInfo) is rejected. Without the change it fails with
> "expected: null but was: (t:1, i:1000)". With it the SCM HA unit suites
> ({{TestSCMHAManagerImpl}}, {{TestSCMStateMachine}},
> {{TestSCMSnapshotProvider}}, 22 tests) and {{TestRDBStore}} and
> {{TestRDBSnapshotProvider}} (18 tests) pass. The mini cluster suites
> {{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests,
> including an install through Ratis) pass and checkstyle is clean.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
