[
https://issues.apache.org/jira/browse/HDDS-16920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16920:
------------------------------
Attachment: TestBugMC2CrashAfterDBMoveOpensEmptyDB.java
> SCM creates the DB inconsistent marker only after moving scm.db away, so a
> stop in between restarts on an empty DB
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16920
> URL: https://issues.apache.org/jira/browse/HDDS-16920
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-2-create-db-marker-before-moving-scm-db.patch,
> TestBugMC2CrashAfterDBMoveOpensEmptyDB.java
>
>
> h3. Mechanism
> {{HAUtils.replaceDBWithCheckpoint}} renames {{scm.db}} to
> {{scm.db.backup.<index>_<time>}} and only then creates
> {{dbInconsistentMarker}}. The marker is meant to stop SCM from starting on a
> half installed DB ("this marker file will fail it from starting up"). If the
> process stops between the rename and the marker, the directory has neither
> {{scm.db}} nor the marker. On the next start {{SCMMetadataStoreImpl.start}}
> checks only the marker, and {{DBStoreBuilder.createDBStore}} creates a new
> empty DB because {{createIfMissing}} is true. The old DB stays in the backup
> directory and nothing restores or deletes it.
> h3. Trigger
> # A follower SCM installs a checkpoint from the leader
> ({{SCMHAManagerImpl.installCheckpoint}}).
> # The SCM process stops (kill, OOM) right after {{Files.move}} of {{scm.db}}
> and before {{Files.createFile}} of the marker. The window is two consecutive
> file system calls; in the reproduction the stop is injected.
> h3. Impact
> * The SCM starts on an empty {{scm.db}} without any error, against the marker
> contract. Its old DB is left orphaned in {{scm.db.backup.*}}.
> * Committed state is not lost cluster wide (the other SCMs have it). The
> window is entered only after Ratis acknowledged the install, so with the
> current code the follower cannot catch up afterwards anyway, whether it
> restarts on the empty DB or on the old one. As long as it stays a follower it
> does not start its datanode server, so no datanode report is processed
> against the empty DB (argued, not run).
> h3. Reproduction
> PASS on the unmodified source. [^TestBugMC2CrashAfterDBMoveOpensEmptyDB.java]
> ({{hadoop-hdds/server-scm}}) writes a real SCM DB with a container and a
> transaction info, runs {{HAUtils.replaceDBWithCheckpoint}} with a static
> {{Files.move}} spy that does the real move and then throws an {{Error}} (the
> stop), and opens a new {{SCMMetadataStoreImpl}} on the same directory.
> Output: {{after the crash: scm.db exists=false, marker exists=false,
> backups=1}}, then {{restart accepted, scm.db exists=true,
> transactionInfo=null, container #1=null}}. A passing test means the defect is
> present.
> h3. Patch
> [^MC-2-create-db-marker-before-moving-scm-db.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> The patch creates the marker before the move and deletes it again if the move
> fails, so a failed backup leaves the directory unchanged. The rest of the
> method is unchanged. A stop anywhere between the move and the end of the copy
> now leaves the marker, and the existing check refuses to start ("Cannot load
> SCM DB as it is in an inconsistent state"). With the patch the reproduction
> fails because the marker exists after the stop.
> Covered by {{testReplaceDBWithCheckpointCreatesMarkerBeforeMovingDB}} and
> {{testReplaceDBWithCheckpointRemovesMarkerIfMoveFails}} in a new
> {{TestHAUtils}} (there is no existing unit test class for {{HAUtils}}).
> Without the change the first fails with "Expecting AtomicBoolean(false) to
> have value: true". With it {{TestHAUtils}}, {{TestRDBSnapshotProvider}} and
> {{TestSCMHAUtils}} (5 tests) and the mini cluster suites
> {{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests)
> pass, and checkstyle is clean. The patch does not add an fsync of the marker
> or the directory, so it covers a process stop; after a power loss the file
> system may still persist the rename without the marker.
> {{OzoneManager.replaceOMDBWithCheckpoint}} still moves the {{om.db}} items to
> the backup before {{moveCheckpointFiles}} creates the marker (by code
> reading); HDDS-16026 changed its failure handling, not this order, and the OM
> side is a separate follow up that this patch does not change.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
