[
https://issues.apache.org/jira/browse/HDDS-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17014:
------------------------------
Attachment: TestBugCR10FinalizeRace.java
> Volume check during SchemaV3 finalization can fail a healthy data volume
> because createDbStore sets dbLoaded before dbParentDir
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17014
> URL: https://issues.apache.org/jira/browse/HDDS-17014
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-10-set-db-location-before-publishing-db-loaded.patch,
> TestBugCR10FinalizeRace.java
>
>
> h3. Mechanism
> {{HddsVolume.createDbStore}} sets {{dbLoaded}} to true and logs "SchemaV3 db
> is created and loaded" before it assigns {{dbVolume}} and {{dbParentDir}}.
> {{loadDbStore}} uses the opposite order. {{HddsVolume.check}} returns early
> while the DB is not loaded and otherwise checks {{new File(dbParentDir,
> "container.db")}}. A check between the two writes sees {{dbLoaded=true}} and
> {{dbParentDir=null}}, looks for {{container.db}} relative to the working
> directory, does not find it and returns FAILED.
> {{MutableVolumeSet.handleVolumeFailures}} then fails the volume and closes
> its DB. Releases from 1.4.0 on are affected: {{check}} has read
> {{dbParentDir}} behind {{isDbLoaded()}} since HDDS-8782, and the order in
> {{createDbStore}} is older (checked in the ozone-1.3.0, 1.4.0, 2.0.0 and
> 2.1.0 tags).
> The locks do not prevent this. {{DatanodeSchemaV3FinalizeAction.execute}}
> holds the data volume set write lock, but
> {{MutableVolumeSet.checkAllVolumes}} and {{checkVolumeAsync}} take no volume
> set lock until {{handleVolumeFailures}}, and {{check}} is synchronized on the
> volume while {{createDbStore}} is not.
> h3. Trigger
> SchemaV3 finalization on a running datanode
> ({{FinalizeVersionCommandHandler}}), while a volume check runs on a volume
> being finalized: the periodic disk check (every 60 minutes by default) or an
> on demand check after an I/O error. First time formatting through
> {{VersionEndpointTask}} is not affected, because the volume checker starts
> later. The natural window is a few field writes and one log call.
> h3. Impact
> * A healthy data volume is failed and its containers are no longer served
> until the datanode restarts, so SCM replicates them elsewhere without need;
> its DB is intact on disk.
> * If it was the last data volume, or a finite tolerance is exceeded, the
> datanode stops (by code reading, not run).
> * Rare: the window is tiny and finalization happens once.
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR10FinalizeRace.java]
> ({{hadoop-hdds/container-service}}, target path and Maven command in the
> class comment) follows
> {{TestDatanodeUpgradeToSchemaV3#testDBCreatedInFinalize}}: two data volumes
> formatted before SchemaV3 without per volume DBs and a pre-finalized datanode
> started, then {{finalizeUpgrade()}} while a second thread loops
> {{MutableVolumeSet.checkAllVolumes()}} with the minimum check gap at 0. To
> hit the window reliably, a log4j appender sleeps 300 ms on the "SchemaV3 db
> is created and loaded" log line, which lies inside it; no logic is changed.
> In 3 of 3 rounds a check logged "failed health check. Could not access
> RocksDB at container.db" and failed a volume whose DB exists on disk. The
> original investigation did not hit the window without the sleep (0 of 10
> rounds). A passing test means the defect is present.
> h3. Patch
> [^CR-10-set-db-location-before-publishing-db-loaded.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3.
> The two assignments move before {{dbLoaded.set(true)}}, inside the {{try}}
> after the DB is created, the same order as {{loadDbStore}}. The write to the
> {{AtomicBoolean}} then also publishes both fields to any thread that reads
> {{isDbLoaded()}} as true, so no further synchronization is needed. By code
> reading the same window also lets a concurrent {{closeDbStore}} compute the
> relative path and leave the new DB open in {{DatanodeStoreCache}}; the
> reorder closes that too (not run).
> Covered by {{testDbParentDirSetWhenDbIsPublishedAsLoaded}} in the existing
> {{TestHddsVolume}}, which runs {{check}} from a log appender at the moment
> {{createDbStore}} logs that the DB is loaded, without threads or sleeps.
> Without the change it fails with "expected: HEALTHY but was: FAILED". With it
> the volume, volume set, health check, upgrade and container suites (135
> tests) pass and checkstyle is clean. With the patch the reproduction fails
> because no volume is failed.
> Found by code review of the datanode volume and per-disk RocksDB lifecycle
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-10230 (closest prior art: the same null {{dbParentDir}}
> symptom after a failed DB load, which added the {{isDbLoadFailure}} branch in
> {{check}}) and HDDS-11485 (volume with a missing DB passes its check). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
