Siyao Meng created HDDS-17014:
---------------------------------
Summary: Volume check during SchemaV3 finalization can fail a
healthy data volume because createDbStore sets dbLoaded before dbParentDir
Key: HDDS-17014
URL: https://issues.apache.org/jira/browse/HDDS-17014
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-10-set-db-location-before-publishing-db-loaded.patch,
TestBugCR10FinalizeRace.java
h3. Mechanism
{{HddsVolume.createDbStore}} sets {{dbLoaded}} to true and logs "SchemaV3 db is
created and loaded" before it assigns {{dbVolume}} and {{dbParentDir}}.
{{loadDbStore}} uses the opposite order. {{HddsVolume.check}} returns early
while the DB is not loaded and otherwise checks {{new File(dbParentDir,
"container.db")}}. A check between the two writes sees {{dbLoaded=true}} and
{{dbParentDir=null}}, looks for {{container.db}} relative to the working
directory, does not find it and returns FAILED.
{{MutableVolumeSet.handleVolumeFailures}} then fails the volume and closes its
DB. Releases from 1.4.0 on are affected: {{check}} has read {{dbParentDir}}
behind {{isDbLoaded()}} since HDDS-8782, and the order in {{createDbStore}} is
older (checked in the ozone-1.3.0, 1.4.0, 2.0.0 and 2.1.0 tags).
The locks do not prevent this. {{DatanodeSchemaV3FinalizeAction.execute}} holds
the data volume set write lock, but {{MutableVolumeSet.checkAllVolumes}} and
{{checkVolumeAsync}} take no volume set lock until {{handleVolumeFailures}},
and {{check}} is synchronized on the volume while {{createDbStore}} is not.
h3. Trigger
SchemaV3 finalization on a running datanode
({{FinalizeVersionCommandHandler}}), while a volume check runs on a volume
being finalized: the periodic disk check (every 60 minutes by default) or an on
demand check after an I/O error. First time formatting through
{{VersionEndpointTask}} is not affected, because the volume checker starts
later. The natural window is a few field writes and one log call.
h3. Impact
* A healthy data volume is failed and its containers are no longer served until
the datanode restarts, so SCM replicates them elsewhere without need; its DB is
intact on disk.
* If it was the last data volume, or a finite tolerance is exceeded, the
datanode stops (by code reading, not run).
* Rare: the window is tiny and finalization happens once.
h3. Reproduction
PASS on unmodified source. [^TestBugCR10FinalizeRace.java]
({{hadoop-hdds/container-service}}, target path and Maven command in the class
comment) follows {{TestDatanodeUpgradeToSchemaV3#testDBCreatedInFinalize}}: two
data volumes formatted before SchemaV3 without per volume DBs and a
pre-finalized datanode started, then {{finalizeUpgrade()}} while a second
thread loops {{MutableVolumeSet.checkAllVolumes()}} with the minimum check gap
at 0. To hit the window reliably, a log4j appender sleeps 300 ms on the
"SchemaV3 db is created and loaded" log line, which lies inside it; no logic is
changed. In 3 of 3 rounds a check logged "failed health check. Could not access
RocksDB at container.db" and failed a volume whose DB exists on disk. The
original investigation did not hit the window without the sleep (0 of 10
rounds). A passing test means the defect is present.
h3. Patch
[^CR-10-set-db-location-before-publishing-db-loaded.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
The two assignments move before {{dbLoaded.set(true)}}, inside the {{try}}
after the DB is created, the same order as {{loadDbStore}}. The write to the
{{AtomicBoolean}} then also publishes both fields to any thread that reads
{{isDbLoaded()}} as true, so no further synchronization is needed. By code
reading the same window also lets a concurrent {{closeDbStore}} compute the
relative path and leave the new DB open in {{DatanodeStoreCache}}; the reorder
closes that too (not run).
Covered by {{testDbParentDirSetWhenDbIsPublishedAsLoaded}} in the existing
{{TestHddsVolume}}, which runs {{check}} from a log appender at the moment
{{createDbStore}} logs that the DB is loaded, without threads or sleeps.
Without the change it fails with "expected: HEALTHY but was: FAILED". With it
the volume, volume set, health check, upgrade and container suites (135 tests)
pass and checkstyle is clean. With the patch the reproduction fails because no
volume is failed.
Found by code review of the datanode volume and per-disk RocksDB lifecycle
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-10230 (closest prior art: the same null {{dbParentDir}}
symptom after a failed DB load, which added the {{isDbLoadFailure}} branch in
{{check}}) and HDDS-11485 (volume with a missing DB passes its check). The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
