Siyao Meng created HDDS-17019:
---------------------------------
Summary: Failure of a datanode DB volume is not propagated to the
data volumes whose RocksDB it holds, so they stay in service
Key: HDDS-17019
URL: https://issues.apache.org/jira/browse/HDDS-17019
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-fail-data-volumes-of-failed-db-volume.patch,
TestBugMC4DbVolumeFailureNotPropagated.java
h3. Mechanism
With {{hdds.datanode.container.db.dir}} set, the schema V3 RocksDB of each data
volume lives on a separate DB volume. {{OzoneContainer}} registers a failed
volume listener only on the data volume set; the DB volume set has none. When
the periodic check fails a DB volume, {{DbVolume.failVolume}} closes and
uncaches the DB stores below it, but the data volumes that use it are left
NORMAL: they keep their containers, are reported to SCM as healthy and are
still chosen for new containers. {{DatanodeStoreCache.getDB}} reopens a store
that is not cached on its next access, so those data volumes go on using a
RocksDB on the failed DB volume.
{{HddsVolume.check}} only checks that the data volume's own RocksDB files can
be read and opened. It does not look at the DB volume's state, so a DB volume
that fails its own check while those files stay readable (its root directory no
longer writable, as reproduced here, or a file system remounted read only after
errors) leaves the data volumes in service. A DB disk that becomes unreadable
is already caught by each data volume's own check.
h3. Trigger
# A datanode with two or more DB volumes. With a single DB volume the datanode
already shuts down when it fails, because the default
{{hdds.datanode.failed.db.volumes.tolerated}} of -1 requires one healthy DB
volume.
# A DB volume fails its periodic disk check while the RocksDB files on it stay
readable.
No client action is involved.
h3. Impact
* The data volumes whose DB is on the failed DB volume stay in service until
the datanode restarts. SCM is not told and their containers are not
re-replicated.
* New containers are placed on those data volumes and their block metadata is
written to the RocksDB that the cache reopened on the failed DB volume. In the
reproduction the DB files themselves are still writable, so those writes
succeed. On a file system remounted read only the reopen or the writes would
fail instead (argued from the code, not run).
h3. Reproduction
PASS, deterministic, unmodified source, needs a POSIX file system and a non
root user. [^TestBugMC4DbVolumeFailureNotPropagated.java]
({{hadoop-hdds/container-service}}) starts a real {{OzoneContainer}} with two
data volumes and two DB volumes; one data volume has its RocksDB on a DB
volume, the other on itself. The root directory of that DB volume is made read
only and the real {{StorageVolumeChecker}} fails it through
{{MutableVolumeSet.checkAllVolumes}}. The data volume set check that follows
passes, because the RocksDB files stay readable. The data volume stays NORMAL
with its DB loaded, a new container is placed on it with the round robin
policy, its block is committed to a DB file under the failed DB volume, and the
node report shows the data volume as not failed. A passing test means the
defect is present.
h3. Patch
[^MC-4-fail-data-volumes-of-failed-db-volume.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
{{OzoneContainer}} registers a failed volume listener on the DB volume set. It
collects the active data volumes whose DB volume has failed and passes them to
{{MutableVolumeSet.handleVolumeFailures}} of the data volume set (made public
for this), which is the existing path for a failed data volume: it marks them
failed, closes their DBs, applies
{{hdds.datanode.failed.data.volumes.tolerated}} and runs the data set listener
that removes their containers and sends a full container report. The listener
runs after the DB set has released its lock, so no locks are nested. Checking
the DB volume from {{HddsVolume.check}} instead would only act in the next
periodic check, because data volumes are checked before DB volumes in each
cycle.
This changes behaviour on purpose, on nodes with two or more DB volumes: losing
a DB disk now fails every data volume whose DB is on it, their containers are
re-replicated, and if that exceeds the tolerated number of failed data volumes
the datanode shuts down, as for any other data volume loss.
Not covered: a DB volume that fails {{StorageVolumeUtil.checkVolume}} during
the SCM version handshake at startup. {{VersionEndpointTask}} fails it directly
through {{MutableVolumeSet.failVolume}} while holding the DB set lock, so the
listener does not run there, and the data volumes whose DB was already loaded
on it stay in service. Covering it would need a separate check in that task.
Covered by {{testDbVolumeFailureFailsDataVolumesWithDbOnIt}} in the existing
{{TestOzoneContainer}}, with three data volumes whose RocksDB is on the failing
DB volume, on the other DB volume and on the data volume itself. Without the
change it fails because no data volume is failed; it also fails if every data
volume with a DB volume is failed. With it the {{TestOzoneContainer}}, volume,
DB volume, volume checker and volume health check suites (138 tests) and
{{TestOzoneContainerWithTLS}}, an integration test that starts a real
{{OzoneContainer}} with a DB volume configured, pass, and checkstyle is clean.
With the patch the reproduction fails because the data volume is FAILED.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-6616 and HDDS-6541, which added the DB volume check and the
separate DB location. The attached patch is a proposal for review. Generated
with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
