Siyao Meng created HDDS-17026:
---------------------------------
Summary: Datanode container create that fails with an unchecked
exception leaks the reserved container size on the volume until restart
Key: HDDS-17026
URL: https://issues.apache.org/jira/browse/HDDS-17026
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-10-release-reservation-on-unchecked-create-failure.patch,
TestBugMC10DbLoadReservationLeak.java
h3. Mechanism
{{KeyValueContainer.create}} reserves the container size ({{maxSize}}, 5 GB by
default) on the volume that {{chooseVolume}} picks.
{{KeyValueHandler.handleCreateContainer}} gives that reservation back only in
its {{catch (StorageContainerException)}} branch. When {{create}} fails with an
unchecked exception after the volume was chosen, the exception passes that
branch, {{KeyValueHandler.handle}} turns it into a {{CONTAINER_INTERNAL_ERROR}}
response, and the volume's committed bytes stay raised by the full container
size. Nothing recomputes committed bytes while the datanode runs, so each such
attempt adds another {{maxSize}} until restart. Present since HDDS-12810
(2.1.0), which moved the reservation into {{chooseVolume}}; 2.0.0 is not
affected (checked in the ozone-2.0.0, 2.1.0 and 2.2.1 tags).
h3. Trigger
Today this is reached through the state described in HDDS-11485: a data volume
whose per-volume DB did not load at restart (for example its {{DS-<id>}}
directory or {{container.db}} is gone, or a dedicated DB volume was lost) stays
in service with {{dbParentDir}} unset. Every container create placed on it,
including the implicit create {{HddsDispatcher}} issues for the first write to
a new container, throws {{NullPointerException("dbParentDir == null")}} from
{{KeyValueContainerLocationUtil.getContainerDBFile}} after the reservation. Any
other unchecked failure in {{create}} after the volume is chosen leaks the same
way.
h3. Impact
* Each failed create leaves {{maxSize}} committed on the volume. In the
reproduction three creates left 15 GB committed on an empty volume, and the
storage report sent to SCM carries it.
* The volume choosing policy counts committed bytes against the volume's free
space, so the phantom bytes grow until they fill it; after that the policy
stops picking the volume (by code reading, not run). On the known trigger every
create on that volume fails anyway, so the harm is the wrong committed and free
space figures, on the datanode and in what SCM sees.
* The creates themselves fail either way; this report is only about the space
they leave reserved.
h3. Reproduction
PASS on unmodified source. [^TestBugMC10DbLoadReservationLeak.java]
({{hadoop-hdds/container-service}}, target path and Maven command in the class
comment) formats a data volume whose DB lives on a dedicated DB volume, loses
the DB disk and drops it from the configuration, then rebuilds the volume set
with a real {{StorageVolumeChecker}}, loads the DBs and runs the startup volume
check. The volume is {{NORMAL}} with {{isDbLoaded=false}} and
{{dbParentDir=null}}. Three CreateContainer requests go through
{{KeyValueHandler.handle}}; each returns {{CONTAINER_INTERNAL_ERROR}} with
"dbParentDir == null", no container is added, and committed bytes grow to
5368709120, 10737418240 and 16106127360. A later volume check still finds the
volume healthy and its report carries 3 x 5 GB committed. A passing test means
the defect is present.
h3. Patch
[^MC-10-release-reservation-on-unchecked-create-failure.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3.
{{handleCreateContainer}} now releases the reservation in its {{finally}} block
whenever the container was not created, instead of only in the
{{StorageContainerException}} branch. {{ContainerData.releaseCommitSpace}},
unchanged, already gives back the space only while the reservation is still
held and then clears the flag, so a path that already released it is not
released twice, and the branch for an already existing container, which
reserves nothing, is unaffected. Volume state handling is not changed: with the
patch such a volume still stays in service and the creates still fail, but
nothing stays reserved. The root cause of that volume state is tracked in
HDDS-11485; this fix is independent of it.
Covered by {{testCreateContainerReleasesSpaceOnUncheckedFailure}} in the
existing {{TestKeyValueHandler}}: a schema V3 volume with its working directory
but no per-volume DB, one CreateContainer through {{KeyValueHandler.handle}}.
Without the change it fails with "expected: 0L but was: 5368709120L"; with it
the volume has no committed bytes. The new
{{testCreateContainerKeepsSpaceOfCreatedContainer}} covers the guard: a
successful create keeps {{maxSize}} committed with the reservation flag set,
and a repeated create of the same container leaves both unchanged; it fails if
the release is made unconditional. The existing
{{testCreateContainerWithFailure}}, which makes {{ContainerSet.addContainer}}
throw {{StorageContainerException}}, covers the checked path and passes
unchanged. With the change the handler, container, dispatcher, persistence and
{{OzoneContainer}} suites (551 tests) pass and checkstyle is clean. With the
patch the reproduction fails at its committed bytes check after the first
create, while the volume is still {{NORMAL}}.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-11485 (the volume state that reaches this today) and
HDDS-17009 (releases the reservation on a volume that the retry loop in
{{KeyValueContainer.create}} gives up on; the two patches are independent and
apply together). The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
