Siyao Meng created HDDS-17023:
---------------------------------
Summary: Failed container push releases the receiver's space
reservation twice when the sender's half close arrives after the error
Key: HDDS-17023
URL: https://issues.apache.org/jira/browse/HDDS-17023
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-7-release-push-import-reservation-once.patch,
TestBugMC7PushImportDoubleRelease.java
h3. Mechanism
{{SendContainerRequestHandler}} reserves space on the first message of a push
and releases it in the {{finally}} of {{onError}} and of {{onCompleted}} (both
added by HDDS-12235 in 2.1.0). {{onError}} closes the output and sets it to
null but keeps {{volume}} and {{spaceToReserve}}. When the sender's half close
is delivered after that, {{onCompleted}} returns early ("Received container
without any parts") and its {{finally}} releases the same reservation again.
gRPC still delivers a half close it has already received after the server
called {{onError}}: the reproduction below, with a real Netty server and the
production sender, logs {{onError}} and then {{onCompleted}} on the same call
and ends with the reservation released twice. A later {{onNext}} that fails on
the closed output would release again (argued, not observed: gRPC dropped
queued messages in the runs).
h3. Trigger
An I/O error on the receiving datanode while it handles the last message of a
push, for example when opening or writing the tarball in the untar directory: a
container whose tarball fits in one chunk, or an error such as ENOSPC on the
last chunk of a large one. {{GrpcOutputStream.close}} sends the last chunk and
the half close without waiting. A failure on an earlier chunk releases once:
the sender sees the error before it reaches the half close.
h3. Impact
Each such failure lowers the volume's committed bytes by one extra reservation,
which is {{2 x bytesUsed}} of the pushed container as sent by the source (the
default {{2 x ozone.scm.container.size}} if the source sends no size), until
restart. The value can go below zero and is reported to SCM.
{{AvailableSpaceFilter}} then overstates the free space by that amount (argued
from the arithmetic, not run).
h3. Reproduction
[^TestBugMC7PushImportDoubleRelease.java] ({{hadoop-hdds/container-service}})
pushes with the production {{GrpcContainerUploader}} to a real Netty
{{ReplicationServer}} with a real {{ContainerImporter}} and
{{MutableVolumeSet}}. The fault is a real filesystem error: the receiver's
untar directory is read only. The sender has no local container, so no size is
sent and the default reservation applies (2 MB with a 1 MB container size). It
waits for the two handler log lines, then reads {{committed after failed
push=-2097152 reported to SCM=-2097152}}. Passed in 3 of 3 runs. A passing test
means the defect is present. Not exercised: a failure on the last chunk of a
large container, a real EIO or ENOSPC.
h3. Patch
[^MC-7-release-push-import-reservation-once.patch], against
cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
2536fe9cbc3. Released 2.1.0 to 2.2.1 are affected.
Both {{finally}} blocks call one helper that releases the reservation and sets
{{spaceToReserve}} to 0, so later callbacks of the same stream release nothing.
Covered by the existing
{{TestSendContainerRequestHandler.testSpaceReservedAndReleasedWhenOnNextFails}},
which now also delivers {{onCompleted}} after the error; it fails without the
change (for example {{expected: 0L but was: -10737418240L}}). With it the
replication package passes (131 tests) and checkstyle is clean.
Found by TLA+ model checking and code review of the datanode volume and
per-disk RocksDB lifecycle paths under HDDS-15926, on commit
cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
