[
https://issues.apache.org/jira/browse/HDDS-17023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17023:
------------------------------
Attachment: TestBugMC7PushImportDoubleRelease.java
> Failed container push releases the receiver's space reservation twice when
> the sender's half close arrives after the error
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17023
> URL: https://issues.apache.org/jira/browse/HDDS-17023
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-7-release-push-import-reservation-once.patch,
> TestBugMC7PushImportDoubleRelease.java
>
>
> h3. Mechanism
> {{SendContainerRequestHandler}} reserves space on the first message of a push
> and releases it in the {{finally}} of {{onError}} and of {{onCompleted}}
> (both added by HDDS-12235 in 2.1.0). {{onError}} closes the output and sets
> it to null but keeps {{volume}} and {{spaceToReserve}}. When the sender's
> half close is delivered after that, {{onCompleted}} returns early ("Received
> container without any parts") and its {{finally}} releases the same
> reservation again. gRPC still delivers a half close it has already received
> after the server called {{onError}}: the reproduction below, with a real
> Netty server and the production sender, logs {{onError}} and then
> {{onCompleted}} on the same call and ends with the reservation released
> twice. A later {{onNext}} that fails on the closed output would release again
> (argued, not observed: gRPC dropped queued messages in the runs).
> h3. Trigger
> An I/O error on the receiving datanode while it handles the last message of a
> push, for example when opening or writing the tarball in the untar directory:
> a container whose tarball fits in one chunk, or an error such as ENOSPC on
> the last chunk of a large one. {{GrpcOutputStream.close}} sends the last
> chunk and the half close without waiting. A failure on an earlier chunk
> releases once: the sender sees the error before it reaches the half close.
> h3. Impact
> Each such failure lowers the volume's committed bytes by one extra
> reservation, which is {{2 x bytesUsed}} of the pushed container as sent by
> the source (the default {{2 x ozone.scm.container.size}} if the source sends
> no size), until restart. The value can go below zero and is reported to SCM.
> {{AvailableSpaceFilter}} then overstates the free space by that amount
> (argued from the arithmetic, not run).
> h3. Reproduction
> [^TestBugMC7PushImportDoubleRelease.java] ({{hadoop-hdds/container-service}})
> pushes with the production {{GrpcContainerUploader}} to a real Netty
> {{ReplicationServer}} with a real {{ContainerImporter}} and
> {{MutableVolumeSet}}. The fault is a real filesystem error: the receiver's
> untar directory is read only. The sender has no local container, so no size
> is sent and the default reservation applies (2 MB with a 1 MB container
> size). It waits for the two handler log lines, then reads {{committed after
> failed push=-2097152 reported to SCM=-2097152}}. Passed in 3 of 3 runs. A
> passing test means the defect is present. Not exercised: a failure on the
> last chunk of a large container, a real EIO or ENOSPC.
> h3. Patch
> [^MC-7-release-push-import-reservation-once.patch], against
> cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at
> 2536fe9cbc3. Released 2.1.0 to 2.2.1 are affected.
> Both {{finally}} blocks call one helper that releases the reservation and
> sets {{spaceToReserve}} to 0, so later callbacks of the same stream release
> nothing. Covered by the existing
> {{TestSendContainerRequestHandler.testSpaceReservedAndReleasedWhenOnNextFails}},
> which now also delivers {{onCompleted}} after the error; it fails without
> the change (for example {{expected: 0L but was: -10737418240L}}). With it the
> replication package passes (131 tests) and checkstyle is clean.
> Found by TLA+ model checking and code review of the datanode volume and
> per-disk RocksDB lifecycle paths under HDDS-15926, on commit
> cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
