Siyao Meng created HDDS-16785:
---------------------------------
Summary: Part commit rejected with NO_SUCH_MULTIPART_UPLOAD_ERROR
leaves its open key behind, so its blocks are queued for deletion twice
Key: HDDS-16785
URL: https://issues.apache.org/jira/browse/HDDS-16785
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-delete-part-open-key-on-rejected-part-commit.patch,
TestBugMC5RejectedPartCommitOpenKey.java
h3. Mechanism
When {{S3MultipartUploadCommitPartRequest.validateAndUpdateCache}} finds no
{{multipartInfoTable}} entry for the upload, it throws
{{NO_SUCH_MULTIPART_UPLOAD_ERROR}} before any cache update.
{{S3MultipartUploadCommitPartResponse.checkAndUpdateDB}} then moves the blocks
named in the commit to {{deletedTable}} but does not delete the open key of the
part. Part open keys are not flagged as multipart keys, so
{{OpenKeyCleanupService}} selects the leftover entry after
{{ozone.om.open.key.expire.threshold}} (default 7d) and
{{OMOpenKeysDeleteRequest}} puts the same blocks into {{deletedTable}} a second
time under another key.
The comment in that response branch ("So, delete this part information.") and
{{TestS3MultipartUploadCommitPartResponse.testWithMultipartUploadError}}, which
asserts that the open key is gone but never creates it, indicate that the entry
was meant to be removed.
h3. Trigger
# InitiateMultipartUpload, then open a part and write data.
# The upload is aborted before the part is committed (run with a client abort,
an abort by {{MultipartUploadCleanupService}} removes the same entry).
# The part commit is rejected with {{NO_SUCH_MULTIPART_UPLOAD_ERROR}}.
No fault is needed. A part commit that arrives after CompleteMultipartUpload
takes the same branch (same code path, no separate reproduction).
h3. Impact
* Every rejected part commit leaves an open key entry until the open key
expiry, and its blocks are then sent to SCM for deletion a second time.
Datanodes skip duplicate block deletions since HDDS-11267, so no wrong datanode
state is expected. I ran the OM side only, without datanodes.
* The leftover open key is still a valid write session. The stock client does
not use it after the failed part commit, but a {{CommitKey}} on it is accepted
and publishes a key whose block is already in {{deletedTable}}. The block is
then sent for deletion while the key stays in {{keyTable}} (run against a real
OM with a recording SCM client, actual data deletion not run).
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC5RejectedPartCommitOpenKey.java] ({{hadoop-ozone/ozone-manager}})
drives one real OM with its background services through
{{OzoneManagerProtocol}}, with a recording SCM block client and no datanodes.
The open key expiry is shortened to 200 ms and the two services are held with
their own suspend and resume hooks so the tables can be read between the steps.
The first test shows the same block ID in two {{deletedTable}} entries and sent
to SCM twice. The second commits the leftover open key as a key and shows its
block sent for deletion while the key remains. A passing test means the defect
is present.
h3. Patch
[^MC-5-delete-part-open-key-on-rejected-part-commit.patch], against
d067ba38e7aea917db7efd290b4af53987ec985a. It also applies to master at
a6332fb5987.
On the rejected path the request adds a delete marker for the part open key to
the table cache and the response deletes the key in the batch that writes
{{deletedTable}}. Because the open key is then gone, the blocks that were
allocated to the part but not named in the commit are added to the same
{{deletedTable}} entry. Today only the open key cleanup releases those, so
without that addition they would leak.
Covered by {{testValidateAndUpdateCacheMultipartNotFoundDeletesOpenKey}} in the
existing {{TestS3MultipartUploadCommitPartRequest}} (inherited by the FSO
subclass), and {{testWithMultipartUploadError}} now creates the open key so
that its existing assertion is meaningful. Without the change both fail for
both layouts because the open key is still present. With it the multipart
request and response unit suites and {{TestOMOpenKeysDeleteRequest}} (142
tests), the open key cleanup, multipart cleanup and key deleting service suites
(37 tests) and the multipart tests of {{TestOzoneRpcClient}} on a mini cluster
(29 tests, including {{testAbortUploadFailWithInProgressPartUpload}} and
{{testCommitPartAfterCompleteUpload}}) pass, checkstyle is clean. With the
patch both tests of the reproduction fail, because the open key no longer
exists after the rejected commit. Entries left behind by earlier rejected
commits are not touched, the open key cleanup removes them as before.
Found by TLA+ model checking and code review of the OM expired multipart upload
cleanup paths under HDDS-15926, on commit
d067ba38e7aea917db7efd290b4af53987ec985a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
