Siyao Meng created HDDS-16771:
---------------------------------
Summary: CompleteMultipartUpload over an hsync'ed key does not
flag the hsync writer's open entry
Key: HDDS-16771
URL: https://issues.apache.org/jira/browse/HDDS-16771
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch,
TestBugCR5HsyncMpuComplete.java
h3. Mechanism
{{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} replaces an
existing file table entry without checking {{HSYNC_CLIENT_ID}} on it and
without adding {{OVERWRITTEN_HSYNC_KEY}} to the hsync writer's open entry.
{{OMKeyCommitRequestWithFSO}} ({{:198-213}}) and {{OMKeyCommitRequest}} do both
for a normal overwrite. No code under {{request/s3/multipart/}} reads or writes
hsync metadata, so every bucket layout that allows hsync is affected; it was
shown on FSO.
h3. Trigger
{{ozone.fs.hsync.enabled}} and {{ozone.hbase.enhancements.allowed}} are true
(both default false), FSO bucket.
# Writer A creates a file through {{ofs}} / {{o3fs}} and hsyncs it.
# Another client completes a multipart upload on the same key.
# Writer A hsyncs after crossing a block boundary, or closes.
h3. Impact
Writer A's stale hsync or close is accepted and silently replaces the completed
multipart object, which is moved to the deleted table. A normal overwrite in
the same position rejects writer A with {{KEY_NOT_FOUND}}. At complete time
writer A's hsync'ed version goes to the deleted table with its blocks, and A's
accepted commit then writes a key table entry that references those same
blocks, so the file that comes back points at blocks already queued for
deletion (from code; block deletion was not run).
h3. Reproduction
PASS: [^TestBugCR5HsyncMpuComplete.java] ({{hadoop-ozone/integration-test}}),
public client APIs on a mini cluster.
h3. Suggested fix
In {{S3MultipartUploadCompleteRequest}}, for both layouts, when {{keyToDelete}}
carries {{HSYNC_CLIENT_ID}}, look up that writer's open entry and add
{{OVERWRITTEN_HSYNC_KEY}} in the same transaction, and have the response
persist it. Whether such a complete should be rejected instead of fencing the
writer is a product question.
h3. Patch
[^CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
When the entry being replaced carries {{HSYNC_CLIENT_ID}}, the request looks up
that writer's open key ({{getOpenFileName}} for FSO, {{getOpenKey}} otherwise),
adds {{OVERWRITTEN_HSYNC_KEY}} to the open table cache and hands the entry to
the response, which writes it in the same batch. This is the handling the
commit requests apply to a normal overwrite, so the writer's later hsync or
close is rejected with {{KEY_NOT_FOUND}} and {{OpenKeyCleanupService}} removes
the flagged open key. The marking sits after the last check that can reject the
complete. The response constructors gain two parameters.
Covered by {{testOverwrittenHsyncKeyIsMarked}} in the existing
{{TestS3MultipartUploadCompleteRequest}} (inherited by the FSO variant), which
fails without the change because the open key lacks {{overwrittenHsyncKey}}.
The multipart request and response unit suites (122 tests) and {{TestHSync}}
(42 tests) pass, checkstyle is clean. Run against the fix, the attached
reproduction's stale hsync and stale close are rejected with "is already
deleted/overwritten". The FSO multipart mini cluster suite was not run.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
