[
https://issues.apache.org/jira/browse/HDDS-16771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16771:
------------------------------
Attachment: CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch
> CompleteMultipartUpload over an hsync'ed key does not flag the hsync writer's
> open entry
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-16771
> URL: https://issues.apache.org/jira/browse/HDDS-16771
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch,
> TestBugCR5HsyncMpuComplete.java
>
>
> h3. Mechanism
> {{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} replaces an
> existing file table entry without checking {{HSYNC_CLIENT_ID}} on it and
> without adding {{OVERWRITTEN_HSYNC_KEY}} to the hsync writer's open entry.
> {{OMKeyCommitRequestWithFSO}} ({{:198-213}}) and {{OMKeyCommitRequest}} do
> both for a normal overwrite. No code under {{request/s3/multipart/}} reads or
> writes hsync metadata, so every bucket layout that allows hsync is affected;
> it was shown on FSO.
> h3. Trigger
> {{ozone.fs.hsync.enabled}} and {{ozone.hbase.enhancements.allowed}} are true
> (both default false), FSO bucket.
> # Writer A creates a file through {{ofs}} / {{o3fs}} and hsyncs it.
> # Another client completes a multipart upload on the same key.
> # Writer A hsyncs after crossing a block boundary, or closes.
> h3. Impact
> Writer A's stale hsync or close is accepted and silently replaces the
> completed multipart object, which is moved to the deleted table. A normal
> overwrite in the same position rejects writer A with {{KEY_NOT_FOUND}}. At
> complete time writer A's hsync'ed version goes to the deleted table with its
> blocks, and A's accepted commit then writes a key table entry that references
> those same blocks, so the file that comes back points at blocks already
> queued for deletion (from code; block deletion was not run).
> h3. Reproduction
> PASS: [^TestBugCR5HsyncMpuComplete.java] ({{hadoop-ozone/integration-test}}),
> public client APIs on a mini cluster.
> h3. Suggested fix
> In {{S3MultipartUploadCompleteRequest}}, for both layouts, when
> {{keyToDelete}} carries {{HSYNC_CLIENT_ID}}, look up that writer's open entry
> and add {{OVERWRITTEN_HSYNC_KEY}} in the same transaction, and have the
> response persist it. Whether such a complete should be rejected instead of
> fencing the writer is a product question.
> h3. Patch
> [^CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> 9c6bee807c8.
> When the entry being replaced carries {{HSYNC_CLIENT_ID}}, the request looks
> up that writer's open key ({{getOpenFileName}} for FSO, {{getOpenKey}}
> otherwise), adds {{OVERWRITTEN_HSYNC_KEY}} to the open table cache and hands
> the entry to the response, which writes it in the same batch. This is the
> handling the commit requests apply to a normal overwrite, so the writer's
> later hsync or close is rejected with {{KEY_NOT_FOUND}} and
> {{OpenKeyCleanupService}} removes the flagged open key. The marking sits
> after the last check that can reject the complete. The response constructors
> gain two parameters.
> Covered by {{testOverwrittenHsyncKeyIsMarked}} in the existing
> {{TestS3MultipartUploadCompleteRequest}} (inherited by the FSO variant),
> which fails without the change because the open key lacks
> {{overwrittenHsyncKey}}. The multipart request and response unit suites (122
> tests) and {{TestHSync}} (42 tests) pass, checkstyle is clean. Run against
> the fix, the attached reproduction's stale hsync and stale close are rejected
> with "is already deleted/overwritten". The FSO multipart mini cluster suite
> was not run.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
