[
https://issues.apache.org/jira/browse/HDDS-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16763:
------------------------------
Attachment: TestBugMC2FailedMpuOrphanDir.java
> A failed FSO CompleteMultipartUpload leaves missing parent directories in the
> cache, so later acknowledged creates become orphans
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16763
> URL: https://issues.apache.org/jira/browse/HDDS-16763
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments:
> MC-2-withdraw-cache-entries-of-failed-fso-multipart-complete.patch,
> TestBugMC2FailedMpuOrphanDir.java
>
>
> h3. Mechanism
> {{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} adds the missing
> FSO parent directories
> ({{S3MultipartUploadCompleteRequestWithFSO.addMissingParentsToCache}}) and a
> synthesized open file entry to the table cache before checks that can still
> fail ({{NOT_A_FILE}}, {{NO_SUCH_MULTIPART_UPLOAD_ERROR}}, the part list
> checks, {{QUOTA_EXCEEDED}} for the key). The error response writes nothing to
> RocksDB, and nothing removes those cache entries until the epoch is evicted
> after flush.
> {{OzoneManagerStateMachine.applyTransaction}} does not wait for the double
> buffer flush, so a later request can be applied while the failed request's
> directories are still visible in the cache.
> h3. Trigger
> On an FSO bucket, a {{CompleteMultipartUpload}} that fails (for example an
> unknown upload ID) for a key whose parent directories do not exist, followed
> by a mkdir or key create and commit under those parents, applied before the
> failed request's epoch is flushed and evicted.
> h3. Impact
> The later request is acknowledged OK and persists a directory or file row
> whose parent directory never reaches RocksDB. After eviction the acknowledged
> directory or key is unreachable ({{FILE_NOT_FOUND}} / {{KEY_NOT_FOUND}}),
> stays unreachable after OM restart, and its row and namespace charge remain
> as an orphan. The outcome also depends on each OM's flush timing: an OM that
> applies the later create after its own eviction creates the parents itself,
> so {{directoryTable}} and {{usedNamespace}} can differ between OMs (argued,
> not run).
> h3. Reproduction
> PASS. [^TestBugMC2FailedMpuOrphanDir.java]
> ({{hadoop-ozone/integration-test}}), uninstrumented jars, single OM mini
> cluster. With two racing client threads and no hooks, 4 of 300 acknowledged
> mkdirs were lost. Holding the flush with
> {{OzoneManagerDoubleBuffer.pause/unpause}} makes it deterministic. The cross
> OM consequence is argued, not run.
> h3. Suggested fix
> Withdraw the directory and open file cache entries when the complete fails,
> under the bucket lock, or restructure the method so that it publishes them
> only on success. A plain move of {{addMissingParentsToCache}} and
> {{addMultiPartToCache}} below the validation does not work, because the
> method resolves the key through those cache entries in between (the complete
> after parent deletion case of HDDS-11784 would start failing). HDDS-16117
> fixed the bucket quota side of this site only.
> h3. Patch
> [^MC-2-withdraw-cache-entries-of-failed-fso-multipart-complete.patch],
> against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master
> at 9c6bee807c8.
> {{addMissingParentsToCache}} returns the directory table keys it added, and
> the existing {{catch}} block calls a new hook,
> {{removeMissingEntriesFromCache}}, which the FSO request implements by
> replacing those entries and the synthesized open key entry with entries
> without a value at the same transaction index, while the bucket lock is still
> held. The keys were verified absent under that lock before they were added,
> so this restores the state before the request, and a later request creates
> the parents itself on every OM. The success path is untouched. Publishing
> only on success would be the cleaner rule but needs the key resolution on the
> success path reworked; that is a reviewer's call.
> Covered by {{testFailedCompleteDoesNotLeaveMissingParentsInCache}} in the
> existing {{TestS3MultipartUploadCompleteRequestWithFSO}} and
> {{testDirectoryCreatedAfterFailedCompleteIsPersistedWithItsParent}} in the
> existing {{TestOzoneClientMultipartUploadWithFSO}} (double buffer pause and
> unpause on the shared cluster). Both fail without the change. The multipart
> request and response unit suites (127 tests) and all of
> {{TestOzoneClientMultipartUploadWithFSO}} (33 tests, including the HDDS-16117
> and HDDS-11784 cases) pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
