[
https://issues.apache.org/jira/browse/HDDS-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16117:
------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Failed FSO multipart complete leaks bucket namespace quota
> ----------------------------------------------------------
>
> Key: HDDS-16117
> URL: https://issues.apache.org/jira/browse/HDDS-16117
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> On an FSO bucket, a {{CompleteMultipartUpload}} that fails after a missing
> parent directory is recreated permanently inflates the cached bucket
> {{usedNamespace}}. The durable bucket stays correct, so the cached value
> diverges from the persisted state and over-reports the namespace usage.
> h3. Steps to reproduce
> # Create an FSO bucket.
> # Initiate a multipart upload for a key under a parent directory, for example
> {{parent/file}}. This creates the {{parent}} directory and charges the bucket
> {{usedNamespace}} by 1.
> # Upload one part.
> # Delete the parent directory {{parent/}}. The bucket {{usedNamespace}}
> returns to 0 (cache and durable).
> # Complete the multipart upload with an invalid part ETag. The request first
> recreates the missing {{parent}} directory in the cache (charging
> {{usedNamespace}} by 1), then fails with {{INVALID_PART}}.
> After the failed complete the cached bucket reports {{usedNamespace = 1}}
> with no backing object, while the durable bucket correctly reports
> {{usedNamespace = 0}}.
> h3. Root cause
> {{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} obtains the
> bucket through {{OMKeyRequest.getBucketInfo}}, which returns the cached
> {{OmBucketInfo}} by reference. For FSO,
> {{S3MultipartUploadCompleteRequestWithFSO.addMissingParentsToCache}}
> recreates the missing parent directories and charges the namespace with
> {{omBucketInfo.incrUsedNamespace(...)}}. That charge happens before parts are
> validated and mutates the cached bucket in place. When the complete later
> fails, the failure path builds an error response and skips the success only
> {{updateCache}}, so the charge is never reverted. The complete response
> cleanup evicts the recreated directory cache entry (HDDS-11784 added
> {{DIRECTORY_TABLE}} to the complete response cleanup tables), but the in
> place bucket {{usedNamespace}} mutation is left orphaned.
> h3. Impact
> A client with normal bucket write access can repeatedly issue completes that
> fail after parent recreation (deleted parent plus invalid parts) to drive a
> bucket toward its {{quotaInNamespace}} with no real objects, degrading
> availability of legitimate creates. The leak is confined to the in memory
> cache, so it clears when the OM reloads the bucket from durable state
> (restart or failover) or when a quota repair runs. Durable state is not
> corrupted.
> h3. Relation to HDDS-11784
> HDDS-11784 fixed the analogous namespace leak on the multipart abort path
> ({{incrUsedNamespace}} without a matching revert for missing parent
> directories). The failed complete path was not covered and still leaks.
> h3. Suggested fix
> Do not mutate the cached bucket in place before the complete is known to
> succeed. Operate on a copy of the {{OmBucketInfo}} and publish it to the
> cache and DB only on the success path, so a failed complete discards the copy
> and leaves the cached bucket unchanged. This mirrors the {{copyObject()}}
> usage already present on the abort path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
