Siyao Meng created HDDS-16117:
---------------------------------
Summary: Failed FSO multipart complete leaks bucket namespace quota
Key: HDDS-16117
URL: https://issues.apache.org/jira/browse/HDDS-16117
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
On an FSO bucket, a {{CompleteMultipartUpload}} that fails after a missing
parent directory is recreated permanently inflates the cached bucket
{{usedNamespace}}. The durable bucket stays correct, so the cached value
diverges from the persisted state and over-reports the namespace usage.
h3. Steps to reproduce
# Create an FSO bucket.
# Initiate a multipart upload for a key under a parent directory, for example
{{parent/file}}. This creates the {{parent}} directory and charges the bucket
{{usedNamespace}} by 1.
# Upload one part.
# Delete the parent directory {{parent/}}. The bucket {{usedNamespace}} returns
to 0 (cache and durable).
# Complete the multipart upload with an invalid part ETag. The request first
recreates the missing {{parent}} directory in the cache (charging
{{usedNamespace}} by 1), then fails with {{INVALID_PART}}.
After the failed complete the cached bucket reports {{usedNamespace = 1}} with
no backing object, while the durable bucket correctly reports {{usedNamespace =
0}}.
h3. Root cause
{{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} obtains the bucket
through {{OMKeyRequest.getBucketInfo}}, which returns the cached
{{OmBucketInfo}} by reference. For FSO,
{{S3MultipartUploadCompleteRequestWithFSO.addMissingParentsToCache}} recreates
the missing parent directories and charges the namespace with
{{omBucketInfo.incrUsedNamespace(...)}}. That charge happens before parts are
validated and mutates the cached bucket in place. When the complete later
fails, the failure path builds an error response and skips the success only
{{updateCache}}, so the charge is never reverted. The complete response cleanup
evicts the recreated directory cache entry (HDDS-11784 added
{{DIRECTORY_TABLE}} to the complete response cleanup tables), but the in place
bucket {{usedNamespace}} mutation is left orphaned.
h3. Impact
A client with normal bucket write access can repeatedly issue completes that
fail after parent recreation (deleted parent plus invalid parts) to drive a
bucket toward its {{quotaInNamespace}} with no real objects, degrading
availability of legitimate creates. The leak is confined to the in memory
cache, so it clears when the OM reloads the bucket from durable state (restart
or failover) or when a quota repair runs. Durable state is not corrupted.
h3. Relation to HDDS-11784
HDDS-11784 fixed the analogous namespace leak on the multipart abort path
({{incrUsedNamespace}} without a matching revert for missing parent
directories). The failed complete path was not covered and still leaks.
h3. Suggested fix
Do not mutate the cached bucket in place before the complete is known to
succeed. Operate on a copy of the {{OmBucketInfo}} and publish it to the cache
and DB only on the success path, so a failed complete discards the copy and
leaves the cached bucket unchanged. This mirrors the {{copyObject()}} usage
already present on the abort path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
