smengcl opened a new pull request, #10984: URL: https://github.com/apache/ozone/pull/10984
Generated-by: Claude Code (Opus 4.8) https://issues.apache.org/jira/browse/HDDS-16117 ## What changes were proposed in this pull request? A multipart upload complete on an FSO bucket can leak the bucket `usedNamespace` value in the cache. TLA+ formal verification with Specula found this bug. Trace validation on a real trace broke the `QuotaExactness` invariant. An independent model check broke the `FailedOperationIsolation` invariant. Both point to the failed complete path. The cause is as follows: 1. `validateAndUpdateCache` gets the bucket with `getBucketInfo`. `getBucketInfo` returns the cached `OmBucketInfo` by reference. 2. For FSO, `addMissingParentsToCache` makes a missing parent directory again. It calls `incrUsedNamespace(...)` before it validates the parts. This call changes the cached bucket in place. 3. The complete then fails. For example, it fails with `INVALID_PART`. 4. The failure path makes an error response. It does not call the success only `updateCache`. Thus the OM does not correct the change. 5. The cache cleanup removes the new parent directory, but the `usedNamespace` change stays. The result: the cached bucket shows one namespace object that does not exist. The durable bucket stays correct. A client can send this failed complete many times. This can fill the namespace quota of the bucket with no real objects. An OM restart, a failover, or a quota repair clears the cache. HDDS-11784 corrected the abort path. It did not correct the failed complete path. Fix: use a copy of the cached `OmBucketInfo` in `validateAndUpdateCache`, and publish it with `updateCache` only when the complete is successful. If the complete fails, the OM does not use the copy, and the cache stays correct. This is the same `copyObject()` method as the abort path. A note on `getBucketInfo` tells callers to copy the bucket before they change it. ## How was this patch tested? New test `TestOzoneClientMultipartUploadWithFSO#testFailedCompleteAfterParentDeletionDoesNotLeakNamespaceQuota`: start an FSO MPU below a parent directory, upload a part, delete the parent, then complete with a wrong ETag and get `INVALID_PART`. The cached and durable bucket `usedNamespace` must both be 0. The test fails on master (cached value is 1) and passes with this patch. Also pass: the 4 complete request and response unit test classes; the FSO tests `testCommitPartAfterCompleteUpload`, `testMultipartUploadWithDiscardedUnusedPartSize`, and `testAbortMultipartUploadSuccessWithMissingParentDirectories`; checkstyle for `ozone-manager` and `integration-test`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
