peterxcli opened a new pull request, #11135: URL: https://github.com/apache/ozone/pull/11135
## What changes were proposed in this pull request? `S3MultipartUploadCompleteRequest#validateAndUpdateCache` enqueues the overwritten old key versions to deletedTable without calling `filterOutBlocksStillInUse`, unlike `OMKeyCommitRequest` (and its FSO variant), which have applied the filter since HDDS-9146 with an explicit comment that missing it "causes data loss once those shared blocks are added to deletedTable and processed by KeyDeletingService for deletion". If the key being overwritten shares any `ContainerBlockID` with the newly completed MPU key, those shared blocks are enqueued to deletedTable and physically deleted by KeyDeletingService while the live key table entry still references them. A concrete route to that state exists through bucket versioning history: 1. With bucket versioning enabled, overwriting a key retains the previous version groups (`OMKeyRequest#prepareFileInfo` -> `OmKeyInfo#addNewVersion(..., keepOldVersions=true)`), so the key table entry accumulates multiple location version groups. 2. Bucket versioning is later disabled (supported via `OMBucketSetPropertyRequest`). 3. An S3 MPU complete overwrites the key. `S3MultipartUploadCompleteRequest#getOmKeyInfo` builds the new key from the existing key table entry, and `updateLocationInfoList(partLocations, true, true)` replaces only the latest version's block list, so the older version groups' blocks remain referenced by the new key. At the same time the overwrite branch enqueues ALL of `keyToDelete`'s blocks (older version groups included) to deletedTable. 4. KeyDeletingService deletes blocks that the live key table entry still references -> data loss. Not reachable via retried/replayed completes (a duplicate complete fails with `NO_SUCH_MULTIPART_UPLOAD_ERROR` because the multipartInfoTable entry is removed in the same transaction) nor via hsync block overlap (part blocks are freshly allocated). **The fix** applies the same `filterOutBlocksStillInUse` protection in the MPU complete overwrite path before adding the old key versions to `allKeyInfoToRemove`, mirroring `OMKeyCommitRequest`. The shared base class covers both the OBS/LEGACY and FSO paths. One behavior note: overwriting a key that has no blocks no longer writes a deletedTable entry, because the filter drops block-less keys from the delete list. This matches the commit path, which only enqueues when the filtered list is non-empty; `TestS3MultipartUploadCompleteRequest#testValidateAndUpdateCacheSuccess`'s overwrite expectation is updated accordingly. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16314 ## How was this patch tested? * New unit test `testOverwrittenKeySharedBlocksNotEnqueuedForDeletion` in `TestS3MultipartUploadCompleteRequest`, with FSO helper overrides so it runs in both `TestS3MultipartUploadCompleteRequest` and `TestS3MultipartUploadCompleteRequestWithFSO`. It completes an MPU over a key table entry that shares one block with the committed part and holds one block of its own, then asserts the deletedTable contains only the old-only block while the completed key still references the shared one. The test fails without the production change. * All `TestS3*` unit tests in ozone-manager pass (175 tests), and `checkstyle:check` passes on the module. Generated-by: Claude Code (claude-fable-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
