chungen0126 opened a new pull request, #12106:
URL: https://github.com/apache/ozone/pull/12106

   ## What changes were proposed in this pull request?
   
   On FSO buckets, S3 DeleteObjects with a directory key (`dir/`) deletes the 
whole subtree, including objects that are not listed in the request. A single 
DeleteObject on the same key is a no-op. Since HDDS-10572 (1.4.1), multi-delete 
calls `deleteKeys`. Its FSO handler, `OmKeysDeleteRequestWithFSO`, deletes 
directories recursively, as recursive bucket deletion through ofs needs. S3A 
with `fs.s3a.directory.marker.retention=delete` sends such a request for the 
parent markers after every write, which wipes sibling data. That retention mode 
is the default before Hadoop 3.4 and is used by Flink's S3 plugin.
   
   This PR adds an optional `recursive` flag to `DeleteKeysRequest`. It 
defaults to `true`, so existing callers keep the recursive behavior.
   
   - **OM:** with `recursive=false`, a directory in an FSO bucket is deleted 
only if the other keys in the same request leave it empty.
     - Files are marked deleted first. Directories are then checked with 
`OMFileRequest#hasChildren`, deepest first. This way S3A `rm -r` and other 
list-then-delete clients still remove a whole tree that is sent in one request.
     - A directory that stays non-empty is kept, and its key is reported as 
`DIRECTORY_NOT_EMPTY`.
   - **S3 Gateway:** multi-delete sends `recursive=false` and reports 
`DIRECTORY_NOT_EMPTY` as deleted, because S3 has no directories. Single 
DeleteObject already behaves this way.
   - **Compatibility:** new `OzoneManagerVersion.NON_RECURSIVE_DELETE_KEYS`, a 
pre-ZDU version that is available before finalization.
     - The client sends the flag only when it is `false`, and rejects 
`recursive=false` if the OM is older.
     - With an older OM, the S3 Gateway keeps sending a recursive request.
   - **Client API:** new `OzoneBucket#deleteKeys(List, boolean, boolean)`. The 
existing `ClientProtocol#deleteKeys(..., quiet)` is kept as a default method.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17869
   
   ## How was this patch tested?
   
   - New unit tests:
     - `TestOMKeysDeleteRequestWithFSO`: non-empty directories are kept, and 
directories emptied by the same request are deleted.
     - `TestRpcClient`: the OM version check.
     - `TestOzoneManagerProtocolClientSideTranslatorPB`: the flag is sent only 
when it is `false`.
     - `TestObjectMultiDelete`: the S3 Gateway behavior.
   - New integration tests in `AbstractS3SDKV2Tests`:
     - `testDeleteObjectsKeepsNonEmptyFsoDirectory`: the repro from the Jira. 
It fails without the fix.
     - `testDeleteObjectsDeletesFsoDirectoriesWithTheirContents`: a listed FSO 
tree is deleted in one request.
   - `TestOMVersionManager` and `TestOMUpgradeFinalization` are updated for the 
new version. The existing ofs recursive bucket delete test 
`TestOFSWithFSO#testDeleteVolumeAndBucket` still passes.
   - CI: (link to the fork CI run)
   
   Generated-by: Claude Code (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to