chungen0126 opened a new pull request, #12106:
URL: https://github.com/apache/ozone/pull/12106
## What changes were proposed in this pull request?
On FSO buckets, S3 DeleteObjects with a directory key (`dir/`) deletes the
whole subtree, including objects that are not listed in the request. A single
DeleteObject on the same key is a no-op. Since HDDS-10572 (1.4.1), multi-delete
calls `deleteKeys`. Its FSO handler, `OmKeysDeleteRequestWithFSO`, deletes
directories recursively, as recursive bucket deletion through ofs needs. S3A
with `fs.s3a.directory.marker.retention=delete` sends such a request for the
parent markers after every write, which wipes sibling data. That retention mode
is the default before Hadoop 3.4 and is used by Flink's S3 plugin.
This PR adds an optional `recursive` flag to `DeleteKeysRequest`. It
defaults to `true`, so existing callers keep the recursive behavior.
- **OM:** with `recursive=false`, a directory in an FSO bucket is deleted
only if the other keys in the same request leave it empty.
- Files are marked deleted first. Directories are then checked with
`OMFileRequest#hasChildren`, deepest first. This way S3A `rm -r` and other
list-then-delete clients still remove a whole tree that is sent in one request.
- A directory that stays non-empty is kept, and its key is reported as
`DIRECTORY_NOT_EMPTY`.
- **S3 Gateway:** multi-delete sends `recursive=false` and reports
`DIRECTORY_NOT_EMPTY` as deleted, because S3 has no directories. Single
DeleteObject already behaves this way.
- **Compatibility:** new `OzoneManagerVersion.NON_RECURSIVE_DELETE_KEYS`, a
pre-ZDU version that is available before finalization.
- The client sends the flag only when it is `false`, and rejects
`recursive=false` if the OM is older.
- With an older OM, the S3 Gateway keeps sending a recursive request.
- **Client API:** new `OzoneBucket#deleteKeys(List, boolean, boolean)`. The
existing `ClientProtocol#deleteKeys(..., quiet)` is kept as a default method.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17869
## How was this patch tested?
- New unit tests:
- `TestOMKeysDeleteRequestWithFSO`: non-empty directories are kept, and
directories emptied by the same request are deleted.
- `TestRpcClient`: the OM version check.
- `TestOzoneManagerProtocolClientSideTranslatorPB`: the flag is sent only
when it is `false`.
- `TestObjectMultiDelete`: the S3 Gateway behavior.
- New integration tests in `AbstractS3SDKV2Tests`:
- `testDeleteObjectsKeepsNonEmptyFsoDirectory`: the repro from the Jira.
It fails without the fix.
- `testDeleteObjectsDeletesFsoDirectoriesWithTheirContents`: a listed FSO
tree is deleted in one request.
- `TestOMVersionManager` and `TestOMUpgradeFinalization` are updated for the
new version. The existing ofs recursive bucket delete test
`TestOFSWithFSO#testDeleteVolumeAndBucket` still passes.
- CI: (link to the fork CI run)
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]