alamb commented on code in PR #734:
URL:
https://github.com/apache/arrow-rs-object-store/pull/734#discussion_r3318512718
##########
src/aws/client.rs:
##########
@@ -1153,6 +1169,55 @@ mod tests {
mock.shutdown().await;
}
+ #[tokio::test]
+ async fn test_single_object_delete_request() {
+ let mock = MockServer::new().await;
+ mock.push_fn(|req| {
+ // A single-object delete must use `DELETE /key`, not the bulk
+ // `POST /?delete` (DeleteObjects) API.
+ assert_eq!(req.method(), Method::DELETE);
+ assert_eq!(req.uri().path(), "/test-bucket/test");
+ assert!(req.uri().query().is_none());
+ Response::builder().status(204).body(String::new()).unwrap()
+ });
+
+ let config = default_headers_config(&mock);
+ let client = S3Client::new(config,
HttpClient::new(reqwest::Client::new()));
+ let result = client.delete_request(&Path::from("test")).await;
+
+ assert!(result.is_ok());
+ mock.shutdown().await;
+ }
+
+ #[tokio::test]
+ async fn test_disable_bulk_delete_uses_single_object_delete() {
Review Comment:
is there an existing test that shows that bulk_delete actually uses the POST
/delete API? I didn't see one immediately
If not I think we should add one to document in test the different behaviors
##########
src/aws/mod.rs:
##########
@@ -264,6 +264,25 @@ impl ObjectStore for AmazonS3 {
locations: BoxStream<'static, Result<Path>>,
) -> BoxStream<'static, Result<Path>> {
let client = Arc::clone(&self.client);
+
+ // Some S3-compatible providers do not implement
+ // the bulk `DeleteObjects` API (`POST /?delete`). When bulk delete is
+ // disabled, fall back to parallel single-object `DELETE /key`
requests,
+ // which are part of the core S3 API supported by every provider.
+ if client.config.disable_bulk_delete {
+ return locations
+ .map(move |location| {
+ let client = Arc::clone(&client);
+ async move {
+ let location = location?;
+ client.delete_request(&location).await?;
+ Ok(location)
+ }
+ })
+ .buffered(20)
Review Comment:
I wondered how you picked `20` as the concurrency but it seems to mirror the
`buffered(20)` below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
