[
https://issues.apache.org/jira/browse/HADOOP-19801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072593#comment-18072593
]
ASF GitHub Bot commented on HADOOP-19801:
-----------------------------------------
EnricoMi commented on PR #8306:
URL: https://github.com/apache/hadoop/pull/8306#issuecomment-4223930708
Adding this to `hadoop-tools/hadoop-aws/src/test/resources/log4j.properties`
```
log4j.logger.software.amazon.awssdk.thirdparty.org.apache.http.headers=DEBUG
```
shows request headers:
```
2026-04-10 10:43:27,384 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(133)) -
http-outgoing-0 >> GET
/vast-bucket?list-type=2&delimiter=%2F&max-keys=2&prefix=test%2F HTTP/1.1
2026-04-10 10:43:27,384 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> Host: vast.company.org
2026-04-10 10:43:27,384 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> amz-sdk-request: attempt=1; max=2
2026-04-10 10:43:27,385 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> User-Agent: Hadoop 3.6.0-SNAPSHOT aws-sdk-java/2.35.4
md/io#sync md/http#Apache ua/2.1 api/S3#2.35.x os/Linux#5.15.0-91-generic
lang/java#17.0.18 md/OpenJDK_64-Bit_Server_VM#17.0.18+8-Ubuntu-122.04.1
md/vendor#Ubuntu md/en_GB m/F,N,N,G hll/cross-region
2026-04-10 10:43:27,385 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
2026-04-10 10:43:27,385 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> X-Amz-Date: 20260410T094327Z
2026-04-10 10:43:27,385 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> x-amz-delete-contents: true
2026-04-10 10:43:27,385 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onRequestSubmitted(136)) -
http-outgoing-0 >> Connection: Keep-Alive
2026-04-10 10:43:28,389 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(122)) -
http-outgoing-0 << HTTP/1.1 400 Bad Request
2026-04-10 10:43:28,390 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(125)) -
http-outgoing-0 << x-amz-bucket-region: vast
2026-04-10 10:43:28,390 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(125)) -
http-outgoing-0 << Content-Type: application/xml
2026-04-10 10:43:28,390 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(125)) -
http-outgoing-0 << Transfer-Encoding: chunked
2026-04-10 10:43:28,391 [setup] DEBUG http.headers
(LoggingManagedHttpClientConnection.java:onResponseReceived(125)) -
http-outgoing-0 << Server: vast 5.4.3.1
2026-04-10 10:43:29,402 [setup] DEBUG s3a.Invoker
(Invoker.java:retryUntranslated(474)) - List vast-bucket:/test/ delimiter=/
keys=2 requester pays=null ;
software.amazon.awssdk.services.s3.model.S3Exception: Invalid Argument
(Service: S3, Status Code: 400, Request ID: 30d1310001aac2, Extended Request
ID: 30d1310001aac2) (SDK Attempt Count: 1),
2026-04-10 10:43:29,402 [setup] DEBUG s3a.S3ARetryPolicy
(S3ARetryPolicy.java:shouldRetry(318)) - Retry probe for AWSBadRequestException
with 0 retries and 0 failovers, idempotent=true, due to
org.apache.hadoop.fs.s3a.AWSBadRequestException: List
playpen-enricmin222:/test/ delimiter=/ keys=2 requester pays=null on /:
software.amazon.awssdk.services.s3.model.S3Exception: Invalid Argument
(Service: S3, Status Code: 400, Request ID: 30d1310001aac2, Extended Request
ID: 30d1310001aac2) (SDK Attempt Count: 1):InvalidArgument: Invalid Argument
(Service: S3, Status Code: 400, Request ID: 30d1310001aac2, Extended Request
ID: 30d1310001aac2) (SDK Attempt Count: 1)
org.apache.hadoop.fs.s3a.AWSBadRequestException: List vast-bucket:/test/
delimiter=/ keys=2 requester pays=null on /:
software.amazon.awssdk.services.s3.model.S3Exception: Invalid Argument
(Service: S3, Status Code: 400, Request ID: 30d1310001aac2, Extended Request
ID: 30d1310001aac2) (SDK Attempt Count: 1):InvalidArgument: Invalid Argument
(Service: S3, Status Code: 400, Request ID: 30d1310001aac2, Extended Request
ID: 30d1310001aac2) (SDK Attempt Count: 1)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:271)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:481)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:3106)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4096)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3987)
...
```
> Allow to skip recursive file deletion for non-empty directory
> -------------------------------------------------------------
>
> Key: HADOOP-19801
> URL: https://issues.apache.org/jira/browse/HADOOP-19801
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Reporter: Enrico Minack
> Priority: Major
> Labels: pull-request-available
>
> A non-empty directory, as well as a directory where its emptiness is unknown,
> is deleted by listing all contained files (or objects with matching key
> prefix), then deleting those files (objects) in bulk delete requests, and
> finally deleting the (then) empty directory itself.
> The recursive deletion of objects is not needed for S3 endpoints that support
> deleting non-empty directories (or path prefixes).
> For example: The [VAST S3 endpoint supports deleting non-empty
> directories|https://kb.vastdata.com/documentation/docs/using-trash-folder-for-s3-objects-6]
> with a single request. Such a feature can only be fully leveraged if
> {{hadoop-aws}} coupd skip the recursive deletion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
