[
https://issues.apache.org/jira/browse/HADOOP-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17983676#comment-17983676
]
Steve Loughran commented on HADOOP-19589:
-----------------------------------------
* we can bypass xfer manager and just issue rename commands to the thread pool,
with some throttling
* source etag should be passed in
* auditing to attach referrer header
I see the comment about idempotency and want to know what it means if the
request works, but we get a network error on the response, so don't see the 200
s3a will have to retry, and the source will have been copied. What then?
If the dest etag == source then s3 should be able to consider this a no-op.
does it?
otherwise we repeat, catch whatever error comes back from missing source, probe
dest for having etag and the interpret as success.
* Lots of tests of failure and race conditions, with the fault injection stuff
we've added for other tests.
note also the manifest committer could then run on s3,
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem
no tangible benefit over the magic committer, just an interesting fact. Not
worth the effort to support.
Be more relevant to hive commits, FWIW.
> Support atomic rename() for S3 express
> --------------------------------------
>
> Key: HADOOP-19589
> URL: https://issues.apache.org/jira/browse/HADOOP-19589
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Ahmar Suhail
> Priority: Major
>
> S3 Express now supports atomic renames:
> [https://aws.amazon.com/about-aws/whats-new/2025/06/amazon-s3-express-one-zone-atomic-renaming-objects-api/]
>
> Rename API is available as of SDK version 2.31.66.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
