[
https://issues.apache.org/jira/browse/HADOOP-19576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17955868#comment-17955868
]
ASF GitHub Bot commented on HADOOP-19576:
-----------------------------------------
steveloughran commented on PR #7722:
URL: https://github.com/apache/hadoop/pull/7722#issuecomment-2934912807
cn you point to some docs about s3express and MPUs?
we had lots of pain related to directories not existing but still being
found in list calls, and had to make changes across the code to cope with it.
Are these now superfluous?
> Insert Overwrite Jobs With MagicCommitter Fails On S3 Express Storage
> ---------------------------------------------------------------------
>
> Key: HADOOP-19576
> URL: https://issues.apache.org/jira/browse/HADOOP-19576
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Syed Shameerur Rahman
> Assignee: Syed Shameerur Rahman
> Priority: Major
> Labels: pull-request-available
>
> Query engines which uses Magic Committer to overwrite a directory would
> ideally upload the MPUs (not complete) and then delete the contents of the
> directory before committing the MPU.
>
> For S3 express storage, The directory purge operation is enabled by default.
> Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L688]
> for code pointers.
>
> Due to this, the pending MPU uploads are purged and query fails with
> {{NoSuchUpload: The specified multipart upload does not exist. The upload ID
> might be invalid, or the multipart upload might have been aborted or
> completed. }}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
