[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818440#comment-15818440
]
Steve Loughran commented on HADOOP-13786:
-----------------------------------------
Multipart algorithm summarised by Thomas Demoor on HADOOP-13912
bq. Object stores do not have an efficient rename operation, which is used by
the Hadoop FileOutputCommitter to atomically promote the "winning" attempt out
of the multiple (speculative) attempts to the final path. These slow job
commits are one of the main friction points when using object stores in
Hadoop.There have been quite some attempts at resolving this: HADOOP-9565Link,
Apache Spark DirectOutputCommitters, ... but they have proven not to be robust
in face of adversity (network partitions, ...).
bq. The current ticket proposes to do the atomic commit by using the S3
Multipart API, which allows multiple concurrent uploads on the same objectname,
each in its own "temporary space, identified by the UploadId which is returned
as a response to InitiateMultipartUpload. Every attempt writes directly to the
final outputPath. Data is uploaded using Put Part and as a response an ETag for
the part is returned and stored. The CompleteMultipartUpload is postponed.
Instead, we persist the UploadId (using a _temporary subdir or elsewhere) and
the ETags. When a certain "job" wins CompleteMultipartUpload is called for each
of its files using the proper list of Part ETags.
bq. Completing a MultipartUpload is a metadata only operation (internally in
S3) and is thus orders of magnitude faster than the rename-based approach which
moves all the data.
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
