[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016448#comment-16016448
]
Steve Loughran commented on HADOOP-13786:
-----------------------------------------
Patch 029
# name of MultiplePendingUploads structure is #PendingSet# in classname and docs
# MR job IT now does 20 files on a -Dscale test run, for a little bit more
scaleup
# renaming "delayed commit" -> "pending commit" in code", still "magic commit"
in docs.
# Rename TestS3AHugePendingCommits so scale tests run in the serial part of a
test run
# metrics: track job+ task commit/fail, propagate to tests via the _SUCCESS
data where they are now logged
# review/update the docs, going from design to actual. Maybe I should get a PDF
of the previous version & attach it here for readable history
# address some yetus complaints, mostly about non-encapsulated fields in
datastructures used in tests, datastructures there because java doesn't do
tuples. I've done some and am going to stop at the rest, as checkstyle is
behind the times.
As well as reviews, having people play with this will be nice.
I'm going to focus on the docs, make sure the issues are covered and structure
right, then work on the spark tests in
https://github.com/hortonworks-spark/cloud-integration . the _SUCCESS data
structure can be read there even without requiring to be built against a
compatible Hadoop version (it's just JSON to read...) and so tests are already
verifying which committer is being used. I plan to do more there, and then
create some tests designed to go through the bits of Spark which use the more
complex extensions of the protocol (`CREATE TABLE` for example).
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
> ------------------------------------------------------------------------
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Affects Versions: HADOOP-13345
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch, HADOOP-13786-HADOOP-13345-005.patch,
> HADOOP-13786-HADOOP-13345-006.patch, HADOOP-13786-HADOOP-13345-006.patch,
> HADOOP-13786-HADOOP-13345-007.patch, HADOOP-13786-HADOOP-13345-009.patch,
> HADOOP-13786-HADOOP-13345-010.patch, HADOOP-13786-HADOOP-13345-011.patch,
> HADOOP-13786-HADOOP-13345-012.patch, HADOOP-13786-HADOOP-13345-013.patch,
> HADOOP-13786-HADOOP-13345-015.patch, HADOOP-13786-HADOOP-13345-016.patch,
> HADOOP-13786-HADOOP-13345-017.patch, HADOOP-13786-HADOOP-13345-018.patch,
> HADOOP-13786-HADOOP-13345-019.patch, HADOOP-13786-HADOOP-13345-020.patch,
> HADOOP-13786-HADOOP-13345-021.patch, HADOOP-13786-HADOOP-13345-022.patch,
> HADOOP-13786-HADOOP-13345-023.patch, HADOOP-13786-HADOOP-13345-024.patch,
> HADOOP-13786-HADOOP-13345-025.patch, HADOOP-13786-HADOOP-13345-026.patch,
> HADOOP-13786-HADOOP-13345-027.patch, HADOOP-13786-HADOOP-13345-028.patch,
> HADOOP-13786-HADOOP-13345-028.patch, HADOOP-13786-HADOOP-13345-029.patch,
> objectstore.pdf, s3committer-master.zip
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
