[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944534#comment-15944534
]
Sahil Takiar commented on HIVE-16295:
-------------------------------------
The patch is still a WIP, but since it using it will require significant
changes in Hive, I though I would file a JIRA early. A detailed explanation of
the work can be found here:
https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3a_committer.md
My high level understanding is that there will be two different
{{OutputCommitter}}s in this patch, and they both rely on the following idea:
{quote}
Our proposal for commiting work without rename is: delayed completion of
multi-part PUT operations
That is: tasks write all data as multipart uploads, but delay the final commit
action until until the final, single job commit action. Only that data
committed in the job commit action will be made visible; work from speculative
and failed tasks will not be instiantiated. As there is no rename, there is no
delay while data is copied from a temporary directory to the final directory.
The duration of the commit will be the time needed to determine which commit
operations to construct, and to execute them.
The workers upload the data —but the job committer finalizes all uploads
This is the key point of the algorithm. The data is uploaded, awaiting
instantiation, but it doesn't appear in the object store until the final job
commit operation completes the outstanding multipart uploads. At this point the
new files become visible, which each objects instantiation being atomic.
{quote}
One of the committers is based on a committer from Netflix:
https://github.com/rdblue/s3committer
> Add support for using Hadoop's OutputCommitter
> ----------------------------------------------
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
