[
https://issues.apache.org/jira/browse/HIVE-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665533#comment-15665533
]
Sahil Takiar commented on HIVE-14271:
-------------------------------------
[~spena] looked more into what we discussed this morning, you are correct,
there are two places where the {{FileSinkOperator}} is renaming files. The
first happens in the {{commit(FileSystem)}} method, the method is invoked
inside each map task. The second happens in the {{jobCloseOp(boolean)}} method,
the method is invoked inside HiveServer2.
I think we can break this work down into two JIRAs:
1: Eliminate the rename that occurs in HiveServer2
2: Eliminate the rename that occurs inside each map task
When running on S3, I can't think of a reason why either would be necessary. I
think the first priority will be to eliminate the rename that occurs in
HiveServer2 (as you said this morning).
> FileSinkOperator should not rename files to final paths when S3 is the
> default destination
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-14271
> URL: https://issues.apache.org/jira/browse/HIVE-14271
> Project: Hive
> Issue Type: Sub-task
> Reporter: Sergio Peña
> Assignee: Sergio Peña
>
> FileSinkOperator does a rename of {{outPaths -> finalPaths}} when it finished
> writing all rows to a temporary path. The problem is that S3 does not support
> renaming.
> Two options can be considered:
> a. Use a copy operation instead. After FileSinkOperator writes all rows to
> outPaths, then the commit method will do a copy() call instead of move().
> b. Write row by row directly to the S3 path (see HIVE-1620). This may add
> better performance calls, but we should take care of the cleanup part in case
> of writing errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
