[
https://issues.apache.org/jira/browse/PIG-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-4086:
----------------------------
Attachment: PIG-4086-3.patch
Added "getContext().canCommit()" as discussed in TEZ-1351.
> Fix Orc e2e tests for tez
> -------------------------
>
> Key: PIG-4086
> URL: https://issues.apache.org/jira/browse/PIG-4086
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: PIG-4086-1.patch, PIG-4086-2.patch, PIG-4086-3.patch
>
>
> All Orc e2e tests fail on tez.
> There are two issue:
> 1. hivelibdir etc is not set in tez.conf
> 2. OrcStorage produce empty output file
> Digging into #2, the problem is in this code in PigProcessor:
> {code}
> if (fileOutput.isCommitRequired()) {
> fileOutput.commit();
> }
> {code}
> fileOutput.commit() invokes both RecordWriter.close() and
> committer.commitTask(). However, OrcNewOutputFormate will generate output
> file only after RecordWriter.close (if the output file is small),
> fileOutput.isCommitRequired will not detect this file, thus skip
> fileOutput.commit().
> Changing the code to invoke fileOutput.close explicitly fix the issue.
> fileOutput.commit will invoke close again, but there is no side effect since
> close will check if it has been already called.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
