[
https://issues.apache.org/jira/browse/FLINK-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194435#comment-14194435
]
ASF GitHub Bot commented on FLINK-1139:
---------------------------------------
GitHub user fhueske opened a pull request:
https://github.com/apache/incubator-flink/pull/173
[FLINK-1139] Fix for HadoopOF with DOP > 1
This PR allows to execute HadoopOFs with DOP > 1.
It adds a hook to JM that is executed after the last OutputFormat finished.
Hadoop FileOutputFormats need this hook to clean-up temporary files.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fhueske/incubator-flink hadoopOF
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/173.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #173
----
commit 8f7ba0869d678d1f2f5340fa52b337a0ce8b7d34
Author: Fabian Hueske <[email protected]>
Date: 2014-10-06T14:27:25Z
[FLINK-1139] Added FinalizeOnMaster hook to run code after the last task of
an OutputFormat completed
commit 99a1c9ca204c1c9183a537e46c2fa01ad3632e4f
Author: Fabian Hueske <[email protected]>
Date: 2014-10-06T14:28:00Z
[FLINK-1139] Fixed HadoopOutputFormat to run with DOP > 1
----
> HadoopOutputFormat fails for DOP > 1
> ------------------------------------
>
> Key: FLINK-1139
> URL: https://issues.apache.org/jira/browse/FLINK-1139
> Project: Flink
> Issue Type: Bug
> Components: Hadoop Compatibility
> Reporter: Fabian Hueske
> Assignee: Fabian Hueske
>
> HadoopOutputFormat fails for DOP > 1. The reason is that job finalization
> (removal of the temp directory) is done after the first task finishes.
> All other tasks will fail in that moment, since they cannot continue to write
> their data to the temp directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
