[
https://issues.apache.org/jira/browse/FLINK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727484#comment-14727484
]
ASF GitHub Bot commented on FLINK-2583:
---------------------------------------
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1084#issuecomment-137125224
If it fails in the middle of writing or before sync/flush is called on the
writer then the data can be in an inconsistent state. I see three ways of
dealing with this, one is more long-term.
The long term solution is to make the sink exactly-once aware. Either using
truncate() support in Hadoop 2.7 or a custom Thread that does merging of part
files and throwing away of data that was erroneously written.
The two short term options are:
- Keep it as it is, consumers need to be able to deal with corrupt records
and ignore them. This would give you at-least-once semantics.
- Write to a temporary file. When rolling, close the current bucket and
rename the file to the final filename. This would ensure that the output
doesn't contain corrupt records but you would have neither at-least-once nor
exactly-once semantics because some written records would be lost if checkpoint
restore restores to a state after the writing of the current bucket file
started.
> Add Stream Sink For Rolling HDFS Files
> --------------------------------------
>
> Key: FLINK-2583
> URL: https://issues.apache.org/jira/browse/FLINK-2583
> Project: Flink
> Issue Type: New Feature
> Components: Streaming
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Fix For: 0.10
>
>
> In addition to having configurable file-rolling behavior the Sink should also
> integrate with checkpointing to make it possible to have exactly-once
> semantics throughout the topology.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
