[
https://issues.apache.org/jira/browse/FLINK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730947#comment-14730947
]
ASF GitHub Bot commented on FLINK-2583:
---------------------------------------
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1084#issuecomment-137764994
I (almost) completely reworked the sink. It is now called `RollingSink` and
the module is called `flink-connector-filesystem` to show that it works with
any Hadoop FileSystem.
It is integrated with the checkpointing mechanism to provide exactly-once
semantics. When supported it will use `truncate` for this. Otherwise it will
write a special `.valid-length` file that specifies how many bytes in a file
are valid.
I added an ITCase that verifies the exactly-once behavior.
I added a lot of description about how the sink works in the Javadoc of
`RollingSink`, so if you want to check it out I suggest you start there.
> Add Stream Sink For Rolling HDFS Files
> --------------------------------------
>
> Key: FLINK-2583
> URL: https://issues.apache.org/jira/browse/FLINK-2583
> Project: Flink
> Issue Type: New Feature
> Components: Streaming
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Fix For: 0.10
>
>
> In addition to having configurable file-rolling behavior the Sink should also
> integrate with checkpointing to make it possible to have exactly-once
> semantics throughout the topology.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
