[ https://issues.apache.org/jira/browse/FLINK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730947#comment-14730947 ]
ASF GitHub Bot commented on FLINK-2583: --------------------------------------- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1084#issuecomment-137764994 I (almost) completely reworked the sink. It is now called `RollingSink` and the module is called `flink-connector-filesystem` to show that it works with any Hadoop FileSystem. It is integrated with the checkpointing mechanism to provide exactly-once semantics. When supported it will use `truncate` for this. Otherwise it will write a special `.valid-length` file that specifies how many bytes in a file are valid. I added an ITCase that verifies the exactly-once behavior. I added a lot of description about how the sink works in the Javadoc of `RollingSink`, so if you want to check it out I suggest you start there. > Add Stream Sink For Rolling HDFS Files > -------------------------------------- > > Key: FLINK-2583 > URL: https://issues.apache.org/jira/browse/FLINK-2583 > Project: Flink > Issue Type: New Feature > Components: Streaming > Reporter: Aljoscha Krettek > Assignee: Aljoscha Krettek > Fix For: 0.10 > > > In addition to having configurable file-rolling behavior the Sink should also > integrate with checkpointing to make it possible to have exactly-once > semantics throughout the topology. -- This message was sent by Atlassian JIRA (v6.3.4#6332)