Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1084#issuecomment-137764994
I (almost) completely reworked the sink. It is now called `RollingSink` and
the module is called `flink-connector-filesystem` to show that it works with
any Hadoop FileSystem.
It is integrated with the checkpointing mechanism to provide exactly-once
semantics. When supported it will use `truncate` for this. Otherwise it will
write a special `.valid-length` file that specifies how many bytes in a file
are valid.
I added an ITCase that verifies the exactly-once behavior.
I added a lot of description about how the sink works in the Javadoc of
`RollingSink`, so if you want to check it out I suggest you start there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
