[ https://issues.apache.org/jira/browse/FLINK-3515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aljoscha Krettek closed FLINK-3515. ----------------------------------- Resolution: Duplicate > Make the "file monitoring source" exactly-once > ---------------------------------------------- > > Key: FLINK-3515 > URL: https://issues.apache.org/jira/browse/FLINK-3515 > Project: Flink > Issue Type: Improvement > Components: Streaming > Affects Versions: 0.10.2 > Reporter: Stephan Ewen > > The stream source that watches directories for changes is currently not > "exactly-once". > To make it exactly once, the source (that generates files to be read) and the > flatMap (that reads the files) need to keep track of where they were at the > point of a checkpoint. > Assuming that files do not change after creation (HDFS / S3 style), we can > make this the following way: > - The source can track the files it already emitted downstream via file > creation/modification timestamp, assuming that new files always get newer > timestamps. > - The flatMappers need to always store the path of their current file > fragment, plus the byte offset where they were within that file split. -- This message was sent by Atlassian JIRA (v6.3.4#6332)