[ https://issues.apache.org/jira/browse/BEAM-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15962125#comment-15962125 ]
Jingsong Lee commented on BEAM-1723: ------------------------------------ I think it is necessary to be configurable because the deduplication window is related to the checkpoint interval. > FlinkRunner should deduplicate when an UnboundedSource requires Deduping > ------------------------------------------------------------------------ > > Key: BEAM-1723 > URL: https://issues.apache.org/jira/browse/BEAM-1723 > Project: Beam > Issue Type: Bug > Components: runner-flink > Reporter: Thomas Groh > Assignee: Jingsong Lee > > UnboundedSource implementations can require deduping, and the FlinkRunner > currently logs a warning that this is not supported. > https://github.com/apache/beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java#L139 -- This message was sent by Atlassian JIRA (v6.3.15#6346)