[
https://issues.apache.org/jira/browse/BEAM-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936966#comment-16936966
]
Maximilian Michels commented on BEAM-6733:
------------------------------------------
Yes, the checkpoint barrier alignment could be affected by this. The alignment
happens before {{prepareSnapshotBarrier}} is called, but any actions performed
during {{finishBundle}} in {{prepareSnapshotBarrier}} can affect the barrier
alignment at downstream operators.
Yes, latency for event time dependent processing (e.g. windowing) should not be
affected due to the watermark only being forwarded after a bundle ends.
> Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager
> ----------------------------------------------------------------------------
>
> Key: BEAM-6733
> URL: https://issues.apache.org/jira/browse/BEAM-6733
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Flink 1.6/1.7 provides a hook to execute an action before the snapshot
> barrier is emitted by the operator. At the moment (<=1.5) the Flink Runner
> has to buffer any elements which are emitted during a snapshot because the
> barrier has already been emitted. This leads to a lot of code complexity.
> We can remove the buffering in favor of finishing the current bundle in
> {{DoFnOperator}}'s {{prepareSnapshotPreBarrier}}. The 1.5/1.6/1.7 build setup
> poses a challenge to do that in a way that does not lead to much code
> duplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
