[ https://issues.apache.org/jira/browse/BEAM-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084001#comment-16084001 ]
ASF GitHub Bot commented on BEAM-2571: -------------------------------------- GitHub user aljoscha opened a pull request: https://github.com/apache/beam/pull/3550 [BEAM-2571] CombineTest.testSlidingWindowsCombineWithContext R: @kennknowles @JingsongLi You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/beam jira-2571-fix-flink-watermark-issues Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3550.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3550 ---- commit dee393b2a4f287509a0af5b44841216804cae3b2 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-07-12T12:35:58Z [BEAM-2571] Add check for output watermark in DoFnOperator on close() commit 39e47d973a243dfdf154b3be3e234e7bc598d4c5 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-07-12T12:39:58Z [BEAM-2571] Clarify pushedback variable name in DoFnOperator commit 31791beca9d58a24cf0d851d137a1c1526221c32 Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-07-12T12:42:37Z [BEAM-2571] Change DoFnOperator to use Long.MAX_VALUE as max watermark This is in line with what Flink does and what BoundesSourceWrapper and UnboundedSourceWrapper do. commit d0aee3d08e44de6587daa209592b766aced810bf Author: Aljoscha Krettek <aljoscha.kret...@gmail.com> Date: 2017-07-12T13:38:06Z [BEAM-2571] Respect watermark contract in Flink DoFnOperator In Flink, a watermark T specifies that there will be no elements with a timestamp <= T in the future. In Beam, a watermark T specifies that there will not be element with a timestamp < T in the future. This leads to problems when the watermark is exactly "on the timer timestamp", most prominently, this happened with Triggers, where Flink would fire the Trigger too early and the Trigger would determine (based on the watermark) that it is not yet time to fire the window while Flink thought it was time. This also adds a test that specifially tests the edge case. ---- > Flink ValidatesRunner failing CombineTest.testSlidingWindowsCombineWithContext > ------------------------------------------------------------------------------ > > Key: BEAM-2571 > URL: https://issues.apache.org/jira/browse/BEAM-2571 > Project: Beam > Issue Type: Bug > Components: runner-flink > Reporter: Kenneth Knowles > Assignee: Aljoscha Krettek > Fix For: 2.1.0 > > > This appears to have been caused by https://github.com/apache/beam/pull/3429 > which fixes a couple errors in how trigger timers were processed / final > panes labeled. > I am investigating, considering roll back vs forward fix. Since it is an > esoteric use case where I would advise users to use a stateful DoFn instead, > I think the bug fixed probably outweighs the bug introduced. I would like to > fix for 2.1.0 but will report back soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029)