[
https://issues.apache.org/jira/browse/BEAM-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065429#comment-16065429
]
Eugene Kirpichov edited comment on BEAM-2140 at 6/27/17 8:38 PM:
-----------------------------------------------------------------
Okay, I see that I misunderstood what the watermark hold does, and now I'm not
sure how anything works at all (i.e. why timers set by SDF are not constantly
dropped) - in direct and dataflow runner :-|
For SDF specifically, I think it would make sense to *advance the watermark of
the input same as if the DoFn was not splittable* - i.e. consider the input
element "consumed" only when the ProcessElement call terminates with no
residual restriction. In other words, I guess, set an "input watermark hold"
(in addition to output watermark hold)? Is such a thing possible? Does it make
equal sense for non-splittable DoFn's that use timers?
was (Author: jkff):
Okay, I see that I misunderstood what the watermark hold does, and now I'm not
sure how anything works at all (i.e. why timers set by SDF are not constantly
dropped) - in direct and dataflow runner :-|
For SDF specifically, I think it would make sense to **advance the watermark of
the input same as if the DoFn was not splittable** - i.e. consider the input
element "consumed" only when the ProcessElement call terminates with no
residual restriction. In other words, I guess, set an "input watermark hold"
(in addition to output watermark hold)? Is such a thing possible? Does it make
equal sense for non-splittable DoFn's that use timers?
> Fix SplittableDoFn ValidatesRunner tests in FlinkRunner
> -------------------------------------------------------
>
> Key: BEAM-2140
> URL: https://issues.apache.org/jira/browse/BEAM-2140
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
>
> As discovered as part of BEAM-1763, there is a failing SDF test. We disabled
> the tests to unblock the open PR for BEAM-1763.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
