Any confirmation on this from anyone? Whether per Beam spec, runners are obligated to send ON_TIME panes for AfterWatermark triggers? I'm stuck because this seems fundamental, so it's hard to imagine this is a Dataflow bug, but OTOH it's also hard to imagine that trigger specs like AfterWatermark are "optional"... ?
On Mon, Jan 13, 2020 at 4:18 PM Aaron Dixon <atdi...@gmail.com> wrote: > Yes. Using calendar day-based windows and watermark is completely caught > up to today ... calendar window ends several days ago. I got EARLY panes > for each element but never ON_TIME pane. > > On Mon, Jan 13, 2020 at 4:16 PM Luke Cwik <lc...@google.com> wrote: > >> Is the watermark advancing past the end of the window? >> >> On Mon, Jan 13, 2020 at 2:02 PM Aaron Dixon <atdi...@gmail.com> wrote: >> >>> The window is not empty fwiw; it has elements; I get an early firing >>> pane for the window but well after the watermark passes there is no ON_TIME >>> pane. Would this be a bug in Dataflow? Seems fundamental, so I'm concerned >>> perhaps the Beam spec doesn't obligate ON_TIME firings? >>> >>> >>> >>> >>> >>> On Mon, Jan 13, 2020 at 3:58 PM Luke Cwik <lc...@google.com> wrote: >>> >>>> I would have expected an empty on time pane since the default on time >>>> behavior is FIRE_ALWAYS. >>>> >>>> On Mon, Jan 13, 2020 at 1:54 PM Aaron Dixon <atdi...@gmail.com> wrote: >>>> >>>>> Can anyone confirm? >>>>> >>>>> This is intermittent. Some (it seems, sparse) windows don't get an >>>>> ON_TIME firing after watermark. Is this a bug or is there a reason to not >>>>> expect ON_TIME firings for every window? >>>>> >>>>> On Mon, Jan 13, 2020 at 3:47 PM Rui Wang <ruw...@google.com> wrote: >>>>> >>>>>> If it indeed happened as you have described, I will be very >>>>>> interested in the expected behaviour. >>>>>> >>>>>> Something I remembered before: the trigger condition meets just gives >>>>>> the runner/engine "permission" to fire, but runner/engine may not fire >>>>>> immediately. But I don't know if the engine/runner will guarantee to >>>>>> fire. >>>>>> >>>>>> >>>>>> >>>>>> -Rui >>>>>> >>>>>> On Mon, Jan 13, 2020 at 1:43 PM Aaron Dixon <atdi...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I have the following trigger: >>>>>>> >>>>>>> .apply(Window >>>>>>> .configure() >>>>>>> .triggering(AfterWatermark >>>>>>> .pastEndOfWindow() >>>>>>> .withEarlyFirings(AfterPane >>>>>>> .elementCountAtLeast(1))) >>>>>>> .accumulatingFiredPanes() >>>>>>> .withAllowedLateness(Duration.ZERO) >>>>>>> >>>>>>> But in Dataflow I notice that I never get an ON_TIME firing for my >>>>>>> window -- I only see early firing for elements, and then nothing. >>>>>>> >>>>>>> My assumption is that AfterWatermark should give me a last, on-time >>>>>>> pane under this configuration when the watermark surpasses the window's >>>>>>> end. >>>>>>> >>>>>>> Is my expectation correct? >>>>>>> >>>>>>
