I think AfterWatermark in particular should *alway* produce an ON_TIME pane, regardless of whether there were early panes. (It's less clear with non-watermark triggers like after count or processing time.) This makes it feel like the on time behavior is a property of the trigger, not the windowing strategy.
On Mon, Jan 13, 2020 at 5:06 PM Aaron Dixon <atdi...@gmail.com> wrote: > > Kenn, thank you! There is OnTimeBehavior (default FIRE_ALWAYS) and > ClosingBehavior (default FIRE_IF_NON_EMPTY). Given that OnTimeBehavior is > always-fire, shouldn't I see empty ON_TIME panes? > > Since my lateness config is 0, I'm going to try ClosingBehavior = FIRE_ALWAYS > and see if I can rely on .isLast() to pick out the last pane downstream. But > curious if given that the OnTimeBehavior default is ALWAYS, shouldn't I be > seeing on-time panes in my current config? > > > > On Mon, Jan 13, 2020 at 6:45 PM Kenneth Knowles <k...@apache.org> wrote: >> >> On my phone, so I can't grab the jira so easily, but quickly: EARLY panes >> are "race condition equivalent" to ON_TIME panes. The early panes consume >> all the pending elements then the on time pane is "empty". This is WAI if it >> is what is causing it. You need to explicitly set >> Window.configure().fireAlways()*. I know this is counterintuitive in >> accumulating mode, where the empty pane is not the identity element. >> >> Kenn >> >> *I don't recall if this is the default or not, and also because on phone it >> is slow to look up. From your experience I think not default. >> >> On Mon, Jan 13, 2020, 15:03 Aaron Dixon <atdi...@gmail.com> wrote: >>> >>> Any confirmation on this from anyone? Whether per Beam spec, runners are >>> obligated to send ON_TIME panes for AfterWatermark triggers? I'm stuck >>> because this seems fundamental, so it's hard to imagine this is a Dataflow >>> bug, but OTOH it's also hard to imagine that trigger specs like >>> AfterWatermark are "optional"... ? >>> >>> On Mon, Jan 13, 2020 at 4:18 PM Aaron Dixon <atdi...@gmail.com> wrote: >>>> >>>> Yes. Using calendar day-based windows and watermark is completely caught >>>> up to today ... calendar window ends several days ago. I got EARLY panes >>>> for each element but never ON_TIME pane. >>>> >>>> On Mon, Jan 13, 2020 at 4:16 PM Luke Cwik <lc...@google.com> wrote: >>>>> >>>>> Is the watermark advancing past the end of the window? >>>>> >>>>> On Mon, Jan 13, 2020 at 2:02 PM Aaron Dixon <atdi...@gmail.com> wrote: >>>>>> >>>>>> The window is not empty fwiw; it has elements; I get an early firing >>>>>> pane for the window but well after the watermark passes there is no >>>>>> ON_TIME pane. Would this be a bug in Dataflow? Seems fundamental, so I'm >>>>>> concerned perhaps the Beam spec doesn't obligate ON_TIME firings? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jan 13, 2020 at 3:58 PM Luke Cwik <lc...@google.com> wrote: >>>>>>> >>>>>>> I would have expected an empty on time pane since the default on time >>>>>>> behavior is FIRE_ALWAYS. >>>>>>> >>>>>>> On Mon, Jan 13, 2020 at 1:54 PM Aaron Dixon <atdi...@gmail.com> wrote: >>>>>>>> >>>>>>>> Can anyone confirm? >>>>>>>> >>>>>>>> This is intermittent. Some (it seems, sparse) windows don't get an >>>>>>>> ON_TIME firing after watermark. Is this a bug or is there a reason to >>>>>>>> not expect ON_TIME firings for every window? >>>>>>>> >>>>>>>> On Mon, Jan 13, 2020 at 3:47 PM Rui Wang <ruw...@google.com> wrote: >>>>>>>>> >>>>>>>>> If it indeed happened as you have described, I will be very >>>>>>>>> interested in the expected behaviour. >>>>>>>>> >>>>>>>>> Something I remembered before: the trigger condition meets just gives >>>>>>>>> the runner/engine "permission" to fire, but runner/engine may not >>>>>>>>> fire immediately. But I don't know if the engine/runner will >>>>>>>>> guarantee to fire. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -Rui >>>>>>>>> >>>>>>>>> On Mon, Jan 13, 2020 at 1:43 PM Aaron Dixon <atdi...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> I have the following trigger: >>>>>>>>>> >>>>>>>>>> .apply(Window >>>>>>>>>> .configure() >>>>>>>>>> .triggering(AfterWatermark >>>>>>>>>> .pastEndOfWindow() >>>>>>>>>> .withEarlyFirings(AfterPane >>>>>>>>>> .elementCountAtLeast(1))) >>>>>>>>>> .accumulatingFiredPanes() >>>>>>>>>> .withAllowedLateness(Duration.ZERO) >>>>>>>>>> >>>>>>>>>> But in Dataflow I notice that I never get an ON_TIME firing for my >>>>>>>>>> window -- I only see early firing for elements, and then nothing. >>>>>>>>>> >>>>>>>>>> My assumption is that AfterWatermark should give me a last, on-time >>>>>>>>>> pane under this configuration when the watermark surpasses the >>>>>>>>>> window's end. >>>>>>>>>> >>>>>>>>>> Is my expectation correct?
