Kenn, thank you! There is OnTimeBehavior (default FIRE_ALWAYS) and ClosingBehavior (default FIRE_IF_NON_EMPTY). Given that OnTimeBehavior is always-fire, shouldn't I see empty ON_TIME panes?
Since my lateness config is 0, I'm going to try ClosingBehavior = FIRE_ALWAYS and see if I can rely on .isLast() to pick out the last pane downstream. But curious if given that the OnTimeBehavior default is ALWAYS, shouldn't I be seeing on-time panes in my current config? On Mon, Jan 13, 2020 at 6:45 PM Kenneth Knowles <[email protected]> wrote: > On my phone, so I can't grab the jira so easily, but quickly: EARLY panes > are "race condition equivalent" to ON_TIME panes. The early panes consume > all the pending elements then the on time pane is "empty". This is WAI if > it is what is causing it. You need to explicitly set > Window.configure().fireAlways()*. I know this is counterintuitive in > accumulating mode, where the empty pane is not the identity element. > > Kenn > > *I don't recall if this is the default or not, and also because on phone > it is slow to look up. From your experience I think not default. > > On Mon, Jan 13, 2020, 15:03 Aaron Dixon <[email protected]> wrote: > >> Any confirmation on this from anyone? Whether per Beam spec, runners are >> obligated to send ON_TIME panes for AfterWatermark triggers? I'm stuck >> because this seems fundamental, so it's hard to imagine this is a Dataflow >> bug, but OTOH it's also hard to imagine that trigger specs like >> AfterWatermark are "optional"... ? >> >> On Mon, Jan 13, 2020 at 4:18 PM Aaron Dixon <[email protected]> wrote: >> >>> Yes. Using calendar day-based windows and watermark is completely caught >>> up to today ... calendar window ends several days ago. I got EARLY panes >>> for each element but never ON_TIME pane. >>> >>> On Mon, Jan 13, 2020 at 4:16 PM Luke Cwik <[email protected]> wrote: >>> >>>> Is the watermark advancing past the end of the window? >>>> >>>> On Mon, Jan 13, 2020 at 2:02 PM Aaron Dixon <[email protected]> wrote: >>>> >>>>> The window is not empty fwiw; it has elements; I get an early firing >>>>> pane for the window but well after the watermark passes there is no >>>>> ON_TIME >>>>> pane. Would this be a bug in Dataflow? Seems fundamental, so I'm concerned >>>>> perhaps the Beam spec doesn't obligate ON_TIME firings? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Jan 13, 2020 at 3:58 PM Luke Cwik <[email protected]> wrote: >>>>> >>>>>> I would have expected an empty on time pane since the default on time >>>>>> behavior is FIRE_ALWAYS. >>>>>> >>>>>> On Mon, Jan 13, 2020 at 1:54 PM Aaron Dixon <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Can anyone confirm? >>>>>>> >>>>>>> This is intermittent. Some (it seems, sparse) windows don't get an >>>>>>> ON_TIME firing after watermark. Is this a bug or is there a reason to >>>>>>> not >>>>>>> expect ON_TIME firings for every window? >>>>>>> >>>>>>> On Mon, Jan 13, 2020 at 3:47 PM Rui Wang <[email protected]> wrote: >>>>>>> >>>>>>>> If it indeed happened as you have described, I will be very >>>>>>>> interested in the expected behaviour. >>>>>>>> >>>>>>>> Something I remembered before: the trigger condition meets just >>>>>>>> gives the runner/engine "permission" to fire, but runner/engine may not >>>>>>>> fire immediately. But I don't know if the engine/runner will guarantee >>>>>>>> to >>>>>>>> fire. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -Rui >>>>>>>> >>>>>>>> On Mon, Jan 13, 2020 at 1:43 PM Aaron Dixon <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I have the following trigger: >>>>>>>>> >>>>>>>>> .apply(Window >>>>>>>>> .configure() >>>>>>>>> .triggering(AfterWatermark >>>>>>>>> .pastEndOfWindow() >>>>>>>>> .withEarlyFirings(AfterPane >>>>>>>>> .elementCountAtLeast(1))) >>>>>>>>> .accumulatingFiredPanes() >>>>>>>>> .withAllowedLateness(Duration.ZERO) >>>>>>>>> >>>>>>>>> But in Dataflow I notice that I never get an ON_TIME firing for my >>>>>>>>> window -- I only see early firing for elements, and then nothing. >>>>>>>>> >>>>>>>>> My assumption is that AfterWatermark should give me a last, >>>>>>>>> on-time pane under this configuration when the watermark surpasses the >>>>>>>>> window's end. >>>>>>>>> >>>>>>>>> Is my expectation correct? >>>>>>>>> >>>>>>>>
