Kenn, can you quote some use cases for this, to make it more clear what are the consequences of having this API in this form?
I recall that one of the main use cases was batching DoFn, right? On Tue, Mar 28, 2017 at 1:37 PM Kenneth Knowles <[email protected]> wrote: > On Tue, Mar 28, 2017 at 1:32 PM, Robert Bradshaw < > [email protected]> wrote: > > > Another alternative is to be able to set special timers, e.g. end of > window > > and expiration of window. That at least addresses (2). > > > > Potentially a tangent, but that would perhaps fit in with the idea of > removing TimeDomain from user APIs ( > https://issues.apache.org/jira/browse/BEAM-1308) and instead having > TimerSpecs.eventTimeTimer(), TimerSpecs.processingTimeTimer(), > TimerSpecs.windowExpirationTimer() that each yield distinct sorts of > parameters in @ProcessElement. > > A bit more heavyweight, syntactically. > > Kenn > > > > > > On Tue, Mar 28, 2017 at 1:27 PM, Kenneth Knowles <[email protected] > > > > wrote: > > > > > Hi all, > > > > > > I have a little extension to the stateful DoFn annotations to circulate > > for > > > feedback: Allow a method to be annotated with @OnWindowExpiration to > > > automatically get a callback at some point after the window has > expired, > > > but before the state for the window has been cleared. > > > > > > Today, a user can pretty easily get the same effect by setting a timer > > for > > > the end of the window + allowed lateness in their @ProcessElement > calls. > > > But having just one annotation for it has a couple nice benefits: > > > > > > 1. Some users assume a naive implementation so they are concerned that > > > setting a timer repeatedly is costly. This eliminates the cause for > user > > > alarm and allows a runner to do a better job in case it didn't already > do > > > it efficiently. > > > > > > 2. Getting the allowed lateness to be available to your @ProcessElement > > is > > > a little crufty. > > > > > > 3. Often, if you don't have @OnWindowExpiration, you are leaving behind > > > state that might contain data that is otherwise lost. So I would even > > > consider making it mandatory (with some way of indicating state you > don't > > > care about dropping) though that could be annoying. > > > > > > Another interesting moment in a window's lifecycle is @EndOfWindow. > This > > is > > > not critical for correctness, though. > > > > > > Thoughts? > > > > > > Kenn > > > > > >
