I'm not sure if it's currently legal. However the watermark is generally defined to be monotonic, so if it was allowed it would result in late data in the pipeline.
On Tue, Nov 5, 2019 at 10:29 AM Aaron Dixon <atdi...@gmail.com> wrote: > Thanks Reuven, > > So is my conclusion correct? That it is illegal for any custom window > function (+ combiner policy) to merge in a way that would regress the > watermark? > > What do Runners (eg Dataflow) do if this occurs? > > Does the API obligate runners to fail, or can insanity ensue? :) > > On Tue, Nov 5, 2019 at 10:55 AM Reuven Lax <re...@google.com> wrote: > >> >> >> On Tue, Nov 5, 2019 at 8:07 AM Aaron Dixon <atdi...@gmail.com> wrote: >> >>> I noticed that if I use TimestampCombiner/EARLIEST for session windows >>> that the watermark appears to get held up for sessions that never "close" >>> (or that extend for a long time). >>> >> >> Correct - because the watermark is then being held up by the earliest >> timestamp in any extant session window. >> >> >>> But if I use default (TimestampCombiner/END_OF_WINDOW) the watermark >>> doesn't get held. >>> >> >> Yes - because then the watermark gets held up by the current end of >> window. >> >> >>> >>> Does this mean that the watermark is adjusted whenever windows are >>> merged, even before they "close"? >>> >> >> In the second case, yes. Every time windows merge, the end of window for >> that key is recalculated. The actual watermark will be the minimum of all >> these end-of-windows (as each window is per key) >> >> >>> If that is the case, and I write a custom WindowFn, is this implication >>> of this that I should never move the `end`/`maxTimestamp` of the new/merged >>> window *backwards* in time? >>> >>> >>> >>>
