Re: WindowOperator - element's timestamp

2016-11-15 Thread Aljoscha Krettek
Hi, I understand now. For early (speculative) firing I would suggest to write a custom trigger that repeatedly fires on processing time. We're also working on a Trigger DSL that will make such cases simpler, for example, you would be able to write: window.trigger(EventTime.pastEndOfWindow().withEa

Re: WindowOperator - element's timestamp

2016-11-14 Thread Petr Novotnik
Aljoscha, thanks for your response. The use-case I'm after is basically providing "early" (inaccurate) results to downstream consumers. Suppose we're running aggregations for daily time windows, but we don't want to wait a whole day to see results. The idea is to fire the windows continuously

Re: WindowOperator - element's timestamp

2016-11-14 Thread Aljoscha Krettek
Hi, I'm afraid the ContinuousEventTimeTrigger is a bit misleading and should probably be removed. The problem is that a watermark T signals that we won't see elements with a timestamp < T in the future. It does not signal that we haven't already seen elements with a timestamp > T. So this cannot be

Re: WindowOperator - element's timestamp

2016-11-14 Thread Ufuk Celebi
Looping in Kostas and Aljoscha who should know what's the expected behaviour here ;) On 11 November 2016 at 16:17:23, Petr Novotnik (petr.novot...@firma.seznam.cz) wrote: > Hello, > > I'm struggling to understand the following behaviour of the > `WindowOperator` and would appreciate some insig

WindowOperator - element's timestamp

2016-11-11 Thread Petr Novotnik
Hello, I'm struggling to understand the following behaviour of the `WindowOperator` and would appreciate some insight from experts: In particular I'm thinking about the following hypothetical data flow: input.keyBy(..) .window(TumblingEventTimeWindows.of(..)) .apply(..) ...