Re: Is it OK to have very many session windows?

2017-02-20 Thread Stephan Ewen
With pre-aggregation (which the Reduce does), Flink can handle many windows and many keys, as long as you have the memory and storage to support that. Your case should work. On Mon, Feb 20, 2017 at 4:58 PM, Vadim Vararu wrote: > It's something like: > >

Re: Is it OK to have very many session windows?

2017-02-20 Thread Vadim Vararu
It's something like: DataStreamSource stream = env.addSource(getKafkaConsumer(parameterTool)); stream .map(getEventToDomainMapper()) .keyBy(getKeySelector()) .window(ProcessingTimeSessionWindows.withGap(Time.minutes(90))) .reduce(getReducer())

Re: Is it OK to have very many session windows?

2017-02-20 Thread Timo Walther
Hi Vadim, this of course depends on your use case. The question is how large is your state per pane and how much memory is available for Flink? Are you using incremental aggregates such that only the aggregated value per pane has to be kept in memory? Regards, Timo Am 20/02/17 um 16:34

Is it OK to have very many session windows?

2017-02-20 Thread Vadim Vararu
HI guys, Is it okay to have very many (tens of thousands or hundreds of thousand) of session windows? Thanks, Vadim.