Re: Is it OK to have very many session windows?

2017-02-20 Thread Stephan Ewen 
With pre-aggregation (which the Reduce does), Flink can handle many windows
and many keys, as long as you have the memory and storage to support that.
Your case should work.


On Mon, Feb 20, 2017 at 4:58 PM, Vadim Vararu 
wrote:

> It's something like:
>
> DataStreamSource stream = 
> env.addSource(getKafkaConsumer(parameterTool));stream
> .map(getEventToDomainMapper())
> .keyBy(getKeySelector())
> .window(ProcessingTimeSessionWindows.withGap(Time.minutes(90)))
> .reduce(getReducer())
> .map(getToJsonMapper())
> .addSink(getKafkaProducer(parameterTool));
>
>
> Each new event may be reduced against the existent state from the window,
> so normally it's okay to have only 1 row in memory.
>
> I'm new to Flink and have not yet reached the "incremental aggregates"
> but, if i understand correctly, it fits my case.
>
> Vadim.
>
>
>
>
> On 20.02.2017 17:47, Timo Walther wrote:
>
> Hi Vadim,
>
> this of course depends on your use case. The question is how large is your
> state per pane and how much memory is available for Flink?
> Are you using incremental aggregates such that only the aggregated value
> per pane has to be kept in memory?
>
> Regards,
> Timo
>
>
> Am 20/02/17 um 16:34 schrieb Vadim Vararu:
>
> HI guys,
>
> Is it okay to have very many (tens of thousands or hundreds of thousand)
> of session windows?
>
>
> Thanks, Vadim.
>
>
>
>
>

Re: Is it OK to have very many session windows?

2017-02-20 Thread Vadim Vararu 

It's something like:

DataStreamSource stream = 
env.addSource(getKafkaConsumer(parameterTool)); stream
.map(getEventToDomainMapper())
.keyBy(getKeySelector())
.window(ProcessingTimeSessionWindows.withGap(Time.minutes(90)))
.reduce(getReducer())
.map(getToJsonMapper())
.addSink(getKafkaProducer(parameterTool));


Each new event may be reduced against the existent state from the 
window, so normally it's okay to have only 1 row in memory.


I'm new to Flink and have not yet reached the "incremental aggregates" 
but, if i understand correctly, it fits my case.


Vadim.



On 20.02.2017 17:47, Timo Walther wrote:

Hi Vadim,

this of course depends on your use case. The question is how large is 
your state per pane and how much memory is available for Flink?
Are you using incremental aggregates such that only the aggregated 
value per pane has to be kept in memory?


Regards,
Timo


Am 20/02/17 um 16:34 schrieb Vadim Vararu:

HI guys,

Is it okay to have very many (tens of thousands or hundreds of 
thousand) of session windows?



Thanks, Vadim.

Re: Is it OK to have very many session windows?

2017-02-20 Thread Timo Walther 

Hi Vadim,

this of course depends on your use case. The question is how large is 
your state per pane and how much memory is available for Flink?
Are you using incremental aggregates such that only the aggregated value 
per pane has to be kept in memory?


Regards,
Timo


Am 20/02/17 um 16:34 schrieb Vadim Vararu:

HI guys,

Is it okay to have very many (tens of thousands or hundreds of 
thousand) of session windows?



Thanks, Vadim.

Is it OK to have very many session windows?

2017-02-20 Thread Vadim Vararu 

HI guys,

Is it okay to have very many (tens of thousands or hundreds of thousand) 
of session windows?



Thanks, Vadim.