Re: Behaviour of Process Window Function

2018-09-10 Thread Harshvardhan Agrawal
Hi, Our application is financial data enrichment. What we want to do is that we want to first key the positions by Account Number and then window them. Within a window I want to get all the unique products across all the accounts and make an external service call to hydrate the cache for that wind

Re: Behaviour of Process Window Function

2018-09-07 Thread Hequn Cheng
Hi Harshvardhan, *> 1) Does the state in the process window function qualify as KeyedState or OperatorState? * KeyedState *> We want to be able to rehydrate the guava cache at the beginning of each window by making an external rest call and clear the cache at the end of that respective window. Ho

Re: Behaviour of Process Window Function

2018-09-06 Thread vino yang
Hi Harshvardhan, 1) Yes, ProcessWindowFunction extends AbstractRichFunction, through getRuntimeContext,you can access keyed state API. 2) ProcessWindowFunction has given you considerable flexibility, you can based on processing time / event time / timer / it's clear method / customized implementat

Behaviour of Process Window Function

2018-09-06 Thread Harshvardhan Agrawal
Hello, We have a Flink pipeline where we are windowing our data after a keyBy. i.e. myStream.keyBy().window().process(MyIncrementalAggregation(), MyProcessFunction()). I have two questions about the above line of code: 1) Does the state in the process window function qualify as KeyedState or Oper