Hi, Aljoscha Thanks for your explanation.
About the Storm windows simulation, we had tried your suggestion and gave up due to its complexity and sort of "reinventing the wheel". Without considering the performance, most of our business-logic code have already been transformed to the "Flink style". I am glad to hear that adding the accumulator is just in progress. As far as I can see, the operations it supplies will adequately meet the demands. I will stay focus on this topic. Best, Xingcan On Wed, Jan 11, 2017 at 7:28 PM, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > (I'm just getting back from holidays, therefore the slow response. Sorry > for that.) > > I think you can simulate the way Storm windows work by using a > GlobalWindows assigner and having a custom Trigger and/or Evictor and also > some special logic in your WindowFunction. > > About mergeable state, we're actually in the process of adding something > like this that would be a generalisation of reduce and fold: you can call > it combine or aggregate. The idea is to have these operations: > > - create accumulator > - add value to accumulator > - merge accumulators > - extract output from accumulator > > You have three types: IN for incoming values, ACC for accumulators and OUT > as the result of extracting output from an accumulator. This should cover > most cases. > > What do you think? > > Cheers, > Aljoscha > > On Thu, 22 Dec 2016 at 07:13 xingcan <xingc...@gmail.com> wrote: > > Hi Aljoscha, > > First of all, sorry for that I missed your prompt reply : ( > > In these days, I've been learning the implementation mechanism of window > in Flink. > > I think the main difference between the window in Storm and Flink (from > the API level) is that, Storm maintains only one window while Flink > maintains several isolated windows. Due to that, Storm users can be aware > of the transformation (tuple add and expire) of a window and take actions > on each window modification (sliding window forwarding) while Flink users > can only implement functions on one and another complete window as if they > are independent of each other (actually they may get quite a few tuples in > common). > > Objectively speaking, the window API provided by Flink is more formalize > and easy to use. However, for sliding window with high-capacity and short > interval (e.g. 5m and 1s), each tuple will be calculated redundantly (maybe > 300 times in the example?). Though it provide the pane optimization, I > think it's far from enough as the optimization can only be applied on > reduce functions which restrict the input and output data type to be the > same. Some other functions, e.g., the MaxAndMin function which take numbers > as input and output a max&min pair and the Average function, which should > avoid redundant calculations can not be satisfied. > > Actually, I just wondering if a "mergeable fold function" could be added > (just *like* this https://en.wikipedia.org/wiki/Mergeable_heap). I know > it may violate some principles of Flink (probably about states), but I > insist that unnecessary calculations should be avoided in stream processing. > > So, could you give some advices, I am all ears : ), or if you think that > is feasible, I'll think carefully and try to complete it. > > Thank you and merry Christmas. > > Best, > > - Xingcan > > On Thu, Dec 1, 2016 at 7:56 PM, Aljoscha Krettek <aljos...@apache.org> > wrote: > > I'm not aware of how windows work in Storm. If you could maybe give some > details on your use case we could figure out together how that would map to > Flink windows. > > Cheers, > Aljoscha > > On Tue, 29 Nov 2016 at 15:47 xingcan <xingc...@gmail.com> wrote: > > Hi all, > > Recently I tried to transfer some old applications from Storm to Flink. > In Storm, the window implementation (TupleWindow) gets two methods named > getNew() and getExpired() which supply the delta information of a window > and therefore we wrote some stateful caches that are aware of them. > However, it seems that Flink deals with the window in a different way and > supplies more "formalized" APIs. > So, any tips on how to adapt these delta awareness caches in Flink or do > some refactors to make them suitable? > > Thanks. > > Best, > Xingcan > > >
