Hi, the elements are currently not being removed from the buffers. That's a bug that we could fix while adding the new Evictor interface.
Cheers, Aljoscha On Mon, 25 Jul 2016 at 13:00 Radu Tudoran <radu.tudo...@huawei.com> wrote: > Hi Aljoscha, > > Can you point us to the way it is handled now. Is there anything else for > the removing of elements other than the skip in EvictingWindowOperator. Is > there something as it was before version 1.x where you had an explicit > remove from window buffers? > > Dr. Radu Tudoran > Research Engineer - Big Data Expert > IT R&D Division > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > European Research Center > Riesstrasse 25, 80992 München > > E-mail: radu.tudo...@huawei.com > Mobile: +49 15209084330 > Telephone: +49 891588344173 > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com > Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063, > Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN > Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063, > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN > This e-mail and its attachments contain confidential information from > HUAWEI, which is intended only for the person or entity whose address is > listed above. Any use of the information contained herein in any way > (including, but not limited to, total or partial disclosure, reproduction, > or dissemination) by persons other than the intended recipient(s) is > prohibited. If you receive this e-mail in error, please notify the sender > by phone or email immediately and delete it! > > > -----Original Message----- > From: Aljoscha Krettek [mailto:aljos...@apache.org] > Sent: Monday, July 25, 2016 11:45 AM > To: dev@flink.apache.org > Subject: Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink > > Hi, > I think there is not yet a clear specification for how the actual removal > of elements from the buffer will work. I think naively one can do: > > Iterable<E> currentElements = state.get() > evictor.evict(currentElements); // this will remove some stuff from there, > or mark for removal > > state.clear() > // the Iterable does not loop over the removed/marked elements > for (E element : currentElements) { > state.add(element) > } > > This is very costly but the only way I see of doing this right now with > every state backend. > > Cheers, > Aljoscha > > On Mon, 25 Jul 2016 at 09:46 Radu Tudoran <radu.tudo...@huawei.com> wrote: > > > Hi, > > > > Thanks for the clarification. Can someone point to where the events are > > removed from buffers - I am trying to understand the new logic of > handling > > the eviction in this new API. Thanks > > > > > > > > -----Original Message----- > > From: Vishnu Viswanath [mailto:vishnu.viswanat...@gmail.com] > > Sent: Saturday, July 23, 2016 3:04 AM > > To: Dev > > Subject: Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink > > > > Hi Radu, > > > > - Yes we can remove elements from the iterator. > > - Right now the EvictingWindowOperator just skips the elements from the > > Iterable before passing to the window function(Yes this has to be changed > > in the new API) > > - Regarding how the last question on how elements are being removed from > > the window buffer. I am not sure how it is working right now, but when > > trying out the new API that I am working on, I did find a bug where the > > evicted elements are not actually removed from the State. I have added a > > fix for that. (You can see a mail regarding that in this mail chain) > > > > Thanks, > > Vishnu > > > > On Fri, Jul 22, 2016 at 1:03 PM, Radu Tudoran <radu.tudo...@huawei.com> > > wrote: > > > > > Hi, > > > > > > Overall I believe that the interfaces and the proposal is good. I have > > the > > > following question though: can you delete via the iterator > > > (Iterable<StreamRecord<T>> elements) the elements? > > > > > > I tried to look over the code where the eviction happens (I did not do > > > these since version 0.10...looks very different now :) )...the only > > > reference I found was the EvictingWindowOperator which at the > > > fireOrContinue has a "skip" based on the number of elements returned > from > > > the evictor...and these are not put in the collection to be given to > the > > > user function to be applied. I think these will also need to be changed > > to > > > adjust to the "any operator from anywhere in the window buffer". > > > Also - as we are on this topic - can someone explain how these elements > > > that are not consider anymore for the user function are actually > deleted > > > from the window buffer?..i did not manage to find this.. some reference > > to > > > classes/code where this happens would be useful > > > > > > > > > Dr. Radu Tudoran > > > Research Engineer - Big Data Expert > > > IT R&D Division > > > > > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > > European Research Center > > > Riesstrasse 25, 80992 München > > > > > > E-mail: radu.tudo...@huawei.com > > > Mobile: +49 15209084330 > > > Telephone: +49 891588344173 > > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > > Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com > > > Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063, > > > Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN > > > Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063, > > > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN > > > This e-mail and its attachments contain confidential information from > > > HUAWEI, which is intended only for the person or entity whose address > is > > > listed above. Any use of the information contained herein in any way > > > (including, but not limited to, total or partial disclosure, > > reproduction, > > > or dissemination) by persons other than the intended recipient(s) is > > > prohibited. If you receive this e-mail in error, please notify the > sender > > > by phone or email immediately and delete it! > > > > > > > > > -----Original Message----- > > > From: Vishnu Viswanath [mailto:vishnu.viswanat...@gmail.com] > > > Sent: Friday, July 22, 2016 12:43 PM > > > To: Dev > > > Subject: Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink > > > > > > Hi, > > > > > > I have created a FLIP page for this enhancement > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor > > > > > > Thanks, > > > Vishnu > > > > > > On Thu, Jul 21, 2016 at 6:53 AM, Vishnu Viswanath < > > > vishnu.viswanat...@gmail.com> wrote: > > > > > > > Thanks Aljoscha. > > > > > > > > On Thu, Jul 21, 2016 at 4:46 AM, Aljoscha Krettek < > aljos...@apache.org > > > > > > > wrote: > > > > > > > >> Hi, > > > >> this, in fact, seems to be a bug. There should be something like > > > >> windowState.clear(); > > > >> for (IN element: projectedContents) { > > > >> windowState.add(element); > > > >> } > > > >> > > > >> after passing the elements to the window function. > > > >> > > > >> This is very inefficient but the only way I see of doing it right > now. > > > >> > > > >> Cheers, > > > >> Aljoscha > > > >> > > > >> > > > >> On Thu, 21 Jul 2016 at 01:32 Vishnu Viswanath < > > > >> vishnu.viswanat...@gmail.com> > > > >> wrote: > > > >> > > > >> > Hi, > > > >> > > > > >> > When we use RocksDB as state backend, how does the backend state > get > > > >> > updated after some elements are evicted from the window? > > > >> > I don't see any update call being made to remove the element from > > the > > > >> state > > > >> > stored in RocksDB. > > > >> > > > > >> > It looks like the RocksDBListState is only having get() and add() > > > >> methods > > > >> > since it is an AppendingState, but that causes the evicted > elements > > to > > > >> come > > > >> > back when the trigger is fired next time. (It works fine when I > use > > > >> > MemoryStateBackend) > > > >> > > > > >> > Is this expected behavior or am I missing something. > > > >> > > > > >> > Thanks, > > > >> > Vishnu > > > >> > > > > >> > On Mon, Jul 18, 2016 at 7:15 AM, Vishnu Viswanath < > > > >> > vishnu.viswanat...@gmail.com> wrote: > > > >> > > > > >> > > Hi Aljoscha, > > > >> > > > > > >> > > Thanks! Yes, I have the create page option now in wiki. > > > >> > > > > > >> > > Regards, > > > >> > > Vishnu Viswanath, > > > >> > > > > > >> > > On Mon, Jul 18, 2016 at 6:34 AM, Aljoscha Krettek < > > > >> aljos...@apache.org> > > > >> > > wrote: > > > >> > > > > > >> > >> @Radu, addition of more window types and sorting should be part > > of > > > >> > another > > > >> > >> design proposal. This is interesting stuff but I think we > should > > > keep > > > >> > >> issues separated because things can get complicated very > quickly. > > > >> > >> > > > >> > >> On Mon, 18 Jul 2016 at 12:32 Aljoscha Krettek < > > aljos...@apache.org > > > > > > > >> > >> wrote: > > > >> > >> > > > >> > >> > Hi, > > > >> > >> > about TimeEvictor, yes, I think there should be specific > > evictors > > > >> for > > > >> > >> > processing time and event time. Also, the current time should > > be > > > >> > >> > retrievable from the EvictorContext. > > > >> > >> > > > > >> > >> > For the wiki you will need permissions. This was recently > > changed > > > >> > >> because > > > >> > >> > there was too much spam. I gave you permission to add pages. > > Can > > > >> you > > > >> > >> please > > > >> > >> > try and check if it works? > > > >> > >> > > > > >> > >> > Cheers, > > > >> > >> > Aljoscha > > > >> > >> > > > > >> > >> > On Fri, 15 Jul 2016 at 13:28 Vishnu Viswanath < > > > >> > >> > vishnu.viswanat...@gmail.com> wrote: > > > >> > >> > > > > >> > >> >> Hi all, > > > >> > >> >> > > > >> > >> >> How do we create a FLIP page, is there any permission setup > > > >> > required? I > > > >> > >> >> don't see any "Create" page(after logging in) option in the > > > >> header as > > > >> > >> >> mentioned in > > > >> > >> >> > > > >> > >> >> > > > >> > >> > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > >> > >> >> > > > >> > >> >> > > > >> > >> >> Thanks, > > > >> > >> >> Vishnu > > > >> > >> >> > > > >> > >> >> On Wed, Jul 13, 2016 at 10:22 PM, Vishnu Viswanath < > > > >> > >> >> vishnu.viswanat...@gmail.com> wrote: > > > >> > >> >> > > > >> > >> >> > Hi Aljoscha, > > > >> > >> >> > > > > >> > >> >> > I agree, the user will know exactly that they are creating > > an > > > >> > >> EventTime > > > >> > >> >> > based evictor or ProcessingTime based evictor looking at > the > > > >> code. > > > >> > >> >> > So do you think it will be ok to have multiple versions of > > > >> > >> TimeEvictor > > > >> > >> >> > (one for event time and one for processing time) and also > a > > > >> > >> DeltaEvcitor > > > >> > >> >> > (again 2 versions- for event time and processing time) ? > > > >> > >> >> > > > > >> > >> >> > Please note that the existing behavior of > > > >> TimeEvictor/DeltaEvictor > > > >> > >> does > > > >> > >> >> > not consider if it is EventTime or ProcessingTime > > > >> > >> >> > e.g., in TimeEvictor the current time is considered as the > > > >> > timestamp > > > >> > >> of > > > >> > >> >> > the last element in the window > > > >> > >> >> > > > > >> > >> >> > *long currentTime = > > > Iterables.getLast(elements).getTimestamp();* > > > >> > >> >> > > > > >> > >> >> > not the highest timestamp of all elements > > > >> > >> >> > what I am trying to achieve is something like: > > > >> > >> >> > > > > >> > >> >> > *long currentTime;* > > > >> > >> >> > * if (ctx.isEventTime()) {* > > > >> > >> >> > * currentTime = getMaxTimestamp(elements);* > > > >> > >> >> > * } else {* > > > >> > >> >> > * currentTime = > Iterables.getLast(elements).getTimestamp();* > > > >> > >> >> > * }* > > > >> > >> >> > > > > >> > >> >> > Similarly, in DeltaEvictor the *`lastElement`* is > > > >> > >> >> > *`Iterables.getLast(elements);`* and I am thinking we > should > > > >> > consider > > > >> > >> >> the > > > >> > >> >> > element with max timestamp as the last element instead of > > just > > > >> > >> getting > > > >> > >> >> the > > > >> > >> >> > last inserted element as *`lastElement`* > > > >> > >> >> > > > > >> > >> >> > Do you think it is the right thing to do or leave the > > behavior > > > >> > >> Evictors > > > >> > >> >> as > > > >> > >> >> > is, w.r.t to choosing the last element? > > > >> > >> >> > > > > >> > >> >> > Thanks, > > > >> > >> >> > Vishnu > > > >> > >> >> > > > > >> > >> >> > On Wed, Jul 13, 2016 at 11:07 AM, Aljoscha Krettek < > > > >> > >> aljos...@apache.org > > > >> > >> >> > > > > >> > >> >> > wrote: > > > >> > >> >> > > > > >> > >> >> >> I still think it should be explicit in the class. For > > > example, > > > >> if > > > >> > >> you > > > >> > >> >> have > > > >> > >> >> >> this code: > > > >> > >> >> >> > > > >> > >> >> >> input > > > >> > >> >> >> .keyBy() > > > >> > >> >> >> .window() > > > >> > >> >> >> .trigger(EventTimeTrigger.create()) > > > >> > >> >> >> .evictor(TimeTrigger.create()) > > > >> > >> >> >> > > > >> > >> >> >> the time behavior of the trigger is explicitly specified > > > while > > > >> the > > > >> > >> >> evictor > > > >> > >> >> >> would dynamically adapt based on internal workings that > the > > > >> user > > > >> > >> might > > > >> > >> >> not > > > >> > >> >> >> be aware of. Having the behavior explicit at the call > site > > is > > > >> very > > > >> > >> >> >> important, in my opinion. > > > >> > >> >> >> > > > >> > >> >> >> On Wed, 13 Jul 2016 at 16:28 Vishnu Viswanath < > > > >> > >> >> >> vishnu.viswanat...@gmail.com> > > > >> > >> >> >> wrote: > > > >> > >> >> >> > > > >> > >> >> >> > Hi, > > > >> > >> >> >> > > > > >> > >> >> >> > I was hoping to use the isEventTime method in the > > > >> WindowAssigner > > > >> > >> to > > > >> > >> >> set > > > >> > >> >> >> > that information in the EvictorContext. > > > >> > >> >> >> > What do you think?. > > > >> > >> >> >> > > > > >> > >> >> >> > Thanks and Regards, > > > >> > >> >> >> > Vishnu Viswanath, > > > >> > >> >> >> > > > > >> > >> >> >> > On Wed, Jul 13, 2016 at 10:09 AM, Aljoscha Krettek < > > > >> > >> >> aljos...@apache.org > > > >> > >> >> >> > > > > >> > >> >> >> > wrote: > > > >> > >> >> >> > > > > >> > >> >> >> > > Hi, > > > >> > >> >> >> > > I think the way to go here is to add both an > > > >> EventTimeEvictor > > > >> > >> and a > > > >> > >> >> >> > > ProcessingTimeEvictor. The problem is that > > "isEventTime" > > > >> > cannot > > > >> > >> >> >> really be > > > >> > >> >> >> > > determined. That's also the reason why there is an > > > >> > >> EventTimeTrigger > > > >> > >> >> >> and a > > > >> > >> >> >> > > ProcessingTimeTrigger. It was just an oversight that > > the > > > >> > >> >> TimeEvictor > > > >> > >> >> >> does > > > >> > >> >> >> > > not also have these two versions. > > > >> > >> >> >> > > > > > >> > >> >> >> > > About EvictingWindowOperator, I think you can make > the > > > two > > > >> > >> methods > > > >> > >> >> >> > > non-final in WindowOperator, yes. > > > >> > >> >> >> > > > > > >> > >> >> >> > > Cheers, > > > >> > >> >> >> > > Aljoscha > > > >> > >> >> >> > > > > > >> > >> >> >> > > On Wed, 13 Jul 2016 at 14:32 Vishnu Viswanath < > > > >> > >> >> >> > > vishnu.viswanat...@gmail.com> > > > >> > >> >> >> > > wrote: > > > >> > >> >> >> > > > > > >> > >> >> >> > > > Hi Aljoscha, > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > I am thinking of adding a method boolean > > isEventTime(); > > > >> in > > > >> > the > > > >> > >> >> >> > > > EvictorContext apart from > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > long getCurrentProcessingTime(); > > > >> > >> >> >> > > > MetricGroup getMetricGroup(); > > > >> > >> >> >> > > > long getCurrentWatermark(); > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > This method can be used to make the Evictor not > > iterate > > > >> > >> through > > > >> > >> >> all > > > >> > >> >> >> the > > > >> > >> >> >> > > > elements in TimeEvictor. There will be a few > changes > > in > > > >> the > > > >> > >> >> existing > > > >> > >> >> >> > > > behavior of TimeEvictor and DeltaEvictor (I have > > > >> mentioned > > > >> > >> this > > > >> > >> >> in > > > >> > >> >> >> the > > > >> > >> >> >> > > > design doc) > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > Also, is there any specific reason why the open and > > > close > > > >> > >> method > > > >> > >> >> in > > > >> > >> >> >> > > > WindowEvictor is made final? Since the > EvictorContext > > > >> will > > > >> > be > > > >> > >> in > > > >> > >> >> the > > > >> > >> >> >> > > > EvictingWindowOperator, I need to override the open > > and > > > >> > close > > > >> > >> in > > > >> > >> >> >> > > > EvitingWindowOperator to make the reference of > > > >> > EvictorContext > > > >> > >> >> null. > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > Thanks and Regards, > > > >> > >> >> >> > > > Vishnu Viswanath, > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > On Fri, Jul 8, 2016 at 7:40 PM, Vishnu Viswanath < > > > >> > >> >> >> > > > vishnu.viswanat...@gmail.com> wrote: > > > >> > >> >> >> > > > > > > >> > >> >> >> > > > My thought process when asking if we can use state > > > >> backend > > > >> > in > > > >> > >> >> window > > > >> > >> >> >> > > > > function was : can we add the elements to be > > evicted > > > >> into > > > >> > >> some > > > >> > >> >> >> state > > > >> > >> >> >> > > and > > > >> > >> >> >> > > > > allow the evictAfter to read it from some context > > and > > > >> > >> remove it > > > >> > >> >> >> from > > > >> > >> >> >> > > the > > > >> > >> >> >> > > > > window? > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > > On Fri, Jul 8, 2016 at 7:30 PM, Vishnu Viswanath > < > > > >> > >> >> >> > > > > vishnu.viswanat...@gmail.com> wrote: > > > >> > >> >> >> > > > > > > > >> > >> >> >> > > > >> Hi Aljoscha, > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >> Thanks for the explanation, and sorry for late > > reply > > > >> was > > > >> > >> busy > > > >> > >> >> >> with > > > >> > >> >> >> > > work. > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >> I did think about this scenario, in fact in my > > > >> previous > > > >> > >> mail I > > > >> > >> >> >> > thought > > > >> > >> >> >> > > > of > > > >> > >> >> >> > > > >> posting this question, then I understood that > this > > > >> > problem > > > >> > >> >> will > > > >> > >> >> >> be > > > >> > >> >> >> > > > >> there which ever method we choose(Trigger > looking > > > for > > > >> > >> pattern > > > >> > >> >> or > > > >> > >> >> >> > > Window > > > >> > >> >> >> > > > >> looking for pattern). > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >> I do have a pretty good watermark but my concern > > is > > > >> that > > > >> > it > > > >> > >> >> >> changes > > > >> > >> >> >> > > > based > > > >> > >> >> >> > > > >> on the key of these messages(I don't know if it > is > > > >> > >> possible, > > > >> > >> >> >> haven't > > > >> > >> >> >> > > > >> started coding that yet. May be you could tell > > me). > > > >> Even > > > >> > if > > > >> > >> >> it is > > > >> > >> >> >> > yes > > > >> > >> >> >> > > > some > > > >> > >> >> >> > > > >> of these watermarks will be long(in days), > which I > > > >> don't > > > >> > >> want > > > >> > >> >> the > > > >> > >> >> >> > > > trigger > > > >> > >> >> >> > > > >> to wait that long. > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >> It looks like it is not easy to have an > evictAfter > > > >> based > > > >> > on > > > >> > >> >> >> window > > > >> > >> >> >> > > > >> function(without introducing coupling), but can > > the > > > >> > current > > > >> > >> >> >> window > > > >> > >> >> >> > > apply > > > >> > >> >> >> > > > >> function be modified to allow it to change the > > > >> elements > > > >> > in > > > >> > >> it > > > >> > >> >> - > > > >> > >> >> >> may > > > >> > >> >> >> > be > > > >> > >> >> >> > > > >> using some state backend(I don't know how > excatly > > > the > > > >> > >> >> internals > > > >> > >> >> >> of > > > >> > >> >> >> > > these > > > >> > >> >> >> > > > >> work, so this might be a wrong question) > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >> Thanks and Regards, > > > >> > >> >> >> > > > >> Vishnu Viswanath, > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >> On Fri, Jul 8, 2016 at 10:20 AM, Aljoscha > Krettek > > < > > > >> > >> >> >> > > aljos...@apache.org> > > > >> > >> >> >> > > > >> wrote: > > > >> > >> >> >> > > > >> > > > >> > >> >> >> > > > >>> Hi Vishnu, > > > >> > >> >> >> > > > >>> how long would these patterns be? The Trigger > > would > > > >> not > > > >> > >> have > > > >> > >> >> to > > > >> > >> >> >> > sort > > > >> > >> >> >> > > > the > > > >> > >> >> >> > > > >>> elements for every new element but just insert > > the > > > >> new > > > >> > >> >> element > > > >> > >> >> >> into > > > >> > >> >> >> > > an > > > >> > >> >> >> > > > >>> internal data structure. Only when it sees that > > the > > > >> > >> >> watermark is > > > >> > >> >> >> > > past a > > > >> > >> >> >> > > > >>> certain point would it check whether the > pattern > > > >> matches > > > >> > >> and > > > >> > >> >> >> > actually > > > >> > >> >> >> > > > >>> Trigger. > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> A general note regarding order and event time: > I > > > >> think > > > >> > >> >> relying > > > >> > >> >> >> on > > > >> > >> >> >> > > this > > > >> > >> >> >> > > > >>> for > > > >> > >> >> >> > > > >>> computation is very tricky unless the watermark > > is > > > >> 100 % > > > >> > >> >> >> correct or > > > >> > >> >> >> > > you > > > >> > >> >> >> > > > >>> completely discard elements that arrive after > the > > > >> > >> watermark, > > > >> > >> >> >> i.e. > > > >> > >> >> >> > > > >>> elements > > > >> > >> >> >> > > > >>> that would break the promise of the watermark > > that > > > no > > > >> > >> >> elements > > > >> > >> >> >> with > > > >> > >> >> >> > > an > > > >> > >> >> >> > > > >>> earlier timestamp will ever arrive. The reason > > for > > > >> this > > > >> > is > > > >> > >> >> that > > > >> > >> >> >> > there > > > >> > >> >> >> > > > >>> could > > > >> > >> >> >> > > > >>> always enter new elements that end up between > > > already > > > >> > seen > > > >> > >> >> >> > elements. > > > >> > >> >> >> > > > For > > > >> > >> >> >> > > > >>> example, let's say we have this sequence of > > > elements > > > >> > when > > > >> > >> the > > > >> > >> >> >> > trigger > > > >> > >> >> >> > > > >>> fires: > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> a-b-a > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> This is the sequence that you are looking for > and > > > you > > > >> > emit > > > >> > >> >> some > > > >> > >> >> >> > > result > > > >> > >> >> >> > > > >>> from > > > >> > >> >> >> > > > >>> the WindowFunction. Now, new elements arrive > that > > > >> fall > > > >> > in > > > >> > >> >> >> between > > > >> > >> >> >> > the > > > >> > >> >> >> > > > >>> elements we already have: > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> a-d-e-b-f-g-a > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> This is an updated, sorted view of the actual > > > >> event-time > > > >> > >> >> stream > > > >> > >> >> >> and > > > >> > >> >> >> > > we > > > >> > >> >> >> > > > >>> didn't realize that the stream actually looks > > like > > > >> this > > > >> > >> >> before. > > > >> > >> >> >> > Does > > > >> > >> >> >> > > > this > > > >> > >> >> >> > > > >>> still match the original pattern or should we > now > > > >> > consider > > > >> > >> >> this > > > >> > >> >> >> as > > > >> > >> >> >> > > > >>> non-matching? If no, then the earlier > successful > > > >> match > > > >> > for > > > >> > >> >> a-b-a > > > >> > >> >> >> > was > > > >> > >> >> >> > > > >>> wrong > > > >> > >> >> >> > > > >>> and we should never have processed it but we > > didn't > > > >> know > > > >> > >> at > > > >> > >> >> the > > > >> > >> >> >> > time. > > > >> > >> >> >> > > > If > > > >> > >> >> >> > > > >>> yes, then pattern matching like this can be > done > > in > > > >> the > > > >> > >> >> Trigger > > > >> > >> >> >> by > > > >> > >> >> >> > > > having > > > >> > >> >> >> > > > >>> something like pattern slots: You don't have to > > > store > > > >> > all > > > >> > >> >> >> elements > > > >> > >> >> >> > in > > > >> > >> >> >> > > > the > > > >> > >> >> >> > > > >>> Trigger, you just need to store possible > > candidates > > > >> that > > > >> > >> >> could > > > >> > >> >> >> > match > > > >> > >> >> >> > > > the > > > >> > >> >> >> > > > >>> pattern and ignore the other (in-between) > > elements. > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> Cheers, > > > >> > >> >> >> > > > >>> Aljoscha > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> On Fri, 8 Jul 2016 at 14:10 Vishnu Viswanath < > > > >> > >> >> >> > > > >>> vishnu.viswanat...@gmail.com> > > > >> > >> >> >> > > > >>> wrote: > > > >> > >> >> >> > > > >>> > > > >> > >> >> >> > > > >>> > Hi Aljoscha, > > > >> > >> >> >> > > > >>> > > > > >> > >> >> >> > > > >>> > That is a good idea, trying to tie it back to > > the > > > >> use > > > >> > >> case, > > > >> > >> >> >> > > > >>> > e.g., suppose trigger is looking for a > pattern, > > > >> a-b-a > > > >> > >> and > > > >> > >> >> >> when it > > > >> > >> >> >> > > > sees > > > >> > >> >> >> > > > >>> such > > > >> > >> >> >> > > > >>> > a pattern, it will trigger the window and it > > > knows > > > >> > that > > > >> > >> now > > > >> > >> >> >> the > > > >> > >> >> >> > > > >>> Evictor is > > > >> > >> >> >> > > > >>> > going to evict the element b, and trigger > > updates > > > >> its > > > >> > >> >> state as > > > >> > >> >> >> > a-a > > > >> > >> >> >> > > > >>> (even > > > >> > >> >> >> > > > >>> > before the window & evictor completes) and > will > > > be > > > >> > >> looking > > > >> > >> >> for > > > >> > >> >> >> > the > > > >> > >> >> >> > > > >>> rest of > > > >> > >> >> >> > > > >>> > the pattern i.e., b-a. But I can think of 1 > > > problem > > > >> > >> here, > > > >> > >> >> >> > > > >>> > > > > >> > >> >> >> > > > >>> > - the events can arrive out of order, > i.e., > > > the > > > >> > >> trigger > > > >> > >> >> >> might > > > >> > >> >> >> > be > > > >> > >> >> >> > > > >>> seeing > > > >> > >> >> >> > > > >>> > a pattern a-a-b but actual event time is > > a-b-a > > > >> then > > > >> > >> >> trigger > > > >> > >> >> >> > will > > > >> > >> >> >> > > > >>> have to > > > >> > >> >> >> > > > >>> > sort the elements in the window everytime > it > > > >> sees > > > >> > an > > > >> > >> >> >> element. > > > >> > >> >> >> > (I > > > >> > >> >> >> > > > was > > > >> > >> >> >> > > > >>> > planning to do this sorting in the window, > > > which > > > >> > >> will be > > > >> > >> >> >> less > > > >> > >> >> >> > > > often > > > >> > >> >> >> > > > >>> - > > > >> > >> >> >> > > > >>> > only > > > >> > >> >> >> > > > >>> > when the trigger fires) > > > >> > >> >> >> > > > >>> > > > > >> > >> >> >> > > > >>> > Thanks and Regards, > > > >> > >> >> >> > > > >>> > Vishnu Viswanath, > > > >> > >> >> >> > > > >>> > > > > >> > >> >> >> > > > >>> > On Fri, Jul 8, 2016 at 6:04 AM, Aljoscha > > Krettek > > > < > > > >> > >> >> >> > > > aljos...@apache.org> > > > >> > >> >> >> > > > >>> > wrote: > > > >> > >> >> >> > > > >>> > > > > >> > >> >> >> > > > >>> > Hi, > > > >> > >> >> >> > > > >>> > > come to think of it, the right place to put > > > such > > > >> > >> checks > > > >> > >> >> is > > > >> > >> >> >> > > actually > > > >> > >> >> >> > > > >>> the > > > >> > >> >> >> > > > >>> > > Trigger. It would have to be a custom > trigger > > > >> that > > > >> > >> >> observes > > > >> > >> >> >> > time > > > >> > >> >> >> > > > but > > > >> > >> >> >> > > > >>> also > > > >> > >> >> >> > > > >>> > > keeps some internal state machine to decide > > > when > > > >> it > > > >> > >> has > > > >> > >> >> >> > observed > > > >> > >> >> >> > > > the > > > >> > >> >> >> > > > >>> > right > > > >> > >> >> >> > > > >>> > > pattern in the window. Then the window > > function > > > >> > would > > > >> > >> >> just > > > >> > >> >> >> have > > > >> > >> >> >> > > to > > > >> > >> >> >> > > > >>> do the > > > >> > >> >> >> > > > >>> > > processing and you have good separation of > > > >> concerns. > > > >> > >> Does > > > >> > >> >> >> that > > > >> > >> >> >> > > make > > > >> > >> >> >> > > > >>> > sense? > > > >> > >> >> >> > > > >>> > > > > > >> > >> >> >> > > > >>> > > I'm ignoring time and sorting by time for > now > > > >> > because > > > >> > >> we > > > >> > >> >> >> > probably > > > >> > >> >> >> > > > >>> need > > > >> > >> >> >> > > > >>> > > another design document for that. To me it > > > seems > > > >> > like > > > >> > >> a > > > >> > >> >> >> bigger > > > >> > >> >> >> > > > thing. > > > >> > >> >> >> > > > >>> > > > > > >> > >> >> >> > > > >>> > > Cheers, > > > >> > >> >> >> > > > >>> > > Aljoscha > > > >> > >> >> >> > > > >>> > > > > > >> > >> >> >> > > > >>> > > On Thu, 7 Jul 2016 at 23:56 Vishnu > Viswanath > > < > > > >> > >> >> >> > > > >>> > vishnu.viswanat...@gmail.com > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > wrote: > > > >> > >> >> >> > > > >>> > > > > > >> > >> >> >> > > > >>> > > > Hi, > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > > Regarding the evictAfter function, that > > > evicts > > > >> > >> based on > > > >> > >> >> >> some > > > >> > >> >> >> > > > >>> decision > > > >> > >> >> >> > > > >>> > > made > > > >> > >> >> >> > > > >>> > > > by the window function: I think it will > be > > > >> nice > > > >> > if > > > >> > >> we > > > >> > >> >> can > > > >> > >> >> >> > come > > > >> > >> >> >> > > > up > > > >> > >> >> >> > > > >>> with > > > >> > >> >> >> > > > >>> > > > something that is LESS coupled, because I > > can > > > >> > think > > > >> > >> of > > > >> > >> >> >> > several > > > >> > >> >> >> > > > use > > > >> > >> >> >> > > > >>> > cases > > > >> > >> >> >> > > > >>> > > > that depend on it. > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > > Especially in the case where there are > late > > > >> > arriving > > > >> > >> >> >> > messages. > > > >> > >> >> >> > > > Only > > > >> > >> >> >> > > > >>> > after > > > >> > >> >> >> > > > >>> > > > the window function is applied we could > > tell > > > >> what > > > >> > >> to do > > > >> > >> >> >> with > > > >> > >> >> >> > > the > > > >> > >> >> >> > > > >>> > elements > > > >> > >> >> >> > > > >>> > > > in the window. You could apply your > > business > > > >> logic > > > >> > >> >> there > > > >> > >> >> >> to > > > >> > >> >> >> > > > >>> determine > > > >> > >> >> >> > > > >>> > if > > > >> > >> >> >> > > > >>> > > > the window funciton was able to do what > it > > is > > > >> > >> supposed > > > >> > >> >> to > > > >> > >> >> >> do, > > > >> > >> >> >> > > if > > > >> > >> >> >> > > > >>> yes > > > >> > >> >> >> > > > >>> > > evict > > > >> > >> >> >> > > > >>> > > > those elements, else(since the elements > you > > > are > > > >> > >> looking > > > >> > >> >> >> for > > > >> > >> >> >> > > > haven't > > > >> > >> >> >> > > > >>> > > arrived > > > >> > >> >> >> > > > >>> > > > yet) wait and try again when the trigger > > gets > > > >> > fired > > > >> > >> >> next > > > >> > >> >> >> > time. > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > > Thanks and Regards, > > > >> > >> >> >> > > > >>> > > > Vishnu Viswanath, > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > > On Thu, Jul 7, 2016 at 9:19 AM, Radu > > Tudoran > > > < > > > >> > >> >> >> > > > >>> radu.tudo...@huawei.com> > > > >> > >> >> >> > > > >>> > > > wrote: > > > >> > >> >> >> > > > >>> > > > > > > >> > >> >> >> > > > >>> > > > > Hi, > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > @Aljoscha - I can understand the reason > > why > > > >> you > > > >> > >> are > > > >> > >> >> >> > hesitant > > > >> > >> >> >> > > to > > > >> > >> >> >> > > > >>> > > introduce > > > >> > >> >> >> > > > >>> > > > > "slower" windows such as the ones that > > > would > > > >> > >> maintain > > > >> > >> >> >> > sorted > > > >> > >> >> >> > > > >>> items or > > > >> > >> >> >> > > > >>> > > > > windows with bindings between the > > different > > > >> > >> entities > > > >> > >> >> >> > > (evictor, > > > >> > >> >> >> > > > >>> > trigger, > > > >> > >> >> >> > > > >>> > > > > window, apply function). However, I > think > > > >> it's > > > >> > >> >> possible > > > >> > >> >> >> > just > > > >> > >> >> >> > > to > > > >> > >> >> >> > > > >>> > create > > > >> > >> >> >> > > > >>> > > > more > > > >> > >> >> >> > > > >>> > > > > types of windows. The existing ones > > > >> > (timewindows, > > > >> > >> >> global > > > >> > >> >> >> > > > windows > > > >> > >> >> >> > > > >>> ...) > > > >> > >> >> >> > > > >>> > > can > > > >> > >> >> >> > > > >>> > > > > remain, and just add some more flavors > of > > > >> > windows > > > >> > >> >> were > > > >> > >> >> >> more > > > >> > >> >> >> > > > >>> features > > > >> > >> >> >> > > > >>> > > are > > > >> > >> >> >> > > > >>> > > > > enabled or more functionality (e.g., > > access > > > >> to > > > >> > the > > > >> > >> >> each > > > >> > >> >> >> > > element > > > >> > >> >> >> > > > >>> in > > > >> > >> >> >> > > > >>> > the > > > >> > >> >> >> > > > >>> > > > > evictor ; possibility to delete or mark > > for > > > >> > >> eviction > > > >> > >> >> >> > elements > > > >> > >> >> >> > > > in > > > >> > >> >> >> > > > >>> the > > > >> > >> >> >> > > > >>> > > > > function...) > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > Regarding the specific case of sorted > > > >> windows, I > > > >> > >> >> think > > > >> > >> >> >> the > > > >> > >> >> >> > N > > > >> > >> >> >> > > > lon > > > >> > >> >> >> > > > >>> N > > > >> > >> >> >> > > > >>> > > > > complexity to sort (the worst case) is > > very > > > >> > >> >> unlikely. In > > > >> > >> >> >> > fact > > > >> > >> >> >> > > > you > > > >> > >> >> >> > > > >>> > have > > > >> > >> >> >> > > > >>> > > > > almost sorted items/arrays. Moreover, > if > > > you > > > >> > >> consider > > > >> > >> >> >> that > > > >> > >> >> >> > in > > > >> > >> >> >> > > > >>> > > iteration X > > > >> > >> >> >> > > > >>> > > > > all elements were sorted, then in > > iteration > > > >> X+1 > > > >> > >> you > > > >> > >> >> will > > > >> > >> >> >> > need > > > >> > >> >> >> > > > to > > > >> > >> >> >> > > > >>> sort > > > >> > >> >> >> > > > >>> > > > just > > > >> > >> >> >> > > > >>> > > > > the newly arrived elements (M). I would > > > >> expect > > > >> > >> that > > > >> > >> >> this > > > >> > >> >> >> > > > number M > > > >> > >> >> >> > > > >>> > might > > > >> > >> >> >> > > > >>> > > > be > > > >> > >> >> >> > > > >>> > > > > significant smaller then N (elements > that > > > >> > exists). > > > >> > >> >> Then > > > >> > >> >> >> > using > > > >> > >> >> >> > > > an > > > >> > >> >> >> > > > >>> > > > insertion > > > >> > >> >> >> > > > >>> > > > > sort for these new elements you would > > have > > > >> M * > > > >> > N > > > >> > >> >> >> > complexity > > > >> > >> >> >> > > > and > > > >> > >> >> >> > > > >>> if > > > >> > >> >> >> > > > >>> > > M<< N > > > >> > >> >> >> > > > >>> > > > > then the complexity is O(N). > > Alternatively > > > >> you > > > >> > can > > > >> > >> >> use a > > > >> > >> >> >> > > binary > > > >> > >> >> >> > > > >>> > search > > > >> > >> >> >> > > > >>> > > > for > > > >> > >> >> >> > > > >>> > > > > insertion and then you further reduce > the > > > >> > >> complexity > > > >> > >> >> to > > > >> > >> >> >> > > > O(logN). > > > >> > >> >> >> > > > >>> > > > > If M is proportional to N then you can > > > sort M > > > >> > and > > > >> > >> use > > > >> > >> >> >> merge > > > >> > >> >> >> > > > sort > > > >> > >> >> >> > > > >>> for > > > >> > >> >> >> > > > >>> > > > > combining. > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > Dr. Radu Tudoran > > > >> > >> >> >> > > > >>> > > > > Research Engineer - Big Data Expert > > > >> > >> >> >> > > > >>> > > > > IT R&D Division > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > > >> > >> >> >> > > > >>> > > > > European Research Center > > > >> > >> >> >> > > > >>> > > > > Riesstrasse 25, 80992 München > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > E-mail: radu.tudo...@huawei.com > > > >> > >> >> >> > > > >>> > > > > Mobile: +49 15209084330 > > > >> > >> >> >> > > > >>> > > > > Telephone: +49 891588344173 > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > > >> > >> >> >> > > > >>> > > > > Hansaallee 205, 40549 Düsseldorf, > > Germany, > > > >> > >> >> >> www.huawei.com > > > >> > >> >> >> > > > >>> > > > > Registered Office: Düsseldorf, Register > > > Court > > > >> > >> >> >> Düsseldorf, > > > >> > >> >> >> > HRB > > > >> > >> >> >> > > > >>> 56063, > > > >> > >> >> >> > > > >>> > > > > Managing Director: Bo PENG, Wanzhou > MENG, > > > >> Lifang > > > >> > >> CHEN > > > >> > >> >> >> > > > >>> > > > > Sitz der Gesellschaft: Düsseldorf, > > > >> Amtsgericht > > > >> > >> >> >> Düsseldorf, > > > >> > >> >> >> > > HRB > > > >> > >> >> >> > > > >>> 56063, > > > >> > >> >> >> > > > >>> > > > > Geschäftsführer: Bo PENG, Wanzhou MENG, > > > >> Lifang > > > >> > >> CHEN > > > >> > >> >> >> > > > >>> > > > > This e-mail and its attachments contain > > > >> > >> confidential > > > >> > >> >> >> > > > information > > > >> > >> >> >> > > > >>> from > > > >> > >> >> >> > > > >>> > > > > HUAWEI, which is intended only for the > > > >> person or > > > >> > >> >> entity > > > >> > >> >> >> > whose > > > >> > >> >> >> > > > >>> address > > > >> > >> >> >> > > > >>> > > is > > > >> > >> >> >> > > > >>> > > > > listed above. Any use of the > information > > > >> > contained > > > >> > >> >> >> herein > > > >> > >> >> >> > in > > > >> > >> >> >> > > > any > > > >> > >> >> >> > > > >>> way > > > >> > >> >> >> > > > >>> > > > > (including, but not limited to, total > or > > > >> partial > > > >> > >> >> >> > disclosure, > > > >> > >> >> >> > > > >>> > > > reproduction, > > > >> > >> >> >> > > > >>> > > > > or dissemination) by persons other than > > the > > > >> > >> intended > > > >> > >> >> >> > > > >>> recipient(s) is > > > >> > >> >> >> > > > >>> > > > > prohibited. If you receive this e-mail > in > > > >> error, > > > >> > >> >> please > > > >> > >> >> >> > > notify > > > >> > >> >> >> > > > >>> the > > > >> > >> >> >> > > > >>> > > sender > > > >> > >> >> >> > > > >>> > > > > by phone or email immediately and > delete > > > it! > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > -----Original Message----- > > > >> > >> >> >> > > > >>> > > > > From: 吕文龙(吕文龙) [mailto: > > > >> > >> wenlong....@alibaba-inc.com] > > > >> > >> >> >> > > > >>> > > > > Sent: Thursday, July 07, 2016 11:59 AM > > > >> > >> >> >> > > > >>> > > > > To: dev@flink.apache.org > > > >> > >> >> >> > > > >>> > > > > Subject: 答复: [DISCUSS] Enhance Window > > > >> Evictor in > > > >> > >> >> Flink > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > HI, > > > >> > >> >> >> > > > >>> > > > > I think it is necessary to support > sorted > > > >> > window, > > > >> > >> >> which > > > >> > >> >> >> can > > > >> > >> >> >> > > > avoid > > > >> > >> >> >> > > > >>> > > > scanning > > > >> > >> >> >> > > > >>> > > > > all the elements of window while trying > > to > > > >> > >> evicting > > > >> > >> >> >> > element, > > > >> > >> >> >> > > > >>> which > > > >> > >> >> >> > > > >>> > may > > > >> > >> >> >> > > > >>> > > > cost > > > >> > >> >> >> > > > >>> > > > > many IO operations, such as querying > DBs > > to > > > >> get > > > >> > >> >> elements > > > >> > >> >> >> > from > > > >> > >> >> >> > > > >>> state. > > > >> > >> >> >> > > > >>> > > > > What's more, when an window aggregation > > > >> function > > > >> > >> is > > > >> > >> >> >> > > invertible, > > > >> > >> >> >> > > > >>> such > > > >> > >> >> >> > > > >>> > as > > > >> > >> >> >> > > > >>> > > > > sum, which can be updated by adding or > > > >> removing > > > >> > a > > > >> > >> >> single > > > >> > >> >> >> > > > record, > > > >> > >> >> >> > > > >>> > window > > > >> > >> >> >> > > > >>> > > > > results can be incrementally > calculated. > > In > > > >> this > > > >> > >> >> kind of > > > >> > >> >> >> > > case, > > > >> > >> >> >> > > > >>> we can > > > >> > >> >> >> > > > >>> > > > > dramatically improve the performance of > > > >> window > > > >> > >> >> >> aggregation, > > > >> > >> >> >> > > if > > > >> > >> >> >> > > > >>> > evictor > > > >> > >> >> >> > > > >>> > > > can > > > >> > >> >> >> > > > >>> > > > > trigger update of window aggregation > > state > > > by > > > >> > some > > > >> > >> >> >> > mechanism. > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > Best Wishes! > > > >> > >> >> >> > > > >>> > > > > --- > > > >> > >> >> >> > > > >>> > > > > wenlong.lwl > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > -----邮件原件----- > > > >> > >> >> >> > > > >>> > > > > 发件人: Aljoscha Krettek [mailto: > > > >> > aljos...@apache.org > > > >> > >> ] > > > >> > >> >> >> > > > >>> > > > > 发送时间: 2016年7月7日 17:32 > > > >> > >> >> >> > > > >>> > > > > 收件人: dev@flink.apache.org > > > >> > >> >> >> > > > >>> > > > > 主题: Re: [DISCUSS] Enhance Window > Evictor > > in > > > >> > Flink > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > Hi, > > > >> > >> >> >> > > > >>> > > > > regarding "sorting the window by event > > > >> time": I > > > >> > >> also > > > >> > >> >> >> > > considered > > > >> > >> >> >> > > > >>> this > > > >> > >> >> >> > > > >>> > > but > > > >> > >> >> >> > > > >>> > > > > in the end I don't think it's > necessary. > > > >> Sorting > > > >> > >> is > > > >> > >> >> >> rather > > > >> > >> >> >> > > > >>> expensive > > > >> > >> >> >> > > > >>> > > and > > > >> > >> >> >> > > > >>> > > > > making decisions based on the order of > > > >> elements > > > >> > >> can > > > >> > >> >> be > > > >> > >> >> >> > > tricky. > > > >> > >> >> >> > > > An > > > >> > >> >> >> > > > >>> > > extreme > > > >> > >> >> >> > > > >>> > > > > example of why this can be problematic > is > > > the > > > >> > case > > > >> > >> >> where > > > >> > >> >> >> > all > > > >> > >> >> >> > > > >>> elements > > > >> > >> >> >> > > > >>> > > in > > > >> > >> >> >> > > > >>> > > > > the window have the same timestamp. > Now, > > if > > > >> you > > > >> > >> >> decide > > > >> > >> >> >> to > > > >> > >> >> >> > > evict > > > >> > >> >> >> > > > >>> the > > > >> > >> >> >> > > > >>> > > > first 5 > > > >> > >> >> >> > > > >>> > > > > elements based on timestamp order you > > > >> basically > > > >> > >> >> >> arbitrarily > > > >> > >> >> >> > > > >>> evict 5 > > > >> > >> >> >> > > > >>> > > > > elements. I think the better solution > for > > > >> doing > > > >> > >> >> >> time-based > > > >> > >> >> >> > > > >>> eviction > > > >> > >> >> >> > > > >>> > is > > > >> > >> >> >> > > > >>> > > to > > > >> > >> >> >> > > > >>> > > > > do one pass over the elements to get an > > > >> overview > > > >> > >> of > > > >> > >> >> the > > > >> > >> >> >> > > > timestamp > > > >> > >> >> >> > > > >>> > > > > distribution, then do a second pass and > > > evict > > > >> > >> >> elements > > > >> > >> >> >> > based > > > >> > >> >> >> > > on > > > >> > >> >> >> > > > >>> what > > > >> > >> >> >> > > > >>> > > was > > > >> > >> >> >> > > > >>> > > > > learned in the first pass. This has > > > >> complexity > > > >> > 2*n > > > >> > >> >> >> compared > > > >> > >> >> >> > > to > > > >> > >> >> >> > > > >>> the > > > >> > >> >> >> > > > >>> > > n*log > > > >> > >> >> >> > > > >>> > > > n > > > >> > >> >> >> > > > >>> > > > > (plus the work of actually deciding > what > > to > > > >> > >> evict) of > > > >> > >> >> >> the > > > >> > >> >> >> > > sort > > > >> > >> >> >> > > > >>> based > > > >> > >> >> >> > > > >>> > > > > strategy. > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > I might be wrong, though, and there > could > > > be > > > >> a > > > >> > >> valid > > > >> > >> >> >> > use-case > > > >> > >> >> >> > > > not > > > >> > >> >> >> > > > >>> > > covered > > > >> > >> >> >> > > > >>> > > > > by the above idea. > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > regarding Vishnu's other use case of > > > evicting > > > >> > >> based > > > >> > >> >> on > > > >> > >> >> >> some > > > >> > >> >> >> > > > >>> decision > > > >> > >> >> >> > > > >>> > in > > > >> > >> >> >> > > > >>> > > > the > > > >> > >> >> >> > > > >>> > > > > WindowFunction: could this be solved by > > > doing > > > >> > the > > > >> > >> >> check > > > >> > >> >> >> for > > > >> > >> >> >> > > the > > > >> > >> >> >> > > > >>> > pattern > > > >> > >> >> >> > > > >>> > > > in > > > >> > >> >> >> > > > >>> > > > > the evictor itself instead of in the > > window > > > >> > >> function? > > > >> > >> >> >> I'm > > > >> > >> >> >> > > very > > > >> > >> >> >> > > > >>> > hesitant > > > >> > >> >> >> > > > >>> > > > to > > > >> > >> >> >> > > > >>> > > > > introduce a coupling between the > > different > > > >> > >> >> components of > > > >> > >> >> >> > the > > > >> > >> >> >> > > > >>> > windowing > > > >> > >> >> >> > > > >>> > > > > system, i.e. assigner, trigger, evictor > > and > > > >> > window > > > >> > >> >> >> > function. > > > >> > >> >> >> > > > The > > > >> > >> >> >> > > > >>> > reason > > > >> > >> >> >> > > > >>> > > > is > > > >> > >> >> >> > > > >>> > > > > that using an evictor has a huge > > > performance > > > >> > >> impact > > > >> > >> >> >> since > > > >> > >> >> >> > the > > > >> > >> >> >> > > > >>> system > > > >> > >> >> >> > > > >>> > > > always > > > >> > >> >> >> > > > >>> > > > > has to keep all elements and cannot to > > > >> > incremental > > > >> > >> >> >> > > aggregation > > > >> > >> >> >> > > > of > > > >> > >> >> >> > > > >>> > > window > > > >> > >> >> >> > > > >>> > > > > results and I therefore don't want to > put > > > >> > specific > > > >> > >> >> >> features > > > >> > >> >> >> > > > >>> regarding > > > >> > >> >> >> > > > >>> > > > > eviction into the other components. > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > Cheers, > > > >> > >> >> >> > > > >>> > > > > Aljoscha > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > On Thu, 7 Jul 2016 at 10:00 Radu > Tudoran > > < > > > >> > >> >> >> > > > >>> radu.tudo...@huawei.com> > > > >> > >> >> >> > > > >>> > > > wrote: > > > >> > >> >> >> > > > >>> > > > > > > > >> > >> >> >> > > > >>> > > > > > Hi, > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > I think the situation Vishnu raised > is > > > >> > something > > > >> > >> >> that > > > >> > >> >> >> > > should > > > >> > >> >> >> > > > be > > > >> > >> >> >> > > > >>> > > > > accounted. > > > >> > >> >> >> > > > >>> > > > > > It can happen indeed that you want to > > > >> > condition > > > >> > >> >> what > > > >> > >> >> >> you > > > >> > >> >> >> > > > evict > > > >> > >> >> >> > > > >>> from > > > >> > >> >> >> > > > >>> > > > > > the window based on the result of the > > > >> function > > > >> > >> to > > > >> > >> >> be > > > >> > >> >> >> > > applied. > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > My 2 cents... > > > >> > >> >> >> > > > >>> > > > > > I would suggest adding a list for the > > > >> elements > > > >> > >> of > > > >> > >> >> the > > > >> > >> >> >> > > stream > > > >> > >> >> >> > > > >>> where > > > >> > >> >> >> > > > >>> > > you > > > >> > >> >> >> > > > >>> > > > > > can MARK them to be delete. > > Alternatively > > > >> the > > > >> > >> >> iterator > > > >> > >> >> >> > can > > > >> > >> >> >> > > be > > > >> > >> >> >> > > > >>> > > extended > > > >> > >> >> >> > > > >>> > > > > > to have a function > > > >> > >> Iterator.markForEviction(int); > > > >> > >> >> >> These > > > >> > >> >> >> > can > > > >> > >> >> >> > > > be > > > >> > >> >> >> > > > >>> made > > > >> > >> >> >> > > > >>> > > > > > available also in the apply function. > > > >> > Moreover, > > > >> > >> we > > > >> > >> >> can > > > >> > >> >> >> > use > > > >> > >> >> >> > > > >>> this to > > > >> > >> >> >> > > > >>> > > > > > extend the functionality such that > you > > > add > > > >> > MARKs > > > >> > >> >> >> either > > > >> > >> >> >> > for > > > >> > >> >> >> > > > >>> > eviction > > > >> > >> >> >> > > > >>> > > > > > after the function has finished > > > triggering > > > >> or > > > >> > >> to be > > > >> > >> >> >> > evicted > > > >> > >> >> >> > > > in > > > >> > >> >> >> > > > >>> the > > > >> > >> >> >> > > > >>> > > next > > > >> > >> >> >> > > > >>> > > > > iteration. > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > Dr. Radu Tudoran > > > >> > >> >> >> > > > >>> > > > > > Research Engineer - Big Data Expert > > > >> > >> >> >> > > > >>> > > > > > IT R&D Division > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > > >> > >> >> >> > > > >>> > > > > > European Research Center > > > >> > >> >> >> > > > >>> > > > > > Riesstrasse 25, 80992 München > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > E-mail: radu.tudo...@huawei.com > > > >> > >> >> >> > > > >>> > > > > > Mobile: +49 15209084330 > > > >> > >> >> >> > > > >>> > > > > > Telephone: +49 891588344173 > > > >> > >> >> >> > > > >>> > > > > > > > > >> > >> >> >> > > > >>> > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > > >> > >> >> >> > > > >>> > > > > > Hansaallee 205, 40549 Düsseldorf, > > > Germany, > > > >> > >> >> >> > www.huawei.com > > > >> > >> >> >> > > > >>> > Registered > > > >> > >> >> >> > > > >>> > > > > > Office: Düsseldorf, Register Court > > > >> Düsseldorf, > > > >> > >> HRB > > > >> > >> >> >> 56063, > > > >> > >> >> >> > > > >>> Managing > > > >> > >> >> >> > > > >>> > > > > > Director: Bo PENG, Wanzhou MENG, > Lifang > > > >> CHEN > > > >> > >> Sitz > > > >> > >> >> der > > > >> > >> >> >> > > > >>> Gesellschaft: > > > >> > >> >> >> > > > >>> > > > > > Düsseldorf, Amtsgericht Düsseldorf, > HRB > > > >> 56063, > > > >> > >> >> >> > > > >>> > > > > > Geschäftsführer: Bo PENG, Wanzhou > MENG, > > > >> Lifang > > > >> > >> CHEN > > > >> > >> >> >> This > > > >> > >> >> >> > > > >>> e-mail and
