Hi,
this, in fact, seems to be a bug. There should be something like
windowState.clear();
for (IN element: projectedContents) {
windowState.add(element);
}
after passing the elements to the window function.
This is very inefficient but the only way I see of doing it right now.
Cheers,
Aljoscha
On Thu, 21 Jul 2016 at 01:32 Vishnu Viswanath <vishnu.viswanat...@gmail.com>
wrote:
> Hi,
>
> When we use RocksDB as state backend, how does the backend state get
> updated after some elements are evicted from the window?
> I don't see any update call being made to remove the element from the state
> stored in RocksDB.
>
> It looks like the RocksDBListState is only having get() and add() methods
> since it is an AppendingState, but that causes the evicted elements to come
> back when the trigger is fired next time. (It works fine when I use
> MemoryStateBackend)
>
> Is this expected behavior or am I missing something.
>
> Thanks,
> Vishnu
>
> On Mon, Jul 18, 2016 at 7:15 AM, Vishnu Viswanath <
> vishnu.viswanat...@gmail.com> wrote:
>
> > Hi Aljoscha,
> >
> > Thanks! Yes, I have the create page option now in wiki.
> >
> > Regards,
> > Vishnu Viswanath,
> >
> > On Mon, Jul 18, 2016 at 6:34 AM, Aljoscha Krettek <aljos...@apache.org>
> > wrote:
> >
> >> @Radu, addition of more window types and sorting should be part of
> another
> >> design proposal. This is interesting stuff but I think we should keep
> >> issues separated because things can get complicated very quickly.
> >>
> >> On Mon, 18 Jul 2016 at 12:32 Aljoscha Krettek <aljos...@apache.org>
> >> wrote:
> >>
> >> > Hi,
> >> > about TimeEvictor, yes, I think there should be specific evictors for
> >> > processing time and event time. Also, the current time should be
> >> > retrievable from the EvictorContext.
> >> >
> >> > For the wiki you will need permissions. This was recently changed
> >> because
> >> > there was too much spam. I gave you permission to add pages. Can you
> >> please
> >> > try and check if it works?
> >> >
> >> > Cheers,
> >> > Aljoscha
> >> >
> >> > On Fri, 15 Jul 2016 at 13:28 Vishnu Viswanath <
> >> > vishnu.viswanat...@gmail.com> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> How do we create a FLIP page, is there any permission setup
> required? I
> >> >> don't see any "Create" page(after logging in) option in the header as
> >> >> mentioned in
> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Vishnu
> >> >>
> >> >> On Wed, Jul 13, 2016 at 10:22 PM, Vishnu Viswanath <
> >> >> vishnu.viswanat...@gmail.com> wrote:
> >> >>
> >> >> > Hi Aljoscha,
> >> >> >
> >> >> > I agree, the user will know exactly that they are creating an
> >> EventTime
> >> >> > based evictor or ProcessingTime based evictor looking at the code.
> >> >> > So do you think it will be ok to have multiple versions of
> >> TimeEvictor
> >> >> > (one for event time and one for processing time) and also a
> >> DeltaEvcitor
> >> >> > (again 2 versions- for event time and processing time) ?
> >> >> >
> >> >> > Please note that the existing behavior of TimeEvictor/DeltaEvictor
> >> does
> >> >> > not consider if it is EventTime or ProcessingTime
> >> >> > e.g., in TimeEvictor the current time is considered as the
> timestamp
> >> of
> >> >> > the last element in the window
> >> >> >
> >> >> > *long currentTime = Iterables.getLast(elements).getTimestamp();*
> >> >> >
> >> >> > not the highest timestamp of all elements
> >> >> > what I am trying to achieve is something like:
> >> >> >
> >> >> > *long currentTime;*
> >> >> > * if (ctx.isEventTime()) {*
> >> >> > * currentTime = getMaxTimestamp(elements);*
> >> >> > * } else {*
> >> >> > * currentTime = Iterables.getLast(elements).getTimestamp();*
> >> >> > * }*
> >> >> >
> >> >> > Similarly, in DeltaEvictor the *`lastElement`* is
> >> >> > *`Iterables.getLast(elements);`* and I am thinking we should
> consider
> >> >> the
> >> >> > element with max timestamp as the last element instead of just
> >> getting
> >> >> the
> >> >> > last inserted element as *`lastElement`*
> >> >> >
> >> >> > Do you think it is the right thing to do or leave the behavior
> >> Evictors
> >> >> as
> >> >> > is, w.r.t to choosing the last element?
> >> >> >
> >> >> > Thanks,
> >> >> > Vishnu
> >> >> >
> >> >> > On Wed, Jul 13, 2016 at 11:07 AM, Aljoscha Krettek <
> >> aljos...@apache.org
> >> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> I still think it should be explicit in the class. For example, if
> >> you
> >> >> have
> >> >> >> this code:
> >> >> >>
> >> >> >> input
> >> >> >> .keyBy()
> >> >> >> .window()
> >> >> >> .trigger(EventTimeTrigger.create())
> >> >> >> .evictor(TimeTrigger.create())
> >> >> >>
> >> >> >> the time behavior of the trigger is explicitly specified while the
> >> >> evictor
> >> >> >> would dynamically adapt based on internal workings that the user
> >> might
> >> >> not
> >> >> >> be aware of. Having the behavior explicit at the call site is very
> >> >> >> important, in my opinion.
> >> >> >>
> >> >> >> On Wed, 13 Jul 2016 at 16:28 Vishnu Viswanath <
> >> >> >> vishnu.viswanat...@gmail.com>
> >> >> >> wrote:
> >> >> >>
> >> >> >> > Hi,
> >> >> >> >
> >> >> >> > I was hoping to use the isEventTime method in the WindowAssigner
> >> to
> >> >> set
> >> >> >> > that information in the EvictorContext.
> >> >> >> > What do you think?.
> >> >> >> >
> >> >> >> > Thanks and Regards,
> >> >> >> > Vishnu Viswanath,
> >> >> >> >
> >> >> >> > On Wed, Jul 13, 2016 at 10:09 AM, Aljoscha Krettek <
> >> >> aljos...@apache.org
> >> >> >> >
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> > > Hi,
> >> >> >> > > I think the way to go here is to add both an EventTimeEvictor
> >> and a
> >> >> >> > > ProcessingTimeEvictor. The problem is that "isEventTime"
> cannot
> >> >> >> really be
> >> >> >> > > determined. That's also the reason why there is an
> >> EventTimeTrigger
> >> >> >> and a
> >> >> >> > > ProcessingTimeTrigger. It was just an oversight that the
> >> >> TimeEvictor
> >> >> >> does
> >> >> >> > > not also have these two versions.
> >> >> >> > >
> >> >> >> > > About EvictingWindowOperator, I think you can make the two
> >> methods
> >> >> >> > > non-final in WindowOperator, yes.
> >> >> >> > >
> >> >> >> > > Cheers,
> >> >> >> > > Aljoscha
> >> >> >> > >
> >> >> >> > > On Wed, 13 Jul 2016 at 14:32 Vishnu Viswanath <
> >> >> >> > > vishnu.viswanat...@gmail.com>
> >> >> >> > > wrote:
> >> >> >> > >
> >> >> >> > > > Hi Aljoscha,
> >> >> >> > > >
> >> >> >> > > > I am thinking of adding a method boolean isEventTime(); in
> the
> >> >> >> > > > EvictorContext apart from
> >> >> >> > > >
> >> >> >> > > > long getCurrentProcessingTime();
> >> >> >> > > > MetricGroup getMetricGroup();
> >> >> >> > > > long getCurrentWatermark();
> >> >> >> > > >
> >> >> >> > > > This method can be used to make the Evictor not iterate
> >> through
> >> >> all
> >> >> >> the
> >> >> >> > > > elements in TimeEvictor. There will be a few changes in the
> >> >> existing
> >> >> >> > > > behavior of TimeEvictor and DeltaEvictor (I have mentioned
> >> this
> >> >> in
> >> >> >> the
> >> >> >> > > > design doc)
> >> >> >> > > >
> >> >> >> > > > Also, is there any specific reason why the open and close
> >> method
> >> >> in
> >> >> >> > > > WindowEvictor is made final? Since the EvictorContext will
> be
> >> in
> >> >> the
> >> >> >> > > > EvictingWindowOperator, I need to override the open and
> close
> >> in
> >> >> >> > > > EvitingWindowOperator to make the reference of
> EvictorContext
> >> >> null.
> >> >> >> > > >
> >> >> >> > > > Thanks and Regards,
> >> >> >> > > > Vishnu Viswanath,
> >> >> >> > > >
> >> >> >> > > > On Fri, Jul 8, 2016 at 7:40 PM, Vishnu Viswanath <
> >> >> >> > > > vishnu.viswanat...@gmail.com> wrote:
> >> >> >> > > >
> >> >> >> > > > My thought process when asking if we can use state backend
> in
> >> >> window
> >> >> >> > > > > function was : can we add the elements to be evicted into
> >> some
> >> >> >> state
> >> >> >> > > and
> >> >> >> > > > > allow the evictAfter to read it from some context and
> >> remove it
> >> >> >> from
> >> >> >> > > the
> >> >> >> > > > > window?
> >> >> >> > > > >
> >> >> >> > > > >
> >> >> >> > > > > On Fri, Jul 8, 2016 at 7:30 PM, Vishnu Viswanath <
> >> >> >> > > > > vishnu.viswanat...@gmail.com> wrote:
> >> >> >> > > > >
> >> >> >> > > > >> Hi Aljoscha,
> >> >> >> > > > >>
> >> >> >> > > > >> Thanks for the explanation, and sorry for late reply was
> >> busy
> >> >> >> with
> >> >> >> > > work.
> >> >> >> > > > >>
> >> >> >> > > > >> I did think about this scenario, in fact in my previous
> >> mail I
> >> >> >> > thought
> >> >> >> > > > of
> >> >> >> > > > >> posting this question, then I understood that this
> problem
> >> >> will
> >> >> >> be
> >> >> >> > > > >> there which ever method we choose(Trigger looking for
> >> pattern
> >> >> or
> >> >> >> > > Window
> >> >> >> > > > >> looking for pattern).
> >> >> >> > > > >>
> >> >> >> > > > >> I do have a pretty good watermark but my concern is that
> it
> >> >> >> changes
> >> >> >> > > > based
> >> >> >> > > > >> on the key of these messages(I don't know if it is
> >> possible,
> >> >> >> haven't
> >> >> >> > > > >> started coding that yet. May be you could tell me). Even
> if
> >> >> it is
> >> >> >> > yes
> >> >> >> > > > some
> >> >> >> > > > >> of these watermarks will be long(in days), which I don't
> >> want
> >> >> the
> >> >> >> > > > trigger
> >> >> >> > > > >> to wait that long.
> >> >> >> > > > >>
> >> >> >> > > > >> It looks like it is not easy to have an evictAfter based
> on
> >> >> >> window
> >> >> >> > > > >> function(without introducing coupling), but can the
> current
> >> >> >> window
> >> >> >> > > apply
> >> >> >> > > > >> function be modified to allow it to change the elements
> in
> >> it
> >> >> -
> >> >> >> may
> >> >> >> > be
> >> >> >> > > > >> using some state backend(I don't know how excatly the
> >> >> internals
> >> >> >> of
> >> >> >> > > these
> >> >> >> > > > >> work, so this might be a wrong question)
> >> >> >> > > > >>
> >> >> >> > > > >> Thanks and Regards,
> >> >> >> > > > >> Vishnu Viswanath,
> >> >> >> > > > >>
> >> >> >> > > > >> On Fri, Jul 8, 2016 at 10:20 AM, Aljoscha Krettek <
> >> >> >> > > aljos...@apache.org>
> >> >> >> > > > >> wrote:
> >> >> >> > > > >>
> >> >> >> > > > >>> Hi Vishnu,
> >> >> >> > > > >>> how long would these patterns be? The Trigger would not
> >> have
> >> >> to
> >> >> >> > sort
> >> >> >> > > > the
> >> >> >> > > > >>> elements for every new element but just insert the new
> >> >> element
> >> >> >> into
> >> >> >> > > an
> >> >> >> > > > >>> internal data structure. Only when it sees that the
> >> >> watermark is
> >> >> >> > > past a
> >> >> >> > > > >>> certain point would it check whether the pattern matches
> >> and
> >> >> >> > actually
> >> >> >> > > > >>> Trigger.
> >> >> >> > > > >>>
> >> >> >> > > > >>> A general note regarding order and event time: I think
> >> >> relying
> >> >> >> on
> >> >> >> > > this
> >> >> >> > > > >>> for
> >> >> >> > > > >>> computation is very tricky unless the watermark is 100 %
> >> >> >> correct or
> >> >> >> > > you
> >> >> >> > > > >>> completely discard elements that arrive after the
> >> watermark,
> >> >> >> i.e.
> >> >> >> > > > >>> elements
> >> >> >> > > > >>> that would break the promise of the watermark that no
> >> >> elements
> >> >> >> with
> >> >> >> > > an
> >> >> >> > > > >>> earlier timestamp will ever arrive. The reason for this
> is
> >> >> that
> >> >> >> > there
> >> >> >> > > > >>> could
> >> >> >> > > > >>> always enter new elements that end up between already
> seen
> >> >> >> > elements.
> >> >> >> > > > For
> >> >> >> > > > >>> example, let's say we have this sequence of elements
> when
> >> the
> >> >> >> > trigger
> >> >> >> > > > >>> fires:
> >> >> >> > > > >>>
> >> >> >> > > > >>> a-b-a
> >> >> >> > > > >>>
> >> >> >> > > > >>> This is the sequence that you are looking for and you
> emit
> >> >> some
> >> >> >> > > result
> >> >> >> > > > >>> from
> >> >> >> > > > >>> the WindowFunction. Now, new elements arrive that fall
> in
> >> >> >> between
> >> >> >> > the
> >> >> >> > > > >>> elements we already have:
> >> >> >> > > > >>>
> >> >> >> > > > >>> a-d-e-b-f-g-a
> >> >> >> > > > >>>
> >> >> >> > > > >>> This is an updated, sorted view of the actual event-time
> >> >> stream
> >> >> >> and
> >> >> >> > > we
> >> >> >> > > > >>> didn't realize that the stream actually looks like this
> >> >> before.
> >> >> >> > Does
> >> >> >> > > > this
> >> >> >> > > > >>> still match the original pattern or should we now
> consider
> >> >> this
> >> >> >> as
> >> >> >> > > > >>> non-matching? If no, then the earlier successful match
> for
> >> >> a-b-a
> >> >> >> > was
> >> >> >> > > > >>> wrong
> >> >> >> > > > >>> and we should never have processed it but we didn't know
> >> at
> >> >> the
> >> >> >> > time.
> >> >> >> > > > If
> >> >> >> > > > >>> yes, then pattern matching like this can be done in the
> >> >> Trigger
> >> >> >> by
> >> >> >> > > > having
> >> >> >> > > > >>> something like pattern slots: You don't have to store
> all
> >> >> >> elements
> >> >> >> > in
> >> >> >> > > > the
> >> >> >> > > > >>> Trigger, you just need to store possible candidates that
> >> >> could
> >> >> >> > match
> >> >> >> > > > the
> >> >> >> > > > >>> pattern and ignore the other (in-between) elements.
> >> >> >> > > > >>>
> >> >> >> > > > >>> Cheers,
> >> >> >> > > > >>> Aljoscha
> >> >> >> > > > >>>
> >> >> >> > > > >>> On Fri, 8 Jul 2016 at 14:10 Vishnu Viswanath <
> >> >> >> > > > >>> vishnu.viswanat...@gmail.com>
> >> >> >> > > > >>> wrote:
> >> >> >> > > > >>>
> >> >> >> > > > >>> > Hi Aljoscha,
> >> >> >> > > > >>> >
> >> >> >> > > > >>> > That is a good idea, trying to tie it back to the use
> >> case,
> >> >> >> > > > >>> > e.g., suppose trigger is looking for a pattern, a-b-a
> >> and
> >> >> >> when it
> >> >> >> > > > sees
> >> >> >> > > > >>> such
> >> >> >> > > > >>> > a pattern, it will trigger the window and it knows
> that
> >> now
> >> >> >> the
> >> >> >> > > > >>> Evictor is
> >> >> >> > > > >>> > going to evict the element b, and trigger updates its
> >> >> state as
> >> >> >> > a-a
> >> >> >> > > > >>> (even
> >> >> >> > > > >>> > before the window & evictor completes) and will be
> >> looking
> >> >> for
> >> >> >> > the
> >> >> >> > > > >>> rest of
> >> >> >> > > > >>> > the pattern i.e., b-a. But I can think of 1 problem
> >> here,
> >> >> >> > > > >>> >
> >> >> >> > > > >>> > - the events can arrive out of order, i.e., the
> >> trigger
> >> >> >> might
> >> >> >> > be
> >> >> >> > > > >>> seeing
> >> >> >> > > > >>> > a pattern a-a-b but actual event time is a-b-a then
> >> >> trigger
> >> >> >> > will
> >> >> >> > > > >>> have to
> >> >> >> > > > >>> > sort the elements in the window everytime it sees
> an
> >> >> >> element.
> >> >> >> > (I
> >> >> >> > > > was
> >> >> >> > > > >>> > planning to do this sorting in the window, which
> >> will be
> >> >> >> less
> >> >> >> > > > often
> >> >> >> > > > >>> -
> >> >> >> > > > >>> > only
> >> >> >> > > > >>> > when the trigger fires)
> >> >> >> > > > >>> >
> >> >> >> > > > >>> > Thanks and Regards,
> >> >> >> > > > >>> > Vishnu Viswanath,
> >> >> >> > > > >>> >
> >> >> >> > > > >>> > On Fri, Jul 8, 2016 at 6:04 AM, Aljoscha Krettek <
> >> >> >> > > > aljos...@apache.org>
> >> >> >> > > > >>> > wrote:
> >> >> >> > > > >>> >
> >> >> >> > > > >>> > Hi,
> >> >> >> > > > >>> > > come to think of it, the right place to put such
> >> checks
> >> >> is
> >> >> >> > > actually
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > Trigger. It would have to be a custom trigger that
> >> >> observes
> >> >> >> > time
> >> >> >> > > > but
> >> >> >> > > > >>> also
> >> >> >> > > > >>> > > keeps some internal state machine to decide when it
> >> has
> >> >> >> > observed
> >> >> >> > > > the
> >> >> >> > > > >>> > right
> >> >> >> > > > >>> > > pattern in the window. Then the window function
> would
> >> >> just
> >> >> >> have
> >> >> >> > > to
> >> >> >> > > > >>> do the
> >> >> >> > > > >>> > > processing and you have good separation of concerns.
> >> Does
> >> >> >> that
> >> >> >> > > make
> >> >> >> > > > >>> > sense?
> >> >> >> > > > >>> > >
> >> >> >> > > > >>> > > I'm ignoring time and sorting by time for now
> because
> >> we
> >> >> >> > probably
> >> >> >> > > > >>> need
> >> >> >> > > > >>> > > another design document for that. To me it seems
> like
> >> a
> >> >> >> bigger
> >> >> >> > > > thing.
> >> >> >> > > > >>> > >
> >> >> >> > > > >>> > > Cheers,
> >> >> >> > > > >>> > > Aljoscha
> >> >> >> > > > >>> > >
> >> >> >> > > > >>> > > On Thu, 7 Jul 2016 at 23:56 Vishnu Viswanath <
> >> >> >> > > > >>> > vishnu.viswanat...@gmail.com
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > wrote:
> >> >> >> > > > >>> > >
> >> >> >> > > > >>> > > > Hi,
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > > Regarding the evictAfter function, that evicts
> >> based on
> >> >> >> some
> >> >> >> > > > >>> decision
> >> >> >> > > > >>> > > made
> >> >> >> > > > >>> > > > by the window function: I think it will be nice
> if
> >> we
> >> >> can
> >> >> >> > come
> >> >> >> > > > up
> >> >> >> > > > >>> with
> >> >> >> > > > >>> > > > something that is LESS coupled, because I can
> think
> >> of
> >> >> >> > several
> >> >> >> > > > use
> >> >> >> > > > >>> > cases
> >> >> >> > > > >>> > > > that depend on it.
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > > Especially in the case where there are late
> arriving
> >> >> >> > messages.
> >> >> >> > > > Only
> >> >> >> > > > >>> > after
> >> >> >> > > > >>> > > > the window function is applied we could tell what
> >> to do
> >> >> >> with
> >> >> >> > > the
> >> >> >> > > > >>> > elements
> >> >> >> > > > >>> > > > in the window. You could apply your business logic
> >> >> there
> >> >> >> to
> >> >> >> > > > >>> determine
> >> >> >> > > > >>> > if
> >> >> >> > > > >>> > > > the window funciton was able to do what it is
> >> supposed
> >> >> to
> >> >> >> do,
> >> >> >> > > if
> >> >> >> > > > >>> yes
> >> >> >> > > > >>> > > evict
> >> >> >> > > > >>> > > > those elements, else(since the elements you are
> >> looking
> >> >> >> for
> >> >> >> > > > haven't
> >> >> >> > > > >>> > > arrived
> >> >> >> > > > >>> > > > yet) wait and try again when the trigger gets
> fired
> >> >> next
> >> >> >> > time.
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > > Thanks and Regards,
> >> >> >> > > > >>> > > > Vishnu Viswanath,
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > > On Thu, Jul 7, 2016 at 9:19 AM, Radu Tudoran <
> >> >> >> > > > >>> radu.tudo...@huawei.com>
> >> >> >> > > > >>> > > > wrote:
> >> >> >> > > > >>> > > >
> >> >> >> > > > >>> > > > > Hi,
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > @Aljoscha - I can understand the reason why you
> >> are
> >> >> >> > hesitant
> >> >> >> > > to
> >> >> >> > > > >>> > > introduce
> >> >> >> > > > >>> > > > > "slower" windows such as the ones that would
> >> maintain
> >> >> >> > sorted
> >> >> >> > > > >>> items or
> >> >> >> > > > >>> > > > > windows with bindings between the different
> >> entities
> >> >> >> > > (evictor,
> >> >> >> > > > >>> > trigger,
> >> >> >> > > > >>> > > > > window, apply function). However, I think it's
> >> >> possible
> >> >> >> > just
> >> >> >> > > to
> >> >> >> > > > >>> > create
> >> >> >> > > > >>> > > > more
> >> >> >> > > > >>> > > > > types of windows. The existing ones
> (timewindows,
> >> >> global
> >> >> >> > > > windows
> >> >> >> > > > >>> ...)
> >> >> >> > > > >>> > > can
> >> >> >> > > > >>> > > > > remain, and just add some more flavors of
> windows
> >> >> were
> >> >> >> more
> >> >> >> > > > >>> features
> >> >> >> > > > >>> > > are
> >> >> >> > > > >>> > > > > enabled or more functionality (e.g., access to
> the
> >> >> each
> >> >> >> > > element
> >> >> >> > > > >>> in
> >> >> >> > > > >>> > the
> >> >> >> > > > >>> > > > > evictor ; possibility to delete or mark for
> >> eviction
> >> >> >> > elements
> >> >> >> > > > in
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > > > function...)
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > Regarding the specific case of sorted windows, I
> >> >> think
> >> >> >> the
> >> >> >> > N
> >> >> >> > > > lon
> >> >> >> > > > >>> N
> >> >> >> > > > >>> > > > > complexity to sort (the worst case) is very
> >> >> unlikely. In
> >> >> >> > fact
> >> >> >> > > > you
> >> >> >> > > > >>> > have
> >> >> >> > > > >>> > > > > almost sorted items/arrays. Moreover, if you
> >> consider
> >> >> >> that
> >> >> >> > in
> >> >> >> > > > >>> > > iteration X
> >> >> >> > > > >>> > > > > all elements were sorted, then in iteration X+1
> >> you
> >> >> will
> >> >> >> > need
> >> >> >> > > > to
> >> >> >> > > > >>> sort
> >> >> >> > > > >>> > > > just
> >> >> >> > > > >>> > > > > the newly arrived elements (M). I would expect
> >> that
> >> >> this
> >> >> >> > > > number M
> >> >> >> > > > >>> > might
> >> >> >> > > > >>> > > > be
> >> >> >> > > > >>> > > > > significant smaller then N (elements that
> exists).
> >> >> Then
> >> >> >> > using
> >> >> >> > > > an
> >> >> >> > > > >>> > > > insertion
> >> >> >> > > > >>> > > > > sort for these new elements you would have M *
> N
> >> >> >> > complexity
> >> >> >> > > > and
> >> >> >> > > > >>> if
> >> >> >> > > > >>> > > M<< N
> >> >> >> > > > >>> > > > > then the complexity is O(N). Alternatively you
> can
> >> >> use a
> >> >> >> > > binary
> >> >> >> > > > >>> > search
> >> >> >> > > > >>> > > > for
> >> >> >> > > > >>> > > > > insertion and then you further reduce the
> >> complexity
> >> >> to
> >> >> >> > > > O(logN).
> >> >> >> > > > >>> > > > > If M is proportional to N then you can sort M
> and
> >> use
> >> >> >> merge
> >> >> >> > > > sort
> >> >> >> > > > >>> for
> >> >> >> > > > >>> > > > > combining.
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > Dr. Radu Tudoran
> >> >> >> > > > >>> > > > > Research Engineer - Big Data Expert
> >> >> >> > > > >>> > > > > IT R&D Division
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> >> >> >> > > > >>> > > > > European Research Center
> >> >> >> > > > >>> > > > > Riesstrasse 25, 80992 München
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > E-mail: radu.tudo...@huawei.com
> >> >> >> > > > >>> > > > > Mobile: +49 15209084330
> >> >> >> > > > >>> > > > > Telephone: +49 891588344173
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> >> >> >> > > > >>> > > > > Hansaallee 205, 40549 Düsseldorf, Germany,
> >> >> >> www.huawei.com
> >> >> >> > > > >>> > > > > Registered Office: Düsseldorf, Register Court
> >> >> >> Düsseldorf,
> >> >> >> > HRB
> >> >> >> > > > >>> 56063,
> >> >> >> > > > >>> > > > > Managing Director: Bo PENG, Wanzhou MENG, Lifang
> >> CHEN
> >> >> >> > > > >>> > > > > Sitz der Gesellschaft: Düsseldorf, Amtsgericht
> >> >> >> Düsseldorf,
> >> >> >> > > HRB
> >> >> >> > > > >>> 56063,
> >> >> >> > > > >>> > > > > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang
> >> CHEN
> >> >> >> > > > >>> > > > > This e-mail and its attachments contain
> >> confidential
> >> >> >> > > > information
> >> >> >> > > > >>> from
> >> >> >> > > > >>> > > > > HUAWEI, which is intended only for the person or
> >> >> entity
> >> >> >> > whose
> >> >> >> > > > >>> address
> >> >> >> > > > >>> > > is
> >> >> >> > > > >>> > > > > listed above. Any use of the information
> contained
> >> >> >> herein
> >> >> >> > in
> >> >> >> > > > any
> >> >> >> > > > >>> way
> >> >> >> > > > >>> > > > > (including, but not limited to, total or partial
> >> >> >> > disclosure,
> >> >> >> > > > >>> > > > reproduction,
> >> >> >> > > > >>> > > > > or dissemination) by persons other than the
> >> intended
> >> >> >> > > > >>> recipient(s) is
> >> >> >> > > > >>> > > > > prohibited. If you receive this e-mail in error,
> >> >> please
> >> >> >> > > notify
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > sender
> >> >> >> > > > >>> > > > > by phone or email immediately and delete it!
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > -----Original Message-----
> >> >> >> > > > >>> > > > > From: 吕文龙(吕文龙) [mailto:
> >> wenlong....@alibaba-inc.com]
> >> >> >> > > > >>> > > > > Sent: Thursday, July 07, 2016 11:59 AM
> >> >> >> > > > >>> > > > > To: dev@flink.apache.org
> >> >> >> > > > >>> > > > > Subject: 答复: [DISCUSS] Enhance Window Evictor in
> >> >> Flink
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > HI,
> >> >> >> > > > >>> > > > > I think it is necessary to support sorted
> window,
> >> >> which
> >> >> >> can
> >> >> >> > > > avoid
> >> >> >> > > > >>> > > > scanning
> >> >> >> > > > >>> > > > > all the elements of window while trying to
> >> evicting
> >> >> >> > element,
> >> >> >> > > > >>> which
> >> >> >> > > > >>> > may
> >> >> >> > > > >>> > > > cost
> >> >> >> > > > >>> > > > > many IO operations, such as querying DBs to get
> >> >> elements
> >> >> >> > from
> >> >> >> > > > >>> state.
> >> >> >> > > > >>> > > > > What's more, when an window aggregation function
> >> is
> >> >> >> > > invertible,
> >> >> >> > > > >>> such
> >> >> >> > > > >>> > as
> >> >> >> > > > >>> > > > > sum, which can be updated by adding or removing
> a
> >> >> single
> >> >> >> > > > record,
> >> >> >> > > > >>> > window
> >> >> >> > > > >>> > > > > results can be incrementally calculated. In this
> >> >> kind of
> >> >> >> > > case,
> >> >> >> > > > >>> we can
> >> >> >> > > > >>> > > > > dramatically improve the performance of window
> >> >> >> aggregation,
> >> >> >> > > if
> >> >> >> > > > >>> > evictor
> >> >> >> > > > >>> > > > can
> >> >> >> > > > >>> > > > > trigger update of window aggregation state by
> some
> >> >> >> > mechanism.
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > Best Wishes!
> >> >> >> > > > >>> > > > > ---
> >> >> >> > > > >>> > > > > wenlong.lwl
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > -----邮件原件-----
> >> >> >> > > > >>> > > > > 发件人: Aljoscha Krettek [mailto:
> aljos...@apache.org
> >> ]
> >> >> >> > > > >>> > > > > 发送时间: 2016年7月7日 17:32
> >> >> >> > > > >>> > > > > 收件人: dev@flink.apache.org
> >> >> >> > > > >>> > > > > 主题: Re: [DISCUSS] Enhance Window Evictor in
> Flink
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > Hi,
> >> >> >> > > > >>> > > > > regarding "sorting the window by event time": I
> >> also
> >> >> >> > > considered
> >> >> >> > > > >>> this
> >> >> >> > > > >>> > > but
> >> >> >> > > > >>> > > > > in the end I don't think it's necessary. Sorting
> >> is
> >> >> >> rather
> >> >> >> > > > >>> expensive
> >> >> >> > > > >>> > > and
> >> >> >> > > > >>> > > > > making decisions based on the order of elements
> >> can
> >> >> be
> >> >> >> > > tricky.
> >> >> >> > > > An
> >> >> >> > > > >>> > > extreme
> >> >> >> > > > >>> > > > > example of why this can be problematic is the
> case
> >> >> where
> >> >> >> > all
> >> >> >> > > > >>> elements
> >> >> >> > > > >>> > > in
> >> >> >> > > > >>> > > > > the window have the same timestamp. Now, if you
> >> >> decide
> >> >> >> to
> >> >> >> > > evict
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > > first 5
> >> >> >> > > > >>> > > > > elements based on timestamp order you basically
> >> >> >> arbitrarily
> >> >> >> > > > >>> evict 5
> >> >> >> > > > >>> > > > > elements. I think the better solution for doing
> >> >> >> time-based
> >> >> >> > > > >>> eviction
> >> >> >> > > > >>> > is
> >> >> >> > > > >>> > > to
> >> >> >> > > > >>> > > > > do one pass over the elements to get an overview
> >> of
> >> >> the
> >> >> >> > > > timestamp
> >> >> >> > > > >>> > > > > distribution, then do a second pass and evict
> >> >> elements
> >> >> >> > based
> >> >> >> > > on
> >> >> >> > > > >>> what
> >> >> >> > > > >>> > > was
> >> >> >> > > > >>> > > > > learned in the first pass. This has complexity
> 2*n
> >> >> >> compared
> >> >> >> > > to
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > n*log
> >> >> >> > > > >>> > > > n
> >> >> >> > > > >>> > > > > (plus the work of actually deciding what to
> >> evict) of
> >> >> >> the
> >> >> >> > > sort
> >> >> >> > > > >>> based
> >> >> >> > > > >>> > > > > strategy.
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > I might be wrong, though, and there could be a
> >> valid
> >> >> >> > use-case
> >> >> >> > > > not
> >> >> >> > > > >>> > > covered
> >> >> >> > > > >>> > > > > by the above idea.
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > regarding Vishnu's other use case of evicting
> >> based
> >> >> on
> >> >> >> some
> >> >> >> > > > >>> decision
> >> >> >> > > > >>> > in
> >> >> >> > > > >>> > > > the
> >> >> >> > > > >>> > > > > WindowFunction: could this be solved by doing
> the
> >> >> check
> >> >> >> for
> >> >> >> > > the
> >> >> >> > > > >>> > pattern
> >> >> >> > > > >>> > > > in
> >> >> >> > > > >>> > > > > the evictor itself instead of in the window
> >> function?
> >> >> >> I'm
> >> >> >> > > very
> >> >> >> > > > >>> > hesitant
> >> >> >> > > > >>> > > > to
> >> >> >> > > > >>> > > > > introduce a coupling between the different
> >> >> components of
> >> >> >> > the
> >> >> >> > > > >>> > windowing
> >> >> >> > > > >>> > > > > system, i.e. assigner, trigger, evictor and
> window
> >> >> >> > function.
> >> >> >> > > > The
> >> >> >> > > > >>> > reason
> >> >> >> > > > >>> > > > is
> >> >> >> > > > >>> > > > > that using an evictor has a huge performance
> >> impact
> >> >> >> since
> >> >> >> > the
> >> >> >> > > > >>> system
> >> >> >> > > > >>> > > > always
> >> >> >> > > > >>> > > > > has to keep all elements and cannot to
> incremental
> >> >> >> > > aggregation
> >> >> >> > > > of
> >> >> >> > > > >>> > > window
> >> >> >> > > > >>> > > > > results and I therefore don't want to put
> specific
> >> >> >> features
> >> >> >> > > > >>> regarding
> >> >> >> > > > >>> > > > > eviction into the other components.
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > Cheers,
> >> >> >> > > > >>> > > > > Aljoscha
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > On Thu, 7 Jul 2016 at 10:00 Radu Tudoran <
> >> >> >> > > > >>> radu.tudo...@huawei.com>
> >> >> >> > > > >>> > > > wrote:
> >> >> >> > > > >>> > > > >
> >> >> >> > > > >>> > > > > > Hi,
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > I think the situation Vishnu raised is
> something
> >> >> that
> >> >> >> > > should
> >> >> >> > > > be
> >> >> >> > > > >>> > > > > accounted.
> >> >> >> > > > >>> > > > > > It can happen indeed that you want to
> condition
> >> >> what
> >> >> >> you
> >> >> >> > > > evict
> >> >> >> > > > >>> from
> >> >> >> > > > >>> > > > > > the window based on the result of the function
> >> to
> >> >> be
> >> >> >> > > applied.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > My 2 cents...
> >> >> >> > > > >>> > > > > > I would suggest adding a list for the elements
> >> of
> >> >> the
> >> >> >> > > stream
> >> >> >> > > > >>> where
> >> >> >> > > > >>> > > you
> >> >> >> > > > >>> > > > > > can MARK them to be delete. Alternatively the
> >> >> iterator
> >> >> >> > can
> >> >> >> > > be
> >> >> >> > > > >>> > > extended
> >> >> >> > > > >>> > > > > > to have a function
> >> Iterator.markForEviction(int);
> >> >> >> These
> >> >> >> > can
> >> >> >> > > > be
> >> >> >> > > > >>> made
> >> >> >> > > > >>> > > > > > available also in the apply function.
> Moreover,
> >> we
> >> >> can
> >> >> >> > use
> >> >> >> > > > >>> this to
> >> >> >> > > > >>> > > > > > extend the functionality such that you add
> MARKs
> >> >> >> either
> >> >> >> > for
> >> >> >> > > > >>> > eviction
> >> >> >> > > > >>> > > > > > after the function has finished triggering or
> >> to be
> >> >> >> > evicted
> >> >> >> > > > in
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > next
> >> >> >> > > > >>> > > > > iteration.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > Dr. Radu Tudoran
> >> >> >> > > > >>> > > > > > Research Engineer - Big Data Expert
> >> >> >> > > > >>> > > > > > IT R&D Division
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> >> >> >> > > > >>> > > > > > European Research Center
> >> >> >> > > > >>> > > > > > Riesstrasse 25, 80992 München
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > E-mail: radu.tudo...@huawei.com
> >> >> >> > > > >>> > > > > > Mobile: +49 15209084330
> >> >> >> > > > >>> > > > > > Telephone: +49 891588344173
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> >> >> >> > > > >>> > > > > > Hansaallee 205, 40549 Düsseldorf, Germany,
> >> >> >> > www.huawei.com
> >> >> >> > > > >>> > Registered
> >> >> >> > > > >>> > > > > > Office: Düsseldorf, Register Court Düsseldorf,
> >> HRB
> >> >> >> 56063,
> >> >> >> > > > >>> Managing
> >> >> >> > > > >>> > > > > > Director: Bo PENG, Wanzhou MENG, Lifang CHEN
> >> Sitz
> >> >> der
> >> >> >> > > > >>> Gesellschaft:
> >> >> >> > > > >>> > > > > > Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
> >> >> >> > > > >>> > > > > > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang
> >> CHEN
> >> >> >> This
> >> >> >> > > > >>> e-mail and
> >> >> >> > > > >>> > > > > > its attachments contain confidential
> information
> >> >> from
> >> >> >> > > HUAWEI,
> >> >> >> > > > >>> which
> >> >> >> > > > >>> > > is
> >> >> >> > > > >>> > > > > > intended only for the person or entity whose
> >> >> address
> >> >> >> is
> >> >> >> > > > listed
> >> >> >> > > > >>> > above.
> >> >> >> > > > >>> > > > > > Any use of the information contained herein in
> >> any
> >> >> way
> >> >> >> > > > >>> (including,
> >> >> >> > > > >>> > > but
> >> >> >> > > > >>> > > > > > not limited to, total or partial disclosure,
> >> >> >> > reproduction,
> >> >> >> > > or
> >> >> >> > > > >>> > > > > > dissemination) by persons other than the
> >> intended
> >> >> >> > > > recipient(s)
> >> >> >> > > > >>> is
> >> >> >> > > > >>> > > > > > prohibited. If you receive this e-mail in
> error,
> >> >> >> please
> >> >> >> > > > notify
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > > > > > sender by phone or email immediately and
> delete
> >> it!
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > -----Original Message-----
> >> >> >> > > > >>> > > > > > From: Vishnu Viswanath [mailto:
> >> >> >> > > vishnu.viswanat...@gmail.com]
> >> >> >> > > > >>> > > > > > Sent: Thursday, July 07, 2016 1:28 AM
> >> >> >> > > > >>> > > > > > To: Dev
> >> >> >> > > > >>> > > > > > Subject: Re: [DISCUSS] Enhance Window Evictor
> in
> >> >> Flink
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > Thank you Maxim and Aljoscha.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > Yes the beforeEvict and afterEvict should able
> >> >> address
> >> >> >> > > point
> >> >> >> > > > 3.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > I have one more use case in my mind (which I
> >> might
> >> >> >> have
> >> >> >> > to
> >> >> >> > > do
> >> >> >> > > > >>> in
> >> >> >> > > > >>> > the
> >> >> >> > > > >>> > > > > > later stages of POC).
> >> >> >> > > > >>> > > > > > What if the `evictAfter` should behave
> >> differently
> >> >> >> based
> >> >> >> > on
> >> >> >> > > > the
> >> >> >> > > > >>> > > window
> >> >> >> > > > >>> > > > > > function.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > For example.
> >> >> >> > > > >>> > > > > > I have a window that got triggered and my
> evict
> >> >> >> function
> >> >> >> > is
> >> >> >> > > > >>> being
> >> >> >> > > > >>> > > > > > called after the apply function. In such
> cases I
> >> >> >> should
> >> >> >> > be
> >> >> >> > > > >>> able to
> >> >> >> > > > >>> > > > > > decide on what I should evict based on the
> >> window
> >> >> >> > function.
> >> >> >> > > > >>> > > > > > e.g.,
> >> >> >> > > > >>> > > > > > let the window have elements of type `case
> class
> >> >> >> Item(id:
> >> >> >> > > > >>> String,
> >> >> >> > > > >>> > > type:
> >> >> >> > > > >>> > > > > > String)` and let the types be `type1` and
> >> `type2`.
> >> >> >> > > > >>> > > > > > If window function is able to find a sequence
> :
> >> >> `type1
> >> >> >> > > type2
> >> >> >> > > > >>> > type1`,
> >> >> >> > > > >>> > > > > > then evict all elements of the type type2.
> >> >> >> > > > >>> > > > > > or if the window function is able to find a
> >> >> sequence
> >> >> >> > `type2
> >> >> >> > > > >>> type2
> >> >> >> > > > >>> > > > > > type1`, then evict all elements of type type1
> >> else
> >> >> >> don't
> >> >> >> > > > evict
> >> >> >> > > > >>> any
> >> >> >> > > > >>> > > > > elements.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > Is this possible? or at least let the window
> >> >> function
> >> >> >> > > choose
> >> >> >> > > > >>> > between
> >> >> >> > > > >>> > > > > > two Evictor functions -(one for success case
> and
> >> >> one
> >> >> >> > > failure
> >> >> >> > > > >>> case)
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > @Maxim:
> >> >> >> > > > >>> > > > > > regarding the sorted window, actually I wanted
> >> my
> >> >> >> > elements
> >> >> >> > > to
> >> >> >> > > > >>> be
> >> >> >> > > > >>> > > > > > sorted but not for the eviction but while
> >> applying
> >> >> the
> >> >> >> > > window
> >> >> >> > > > >>> > > function
> >> >> >> > > > >>> > > > > > (so thought this could be done easily). But it
> >> >> would
> >> >> >> be
> >> >> >> > > good
> >> >> >> > > > to
> >> >> >> > > > >>> > have
> >> >> >> > > > >>> > > > > > the window sorted based on EventTime.
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > Thanks and Regards,
> >> >> >> > > > >>> > > > > > Vishnu Viswanath,
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > On Wed, Jul 6, 2016 at 3:55 PM, Maxim <
> >> >> >> mfat...@gmail.com
> >> >> >> > >
> >> >> >> > > > >>> wrote:
> >> >> >> > > > >>> > > > > >
> >> >> >> > > > >>> > > > > > > Actually for such evictor to be useful the
> >> window
> >> >> >> > should
> >> >> >> > > be
> >> >> >> > > > >>> > sorted
> >> >> >> > > > >>> > > > > > > by some field, usually event time. What do
> you
> >> >> think
> >> >> >> > > about
> >> >> >> > > > >>> adding
> >> >> >> > > > >>> > > > > > > sorted window abstraction?
> >> >> >> > > > >>> > > > > > >
> >> >> >> > > > >>> > > > > > > On Wed, Jul 6, 2016 at 11:36 AM, Aljoscha
> >> Krettek
> >> >> >> > > > >>> > > > > > > <aljos...@apache.org>
> >> >> >> > > > >>> > > > > > > wrote:
> >> >> >> > > > >>> > > > > > >
> >> >> >> > > > >>> > > > > > > > @Maxim: That's perfect I didn't think
> about
> >> >> using
> >> >> >> > > > >>> > > > > > > > Iterator.remove() for that. I'll update
> the
> >> >> doc.
> >> >> >> What
> >> >> >> > > do
> >> >> >> > > > >>> you
> >> >> >> > > > >>> > > think
> >> >> >> > > > >>> > > > > > > > Vishnu? This should also
> >> >> >> > > > >>> > > > > > > cover
> >> >> >> > > > >>> > > > > > > > your before/after case nicely.
> >> >> >> > > > >>> > > > > > > >
> >> >> >> > > > >>> > > > > > > > @Vishnu: The steps would be these:
> >> >> >> > > > >>> > > > > > > > - Converge on a design in this discussion
> >> >> >> > > > >>> > > > > > > > - Add a Jira issue here:
> >> >> >> > > > >>> > > > > > > >
> https://issues.apache.org/jira/browse/FLINK
> >> >> >> > > > >>> > > > > > > > - Work on the code an create a pull
> >> request on
> >> >> >> > github
> >> >> >> > > > >>> > > > > > > >
> >> >> >> > > > >>> > > > > > > > The steps are also outlined here
> >> >> >> > > > >>> > > > > > > >
> >> http://flink.apache.org/how-to-contribute.html
> >> >> >> and
> >> >> >> > > here
> >> >> >> > > > >>> > > > > > > >
> >> http://flink.apache.org/contribute-code.html.
> >> >> >> > > > >>> > > > > > > >
> >> >> >> > > > >>> > > > > > > > -
> >> >> >> > > > >>> > > > > > > > Aljoscha
> >> >> >> > > > >>> > > > > > > >
> >> >> >> > > > >>> > > > > > > > On Wed, 6 Jul 2016 at 19:45 Maxim <
> >> >> >> mfat...@gmail.com
> >> >> >> > >
> >> >> >> > > > >>> wrote:
> >> >> >> > > > >>> > > > > > > >
> >> >> >> > > > >>> > > > > > > > > The new API forces iteration through
> every
> >> >> >> element
> >> >> >> > of
> >> >> >> > > > the
> >> >> >> > > > >>> > > buffer
> >> >> >> > > > >>> > > > > > > > > even
> >> >> >> > > > >>> > > > > > > if
> >> >> >> > > > >>> > > > > > > > a
> >> >> >> > > > >>> > > > > > > > > single value to be evicted. What about
> >> >> >> implementing
> >> >> >> > > > >>> > > > > > > > > Iterator.remove() method for elements?
> The
> >> >> API
> >> >> >> > would
> >> >> >> > > > look
> >> >> >> > > > >>> > like:
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > public interface Evictor<T, W extends
> >> Window>
> >> >> >> > extends
> >> >> >> > > > >>> > > > > > > > > Serializable {
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > /**
> >> >> >> > > > >>> > > > > > > > > * Optionally evicts elements.
> Called
> >> >> before
> >> >> >> > > > >>> windowing
> >> >> >> > > > >>> > > > > function.
> >> >> >> > > > >>> > > > > > > > > *
> >> >> >> > > > >>> > > > > > > > > * @param elements The elements
> >> currently
> >> >> in
> >> >> >> the
> >> >> >> > > > >>> pane. Use
> >> >> >> > > > >>> > > > > > > > > Iterator.remove to evict.
> >> >> >> > > > >>> > > > > > > > > * @param size The current number of
> >> >> >> elements in
> >> >> >> > > the
> >> >> >> > > > >>> pane.
> >> >> >> > > > >>> > > > > > > > > * @param window The {@link Window}
> >> >> >> > > > >>> > > > > > > > > */
> >> >> >> > > > >>> > > > > > > > > void evictBefore(Iterable<T>
> elements,
> >> int
> >> >> >> size,
> >> >> >> > > > >>> > > > > > > > > EvictorContext
> >> >> >> > > > >>> > > > > > > ctx);
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > /**
> >> >> >> > > > >>> > > > > > > > > * Optionally evicts elements.
> Called
> >> >> after
> >> >> >> > > > windowing
> >> >> >> > > > >>> > > > function.
> >> >> >> > > > >>> > > > > > > > > *
> >> >> >> > > > >>> > > > > > > > > * @param elements The elements
> >> currently
> >> >> in
> >> >> >> the
> >> >> >> > > > >>> pane. Use
> >> >> >> > > > >>> > > > > > > > > Iterator.remove to evict.
> >> >> >> > > > >>> > > > > > > > > * @param size The current number of
> >> >> >> elements in
> >> >> >> > > the
> >> >> >> > > > >>> pane.
> >> >> >> > > > >>> > > > > > > > > * @param window The {@link Window}
> >> >> >> > > > >>> > > > > > > > > */
> >> >> >> > > > >>> > > > > > > > > void evictAfter(Iterable<T> elements,
> >> int
> >> >> >> size,
> >> >> >> > > > >>> > > > > > > > > EvictorContext ctx); }
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > Such API allows to abort iteration at
> any
> >> >> point
> >> >> >> and
> >> >> >> > > > evict
> >> >> >> > > > >>> > > > > > > > > elements in
> >> >> >> > > > >>> > > > > > > any
> >> >> >> > > > >>> > > > > > > > > order.
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > Thanks,
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > Maxim.
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > On Wed, Jul 6, 2016 at 9:04 AM, Vishnu
> >> >> >> Viswanath <
> >> >> >> > > > >>> > > > > > > > > vishnu.viswanat...@gmail.com> wrote:
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > Hi Aljoscha,
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > Thanks. Yes the new interface seems to
> >> >> address
> >> >> >> > > points
> >> >> >> > > > >>> 1 and
> >> >> >> > > > >>> > > 2.
> >> >> >> > > > >>> > > > > > > > > > of
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > *1) I am having a use case where I
> have
> >> to
> >> >> >> > create a
> >> >> >> > > > >>> custom
> >> >> >> > > > >>> > > > > > > > > > Evictor
> >> >> >> > > > >>> > > > > > > that
> >> >> >> > > > >>> > > > > > > > > > will evict elements from the window
> >> based
> >> >> on
> >> >> >> the
> >> >> >> > > > value
> >> >> >> > > > >>> > (e.g.,
> >> >> >> > > > >>> > > > > > > > > > if I
> >> >> >> > > > >>> > > > > > > have
> >> >> >> > > > >>> > > > > > > > > > elements are of case class Item(id:
> Int,
> >> >> >> > > type:String)
> >> >> >> > > > >>> then
> >> >> >> > > > >>> > > > > > > > > > evict
> >> >> >> > > > >>> > > > > > > > elements
> >> >> >> > > > >>> > > > > > > > > > that has type="a"). I believe this is
> >> not
> >> >> >> > currently
> >> >> >> > > > >>> > > possible.*
> >> >> >> > > > >>> > > > > > > > > > *2) this is somewhat related to 1)
> where
> >> >> there
> >> >> >> > > should
> >> >> >> > > > >>> be an
> >> >> >> > > > >>> > > > > > > > > > option to
> >> >> >> > > > >>> > > > > > > > > evict
> >> >> >> > > > >>> > > > > > > > > > elements from anywhere in the window.
> >> not
> >> >> only
> >> >> >> > from
> >> >> >> > > > the
> >> >> >> > > > >>> > > > > > > > > > beginning of
> >> >> >> > > > >>> > > > > > > > the
> >> >> >> > > > >>> > > > > > > > > > window. (e.g., apply the delta
> function
> >> to
> >> >> all
> >> >> >> > > > >>> elements and
> >> >> >> > > > >>> > > > > > > > > > remove
> >> >> >> > > > >>> > > > > > > all
> >> >> >> > > > >>> > > > > > > > > > those don't pass. I checked the code
> and
> >> >> evict
> >> >> >> > > method
> >> >> >> > > > >>> just
> >> >> >> > > > >>> > > > > > > > > > returns
> >> >> >> > > > >>> > > > > > > the
> >> >> >> > > > >>> > > > > > > > > > number of elements to be removed and
> >> >> >> > > > >>> processTriggerResult
> >> >> >> > > > >>> > > just
> >> >> >> > > > >>> > > > > > > > > > skips
> >> >> >> > > > >>> > > > > > > > > those
> >> >> >> > > > >>> > > > > > > > > > many elements from the beginning. *
> >> >> >> > > > >>> > > > > > > > > > *3) Add an option to enables the user
> to
> >> >> >> decide
> >> >> >> > if
> >> >> >> > > > the
> >> >> >> > > > >>> > > > > > > > > > eviction
> >> >> >> > > > >>> > > > > > > should
> >> >> >> > > > >>> > > > > > > > > > happen before the apply function or
> >> after
> >> >> the
> >> >> >> > apply
> >> >> >> > > > >>> > function.
> >> >> >> > > > >>> > > > > > > Currently
> >> >> >> > > > >>> > > > > > > > > it
> >> >> >> > > > >>> > > > > > > > > > is before the apply function, but I
> >> have a
> >> >> use
> >> >> >> > case
> >> >> >> > > > >>> where I
> >> >> >> > > > >>> > > > > > > > > > need to
> >> >> >> > > > >>> > > > > > > > first
> >> >> >> > > > >>> > > > > > > > > > apply the function and evict
> afterward.*
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > I would be interested in contributing
> to
> >> >> the
> >> >> >> code
> >> >> >> > > > base.
> >> >> >> > > > >>> > > Please
> >> >> >> > > > >>> > > > > > > > > > let me
> >> >> >> > > > >>> > > > > > > > > know
> >> >> >> > > > >>> > > > > > > > > > the steps.
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > Thanks and Regards,
> >> >> >> > > > >>> > > > > > > > > > Vishnu Viswanath
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > On Wed, Jul 6, 2016 at 11:49 AM,
> >> Aljoscha
> >> >> >> > Krettek <
> >> >> >> > > > >>> > > > > > > aljos...@apache.org
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > wrote:
> >> >> >> > > > >>> > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > > Hi,
> >> >> >> > > > >>> > > > > > > > > > > as mentioned in the thread on
> >> improving
> >> >> the
> >> >> >> > > > Windowing
> >> >> >> > > > >>> > API I
> >> >> >> > > > >>> > > > > > > > > > > also
> >> >> >> > > > >>> > > > > > > > have a
> >> >> >> > > > >>> > > > > > > > > > > design doc just for improving
> >> >> >> WindowEvictors. I
> >> >> >> > > had
> >> >> >> > > > >>> this
> >> >> >> > > > >>> > in
> >> >> >> > > > >>> > > > > > > > > > > my head
> >> >> >> > > > >>> > > > > > > > for
> >> >> >> > > > >>> > > > > > > > > a
> >> >> >> > > > >>> > > > > > > > > > > while but was hesitant to publish
> but
> >> >> since
> >> >> >> > > people
> >> >> >> > > > >>> are
> >> >> >> > > > >>> > > > > > > > > > > asking about
> >> >> >> > > > >>> > > > > > > > > this
> >> >> >> > > > >>> > > > > > > > > > > now might be a good time to post it.
> >> >> Here's
> >> >> >> the
> >> >> >> > > > doc:
> >> >> >> > > > >>> > > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > > >
> >> >> >> > > > >>> > > > > > >
> >> >> >> > > > >>> > >
> >> >> >> > > >
> >> >> >>
> >> https://docs.google.com/document/d/1Rr7xzlItYqvFXLyyy-Yv0vvw8f29QYAj
> >> >> >> > > > >>> > > > > > > m5
> >> >> >> > > > >>> > > > > > > i9E4A_JlU/edit?usp=sharing
> >> >> >> > > > >>> > > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > > Feedback/Suggestions are very
> welcome!
> >> >> >> Please
> >> >> >> > let
> >> >> >> > > > me
> >> >> >> > > > >>> know
> >> >> >> > > > >>> > > > > > > > > > > what you
> >> >> >> > > > >>> > > > > > > > > think.
> >> >> >> > > > >>> > > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > > @Vishnu: Are you interested in
> >> >> contributing
> >> >> >> a
> >> >> >> > > > >>> solution
> >> >> >> > > > >>> > for
> >> >> >> > > > >>> > > > > > > > > > > this to
> >> >> >> > > > >>> > > > > > > > the
> >> >> >> > > > >>> > > > > > > > > > > Flink code base? I'd be very happy
> to
> >> >> work
> >> >> >> with
> >> >> >> > > you
> >> >> >> > > > >>> on
> >> >> >> > > > >>> > > this.
> >> >> >> > > > >>> > > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > > Cheers,
> >> >> >> > > > >>> > > > > > > > > > > Aljoscha
> >> >> >> > > > >>> > > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > > > > P.S. I think it would be best to
> keep
> >> >> >> > discussions
> >> >> >> > > > to
> >> >> >> > > > >>> the
> >> >> >> > > > >>> > ML
> >> >> >> > > > >>> > > > > > > > > > > because comments on the doc will not
> >> be
> >> >> >> visible
> >> >> >> > > > here
> >> >> >> > > > >>> for
> >> >> >> > > > >>> > > > > > everyone.
> >> >> >> > > > >>> > > > > > > > > > >
> >> >> >> > > > >>> > > > > > > > >
> >> >> >> > > > >>> > > > > > >
