What about the collector? This is only used for emitting elements to the downstream operation.
On Mon, 18 Jul 2016 at 17:52 Radu Tudoran <radu.tudo...@huawei.com> wrote: > Hi, > > I think it looks good and most importantly is that we can extend it in the > directions discussed so far. > > One question though regarding the Collector - are we going to be able to > delete random elements from the list if this is not exposed as a > collection, at least to the evictor? If not, how are we going to extend in > the future to cover this case? > > Regarding the ordering - I also observed that there are situations where > elements do not have a logical order. One example is if you have high rates > of the events. Nevertheless, even if now is not the time for this, I think > in the future we can imagine having also some data structures that offer > some ordering. It can save some computation efforts later in the functions > for some use cases. > > > Best regards, > > > -----Original Message----- > From: Aljoscha Krettek [mailto:aljos...@apache.org] > Sent: Monday, July 18, 2016 3:45 PM > To: dev@flink.apache.org > Subject: Re: [DISCUSS] FLIP-2 Extending Window Function Metadata > > I incorporated the changes. The proposed interface of > ProcessWindowFunction is now this: > > public abstract class ProcessWindowFunction <IN, OUT, KEY, W extends > Window> implements Function { > > public abstract void process(KEY key, Iterable<IN> elements, Context > ctx) throws Exception; > > public abstract class Context { > public abstract W window(); > public abstract void output(OUT value); > } > } > > I'm proposing to not expose Collector anymore because it has the close() > method that should not be called by users. Having the output() call > directly on the context should work just as well. > > Also, I marked the "adding a firing reason" and "adding firing counter" as > future work that are only examples of stuff that can be implemented on top > of the new interface. Initially, this will provide exactly the same > features as the old API but be extensible. I did this to not make the scope > of this proposal to big because Radu also suggested more changes and each > of them should be covered in a separate design doc or FLIP. > > @Radu: On the different buffer types. I think this would be very tricky. > Right now, people should also not rely on the fact that elements are > "FIFO". Some state backends might keep the elements in a different order > and when you have merging windows/session windows the order of the elements > will also not be preserved. > > Cheers, > Aljoscha > > On Wed, 13 Jul 2016 at 18:40 Radu Tudoran <radu.tudo...@huawei.com> wrote: > > > Hi, > > > > If it is to extend the Context to pass more information between the > > stages of processing a window (triggering -> process -> eviction), why > > not adding also a "EvictionInfo"? I think this might actually help > > with the issues discussed in the tread related to the eviction policy. > > I could imagine using this parameter to pass the conditions, from the > > processing stage to the evictor, about what events to be eliminated. > > > > > > > > > > public abstract class Context { > > > > public abstract EvictionInfo evictionInfo(); > > > > ... > > > > > > public abstract KEY key(); > > > > public abstract W window(); > > > > public abstract int id(); > > > > public abstract FiringInfo firingInfo(); > > > > public abstract Iterable<IN> elements(); > > > > public abstract void output(OUT value); > > > > } > > > > > > Also on a slightly unrelated issue - how hard it would be to introduce > > different types of buffers for the windows. Currently the existing one > > is behaving (when under processing) similar with a FIFO queue (in the > > sense that you need to start from beginning, from the oldest element). > > How about enabling for example also LIFO behavior (start iterating > > through the list from the most recent element). As in the source > > queues or stacks are not actually used, perhaps we can just pass > > policies to the iterator - or have custom itrators > > > > > > > > > > > > > > Dr. Radu Tudoran > > Research Engineer - Big Data Expert > > IT R&D Division > > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > European Research Center > > Riesstrasse 25, 80992 München > > > > E-mail: radu.tudo...@huawei.com > > Mobile: +49 15209084330 > > Telephone: +49 891588344173 > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH > > Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com Registered > > Office: Düsseldorf, Register Court Düsseldorf, HRB 56063, Managing > > Director: Bo PENG, Wanzhou MENG, Lifang CHEN Sitz der Gesellschaft: > > Düsseldorf, Amtsgericht Düsseldorf, HRB 56063, > > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN This e-mail and > > its attachments contain confidential information from HUAWEI, which is > > intended only for the person or entity whose address is listed above. > > Any use of the information contained herein in any way (including, but > > not limited to, total or partial disclosure, reproduction, or > > dissemination) by persons other than the intended recipient(s) is > > prohibited. If you receive this e-mail in error, please notify the > > sender by phone or email immediately and delete it! > > > > > > -----Original Message----- > > From: Aljoscha Krettek [mailto:aljos...@apache.org] > > Sent: Wednesday, July 13, 2016 2:24 PM > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS] FLIP-2 Extending Window Function Metadata > > > > Sure, I also thought about this but went for the "extreme" initially. > > If no-one objects I'll update the doc in a bit. > > > > On Wed, 13 Jul 2016 at 14:17 Stephan Ewen <se...@apache.org> wrote: > > > > > Thanks for opening this. > > > > > > I see the need for having an extensible context object for window > > > function invocations, but i think hiding every parameter in the > > > context is a bit unnatural. > > > > > > How about having a function "apply(Key, Values, WindowContext, > > Collector)" > > > ? > > > It should be possible to write the straightforward use cases without > > > accessing the context object. > > > > > > > > > > > > On Wed, Jul 13, 2016 at 1:56 PM, Aljoscha Krettek > > > <aljos...@apache.org> > > > wrote: > > > > > > > Hi, > > > > this is a proposal to introduce a new interface for the window > > > > function > > > to > > > > make it more extensible for the future where we might want to > > > > provide additional information about why a window fired to the > > > > user > > function: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-2+Extending+W > > > in > > > dow+Function+Metadata > > > > > > > > I'd appreciate your thoughts! > > > > > > > > Cheers, > > > > Aljoscha > > > > > > > > > >
