Generally I agree with the new design. Two concerns: 1) Does KeyedDataStream replace GroupedDataStream or is it the latter a special case of the former?
The KeyedDataStream as described in the design document is a bit unclear for me. It lists the following usages: a) It is the first step in building a window stream, on top of which the grouped/windowed aggregation and reduce-style function can be applied b) It allows to use the "by-key" state of functions. Here, every record has access to a state that is scoped by its key. Key-scoped state can be automatically redistributed and repartitioned. The code snippet describes a use case where the computation and the access of the state is used the way currently the GroupedDataStream should work. I suppose this is the example for case b). Would case a) also window elements by key? If yes, then this is practically a renaming and enhancement of the GroupedDataStream functionality with keyed state. Then the StreamExecutionEnvironment.createKeyedStream(Partitioner, KeySelector)construction does not make much sense as the user only operates within the scope of the keyselector and not the partitioner anyway. I personally think KeyedDataStream as a name does not necessarily suggest that the records are grouped by key, it only suggests partitioning by key - at least for me. :) 2) The API for discretization is not convenient IMHO The discretization part declares that the output of DataStream.discretize() is a sequence of DataSets. I love this approach, but then in the code snippet the return value of this function is simply a DataSet and uses it as such. The take home message of that code is the following: this is actually the way you would like to program on these sequence of DataSets, most probably you would like to do the same with each of them. If that is the case we should provide a nice utility for that. I think Spark Streaming's DStream.foreachRDD() is fairly useful for this purpose. On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: > +1 > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org> wrote: > > > If naming is the only concern, then we should go ahead, because we can > > change names easily (before the release). > > > > In fact, I don't think it leaves a bad impression. Global windows are > > non-parallel windows. There are also parallel windows. Pick what you need > > and what works. > > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gyula.f...@gmail.com> > wrote: > > > > > I think we agree on everything its more of a naming issue :) > > > > > > I thought it might be misleading that global time windows are > > > "non-parallel" windows. We dont want to give a bad impression. (Also we > > > dont want them to think that every global window is parallel but thats > > not > > > a problem here) > > > > > > Gyula > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org> wrote: > > > > > > > Okay, what is missing about the windowing in your opinion? > > > > > > > > The core points of the document are: > > > > > > > > - The parallel windows are per group only. > > > > > > > > - The implementation of the parallel windows holds window data in > the > > > > group buffers. > > > > > > > > - The global windows are non-parallel. May have parallel > > > pre-aggregation, > > > > if they are time windows. > > > > > > > > - Time may be operator time (timer thread), or watermark time. > > > Watermark > > > > time can refer to ingress or event time. > > > > > > > > - Windows that do not pre-aggregate may require elements in order. > > Not > > > > part of the first prototype. > > > > > > > > Do we agree on those points? > > > > > > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gyula.f...@gmail.com> > > > wrote: > > > > > > > > > In general I like it, although the main difference between the > > current > > > > and > > > > > the new one is the windowing and that is still not very clear. > > > > > > > > > > Where do we have the full stream time windows for instance?(which > is > > > > > parallel but not keyed) > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek < > > aljos...@apache.org> > > > > > wrote: > > > > > > > > > > > +1 I like it as well. > > > > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <ktzou...@apache.org > > > > > > wrote: > > > > > > > > > > > > > +1 from my side > > > > > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen < > se...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > Do we have consensus on these designs? > > > > > > > > > > > > > > > > If we have, we should get to implementing this soon, because > > > > > basically > > > > > > > all > > > > > > > > streaming patches will have to be revisited in light of > this... > > > > > > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra < > > gyula.f...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > You are right thats an important issue. > > > > > > > > > > > > > > > > > > And I think we should also do some renaming with the > > > "iterations" > > > > > > > because > > > > > > > > > they are not really iterations like in the batch case and > it > > > > might > > > > > > > > confuse > > > > > > > > > some users. > > > > > > > > > Maybe we can call them loops or cycles and rename the api > > calls > > > > to > > > > > > make > > > > > > > > it > > > > > > > > > more intuitive what happens. It is really just a cyclic > > > dataflow. > > > > > > > > > > > > > > > > > > Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: > > > 2015. > > > > > júl. > > > > > > > 7., > > > > > > > > > K, > > > > > > > > > 15:35): > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > I just noticed that we don't have anything about how > > > iterations > > > > > and > > > > > > > > > > timestamps/watermarks should interact. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen < > se...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > As many of you know, there are a ongoing efforts to > > > > consolidate > > > > > > the > > > > > > > > > > > streaming API for the next release, and then graduate > it > > > > (from > > > > > > beta > > > > > > > > > > > status). > > > > > > > > > > > > > > > > > > > > > > In the process of this consolidation, we want to > achieve > > > the > > > > > > > > following > > > > > > > > > > > goals. > > > > > > > > > > > > > > > > > > > > > > - Make the code more robust and simplify it in parts > > > > > > > > > > > > > > > > > > > > > > - Clearly define the semantics of the constructs. > > > > > > > > > > > > > > > > > > > > > > - Prepare it for support of more advanced concepts, > like > > > > > > > > partitionable > > > > > > > > > > > state, and event time. > > > > > > > > > > > > > > > > > > > > > > - Cut support for certain corner cases that were > > > prototyped, > > > > > but > > > > > > > > > turned > > > > > > > > > > > out to be not efficiently doable > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Based on prior discussions on the mailing list, > Aljoscha > > > and > > > > me > > > > > > > > drafted > > > > > > > > > > the > > > > > > > > > > > design documents below, which outline how the > > consolidated > > > > API > > > > > > > would > > > > > > > > > > like. > > > > > > > > > > > We focused in constructs, time, and window semantics. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Design document on how to restructure the Streaming > API: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams > > > > > > > > > > > > > > > > > > > > > > Design document on definitions of time, order, and the > > > > > resulting > > > > > > > > > > semantics: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Note: The design of the interfaces and concepts for > > > advanced > > > > > > state > > > > > > > in > > > > > > > > > > > functions is not in here. That is part of a separate > > design > > > > > > > > discussion > > > > > > > > > > and > > > > > > > > > > > orthogonal to the designs drafted here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please have a look and voice questions and concerns. > > Since > > > we > > > > > > > should > > > > > > > > > not > > > > > > > > > > > break the streaming API more than once, we should make > > sure > > > > > this > > > > > > > > > > > consolidation brings it into the shape we want it to be > > in. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Greetings, > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
