Maybe put everything into a google doc and share the link?
On Fri, May 13, 2016 at 4:04 PM, David Yan <[email protected]> wrote: > Making Siyuan's diagram fixed width so we can actually see it :) > > +-------------------+ > +---------> | WindowedOperator | <--------+ > | +--------+----------+ | > | ^ ^--------------------------------+ > | | | | > | | | | > +------+--------+ +------+------+ +-------+-----+ +------+-----+ > |CombineOperator| |GroupOperator| |KeyedOperator| |JoinOperator| > +---------------+ +-------------+ +------+------+ +-----+------+ > +---------^ ^ ^ > | | | > +--------+---+ +-----+----+ +----+----+ > |KeyedCombine| |KeyedGroup| | CoGroup | > +------------+ +----------+ +---------+ > > On Wed, May 11, 2016 at 11:47 AM, Siyuan Hua <[email protected]> > wrote: > > > Hi guys, > > > > I just create a ticket for windowed operators. It is the prerequisite for > > High-level API and Beam integration. > > > > As per our recent several discussions in the community. A group of > Windowed > > Operators that delivers the window semantic follows the google Data Flow > > model(https://cloud.google.com/dataflow/) is very important. > > The operators should be designed and implemented in a way for > > High-level API > > Beam translation > > Easy to use with other popular operator > > > > Hierarchy of the operators, > > The windowed operators should cover all possible transformations that > > require window, and batch processing is also considered as special window > > called global window > > > > +-------------------+ > > +---------> | WindowedOperator | <--------+ > > | +--------+----------+ | > > | ^ ^--------------------------------+ > > | | | | > > | | | | > > +------+--------+ +------+------+ +-------+-----+ > +------+-----+ > > |CombineOperator| |GroupOperator| |KeyedOperator| > |JoinOperator| > > +---------------+ +-------------+ +------+------+ > +-----+------+ > > +---------^ ^ ^ > > | | | > > +--------+---+ +-----+----+ +----+----+ > > |KeyedCombine| |KeyedGroup| | CoGroup | > > +------------+ +----------+ +---------+ > > > > > > Combine operation includes all operations that combine all tuples in one > > window into one or small number of tuples, Group operation group all > tuples > > in one window, Join and CoGroup are used to join and group tuples from > > different inputs. > > > > Components: > > Window Component, it includes configuration, window state that should be > > checkpointed, etc. It should support NonMergibleWindow(fixed or slide) > > MergibleWindow(Session) > > > > Trigger: > > It should support early trigger, late trigger. It should support > > customizable trigger behaviour > > > > Outside components: > > Watermark generator, can be plugged into input source to generate > watermark > > Tuple util components, The operator should be only working on Tuples with > > schema. It can handle either predefined tuple type or give a declarative > > API to describe the user defined tuple class > > > > Most component API should be reused in High-Level API > > > > This is the umbrella ticket, separate tickets would be created for > > different components and operators respectively > > >
