My initial thinking is that the custom control tuples, just like the existing control tuples, will only be generated from the input operators and will be propagated downstream to all operators in the DAG. So the NxM partitioning scenario works just like how other control tuples work, i.e. the callback will not be called unless all ports have received the control tuple for a particular window. This creates a little bit of complication with multiple input operators though.
David On Fri, Jun 24, 2016 at 12:03 PM, Tushar Gosavi <[email protected]> wrote: > +1 for the feature > > I am in favor of option 1, but we may need an helper method to avoid > compiler error on typed port, as calling port.emit(controlTuple) will > be an error if type of control tuple and port does not match. or new > method in outputPort object , emitControlTuple(ControlTuple). > > Can you give example of piggy backing tuple with current BEGIN_WINDOW > and END_WINDOW control tuples? > > In case of NxM partitioning, each downstream operator will receive N > control tuples. will it call user handler N times for each downstream > operator or just once. > > Regards, > - Tushar. > > > > On Fri, Jun 24, 2016 at 11:52 PM, David Yan <[email protected]> wrote: > > Hi all, > > > > I would like to propose a new feature to the Apex core engine -- the > > support of custom control tuples. Currently, we have control tuples such > as > > BEGIN_WINDOW, END_WINDOW, CHECKPOINT, and so on, but we don't have the > > support for applications to insert their own control tuples. The way > > currently to get around this is to use data tuples and have a separate > port > > for such tuples that sends tuples to all partitions of the downstream > > operators, which is not exactly developer friendly. > > > > We have already seen a number of use cases that can use this feature: > > > > 1) Batch support: We need to tell all operators of the physical DAG when > a > > batch starts and ends, so the operators can do whatever that is needed > upon > > the start or the end of a batch. > > > > 2) Watermark: To support the concepts of event time windowing, the > > watermark control tuple is needed to tell which windows should be > > considered late. > > > > 3) Changing operator properties: We do have the support of changing > > operator properties on the fly, but with a custom control tuple, the > > command to change operator properties can be window aligned for all > > partitions and also across the DAG. > > > > 4) Recording tuples: Like changing operator properties, we do have this > > support now but only at the individual physical operator level, and > without > > control of which window to record tuples for. With a custom control > tuple, > > because a control tuple must belong to a window, all operators in the DAG > > can start (and stop) recording for the same windows. > > > > I can think of two options to achieve this: > > > > 1) new custom control tuple type that takes user's serializable object. > > > > 2) piggy back the current BEGIN_WINDOW and END_WINDOW control tuples. > > > > Please provide your feedback. Thank you. > > > > David >
