+1. Modules and operators should be interchangeable. Should something be recommend in terms of naming convention for Module Class in that case?
- Chinmay. ~ Chinmay. On Tue, Dec 1, 2015 at 9:51 AM, Tushar Gosavi <[email protected]> wrote: > Good point Vlad, if the class implements both Operator and Module > interface. We will consider it as a module by default in json API, But also > provide a way to specify his intention whether it should be considered as > Module or Operator. > > If user use a Java API then intention is clear whether the object should be > considered as Module or Operator (addOperator v/s addModule), will it be > correct to change the intention of the user internally based on the type of > component? > > - Tushar. > > > On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <[email protected]> > wrote: > > > +1. Modules and operators should be interchangeable. It should be > possible > > for an operator designer to replace it with a module without requirement > to > > recompile all applications that use the existing operator. From > application > > designer view both operators and modules may be considered as a black box > > with a private implementation that may be monolithic (operator) or > > distributed (module). > > > > Note that a Java class may implement both Operator and Module interface > > and be added to a DAG using addOperator() method. At run-time (Logical > > Plan->Physical Plan) Apex platform should handle such classes as modules > > and may not depend on how a module was added to a DAG, IMO. > > > > Thank you, > > > > Vlad > > > > > > On 11/30/15 08:41, Tushar Gosavi wrote: > > > >> addOperator method in DAG interface takes object of type Operator, > unless > >> we make Module as subclass of Operator we can not use the same API. > >> > >> - Tushar. > >> > >> > >> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <[email protected]> > >> wrote: > >> > >> +1 > >>> > >>> If there are really module specific constructs, they can be added > later. > >>> > >>> Why is the same not supported in the Java API, i.e. why not allow a > >>> module > >>> to be added like an operator and handle the distinction in the > >>> implementation? > >>> > >>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected] > > > >>> wrote: > >>> > >>> Hi All, > >>>> > >>>> I am going to add a feature to allow use of modules in property file > and > >>>> json api. > >>>> I am planing to use same "operators" array for adding modules as well > as > >>>> operators. The reasoning behind is that user don't have to worry about > >>>> > >>> the > >>> > >>>> component being added to the DAG is actually an operator or a module, > as > >>>> the method to specify their properties and to create instance of them > is > >>>> same. And it results in less complex syntax for writing application > >>>> using > >>>> json/property file. > >>>> > >>>> For example the sample Application specified using Json will look like > >>>> below. > >>>> ```json > >>>> { > >>>> "operators": [ > >>>> { > >>>> "name": "operator1", > >>>> "class": "com.datatorrent.lib.operator.Input", > >>>> "properties": { > >>>> "property1": "value1" > >>>> } > >>>> }, > >>>> { > >>>> "name": "module1", > >>>> "class": "com.datatorrent.module.Module1", > >>>> "properties": { > >>>> "property1": "value1" > >>>> } > >>>> }, > >>>> ], > >>>> "streams": [ > >>>> { > >>>> "name": "s1", > >>>> "source": { > >>>> "operatorName": "operator1", > >>>> "portName": "output" > >>>> }, > >>>> "sinks": [ > >>>> { > >>>> "operatorName": "module", > >>>> "portName": "input" > >>>> } > >>>> ] > >>>> } > >>>> ] > >>>> } > >>>> ``` > >>>> > >>>> In above example "module1" is of type Module. While processing the > Json > >>>> > >>> if > >>> > >>>> the > >>>> class of component is of type Module then we will use addModule api to > >>>> > >>> the > >>> > >>>> DAG. > >>>> > >>>> Let me know, what do you think about this. > >>>> > >>>> Regards, > >>>> - Tushar. > >>>> > >>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi < > [email protected]> > >>>> wrote: > >>>> > >>>> A module is superset of operator before expansion in logical plan, > >>>>> > >>>> except > >>> > >>>> operator > >>>>> attribute are not applicable to modules. As Tim suggested we will > have > >>>>> > >>>> to > >>> > >>>> add checks > >>>>> later to validate these attributes settings, and not allow top level > >>>>> operator and > >>>>> module to share same name. > >>>>> > >>>>> The same question comes when we want to add module to application > using > >>>>> property > >>>>> file API and JSON file API. > >>>>> > >>>>> dt.operator.<operatorname>.classname=<fqcn of operator> > >>>>> > >>>>> One approach we can take is, if class is of type module, then use > >>>>> addModule api > >>>>> to add module to the dag, else use addOperator api. The > initialization > >>>>> > >>>> of > >>> > >>>> object and applying property is generic can be reused by module. > >>>>> > >>>>> Similarly we have Json API where all operators are specified under > >>>>> operators array > >>>>> > >>>>> { > >>>>> "operators": [ > >>>>> {.. operators .. } > >>>>> ], > >>>>> "streams": [ > >>>>> {.. streams .. } > >>>>> ] > >>>>> } > >>>>> > >>>>> We could add module in the operators array, and based on the object > >>>>> > >>>> type > >>> > >>>> we can > >>>>> consider it as module or operator. The mechanism of initializing > object > >>>>> and applying > >>>>> properties remains same. > >>>>> > >>>>> - Tushar. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > > >
