+1 If there are really module specific constructs, they can be added later.
Why is the same not supported in the Java API, i.e. why not allow a module to be added like an operator and handle the distinction in the implementation? On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected]> wrote: > Hi All, > > I am going to add a feature to allow use of modules in property file and > json api. > I am planing to use same "operators" array for adding modules as well as > operators. The reasoning behind is that user don't have to worry about the > component being added to the DAG is actually an operator or a module, as > the method to specify their properties and to create instance of them is > same. And it results in less complex syntax for writing application using > json/property file. > > For example the sample Application specified using Json will look like > below. > ```json > { > "operators": [ > { > "name": "operator1", > "class": "com.datatorrent.lib.operator.Input", > "properties": { > "property1": "value1" > } > }, > { > "name": "module1", > "class": "com.datatorrent.module.Module1", > "properties": { > "property1": "value1" > } > }, > ], > "streams": [ > { > "name": "s1", > "source": { > "operatorName": "operator1", > "portName": "output" > }, > "sinks": [ > { > "operatorName": "module", > "portName": "input" > } > ] > } > ] > } > ``` > > In above example "module1" is of type Module. While processing the Json if > the > class of component is of type Module then we will use addModule api to the > DAG. > > Let me know, what do you think about this. > > Regards, > - Tushar. > > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <[email protected]> > wrote: > > > > > A module is superset of operator before expansion in logical plan, except > > operator > > attribute are not applicable to modules. As Tim suggested we will have to > > add checks > > later to validate these attributes settings, and not allow top level > > operator and > > module to share same name. > > > > The same question comes when we want to add module to application using > > property > > file API and JSON file API. > > > > dt.operator.<operatorname>.classname=<fqcn of operator> > > > > One approach we can take is, if class is of type module, then use > > addModule api > > to add module to the dag, else use addOperator api. The initialization of > > object and applying property is generic can be reused by module. > > > > Similarly we have Json API where all operators are specified under > > operators array > > > > { > > "operators": [ > > {.. operators .. } > > ], > > "streams": [ > > {.. streams .. } > > ] > > } > > > > We could add module in the operators array, and based on the object type > > we can > > consider it as module or operator. The mechanism of initializing object > > and applying > > properties remains same. > > > > - Tushar. > > > > > > > > > > >
