I also prefer an explicit modules section. The user of the module needs to know it's a module, and if my understanding of modules is still correct, modules are not operators because it may be valid for a module's output port to connect to its own input port.
David On Mon, Nov 30, 2015 at 8:40 AM, Siyuan Hua <[email protected]> wrote: > I prefer explicit "modules" section. Just because it's more intuitive. > > On Mon, Nov 30, 2015 at 8:21 AM, Thomas Weise <[email protected]> > wrote: > > > +1 > > > > If there are really module specific constructs, they can be added later. > > > > Why is the same not supported in the Java API, i.e. why not allow a > module > > to be added like an operator and handle the distinction in the > > implementation? > > > > On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected]> > > wrote: > > > > > Hi All, > > > > > > I am going to add a feature to allow use of modules in property file > and > > > json api. > > > I am planing to use same "operators" array for adding modules as well > as > > > operators. The reasoning behind is that user don't have to worry about > > the > > > component being added to the DAG is actually an operator or a module, > as > > > the method to specify their properties and to create instance of them > is > > > same. And it results in less complex syntax for writing application > using > > > json/property file. > > > > > > For example the sample Application specified using Json will look like > > > below. > > > ```json > > > { > > > "operators": [ > > > { > > > "name": "operator1", > > > "class": "com.datatorrent.lib.operator.Input", > > > "properties": { > > > "property1": "value1" > > > } > > > }, > > > { > > > "name": "module1", > > > "class": "com.datatorrent.module.Module1", > > > "properties": { > > > "property1": "value1" > > > } > > > }, > > > ], > > > "streams": [ > > > { > > > "name": "s1", > > > "source": { > > > "operatorName": "operator1", > > > "portName": "output" > > > }, > > > "sinks": [ > > > { > > > "operatorName": "module", > > > "portName": "input" > > > } > > > ] > > > } > > > ] > > > } > > > ``` > > > > > > In above example "module1" is of type Module. While processing the Json > > if > > > the > > > class of component is of type Module then we will use addModule api to > > the > > > DAG. > > > > > > Let me know, what do you think about this. > > > > > > Regards, > > > - Tushar. > > > > > > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <[email protected] > > > > > wrote: > > > > > > > > > > > A module is superset of operator before expansion in logical plan, > > except > > > > operator > > > > attribute are not applicable to modules. As Tim suggested we will > have > > to > > > > add checks > > > > later to validate these attributes settings, and not allow top level > > > > operator and > > > > module to share same name. > > > > > > > > The same question comes when we want to add module to application > using > > > > property > > > > file API and JSON file API. > > > > > > > > dt.operator.<operatorname>.classname=<fqcn of operator> > > > > > > > > One approach we can take is, if class is of type module, then use > > > > addModule api > > > > to add module to the dag, else use addOperator api. The > initialization > > of > > > > object and applying property is generic can be reused by module. > > > > > > > > Similarly we have Json API where all operators are specified under > > > > operators array > > > > > > > > { > > > > "operators": [ > > > > {.. operators .. } > > > > ], > > > > "streams": [ > > > > {.. streams .. } > > > > ] > > > > } > > > > > > > > We could add module in the operators array, and based on the object > > type > > > > we can > > > > consider it as module or operator. The mechanism of initializing > object > > > > and applying > > > > properties remains same. > > > > > > > > - Tushar. > > > > > > > > > > > > > > > > > > > > > > > > > >
