It will be possible to connect output port to input port of same module using old json format. The DAG validation is performed post expansion. Another advantage of keeping json format compatible with the existing format is that it will allow us to use existing tools (dtAssemble) with little/no changes.
On Tue, Dec 1, 2015 at 1:01 AM, David Yan <[email protected]> wrote: > I also prefer an explicit modules section. The user of the module needs to > know it's a module, and if my understanding of modules is still correct, > modules are not operators because it may be valid for a module's output > port to connect to its own input port. > > David > > On Mon, Nov 30, 2015 at 8:40 AM, Siyuan Hua <[email protected]> > wrote: > > > I prefer explicit "modules" section. Just because it's more intuitive. > > > > On Mon, Nov 30, 2015 at 8:21 AM, Thomas Weise <[email protected]> > > wrote: > > > > > +1 > > > > > > If there are really module specific constructs, they can be added > later. > > > > > > Why is the same not supported in the Java API, i.e. why not allow a > > module > > > to be added like an operator and handle the distinction in the > > > implementation? > > > > > > On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected] > > > > > wrote: > > > > > > > Hi All, > > > > > > > > I am going to add a feature to allow use of modules in property file > > and > > > > json api. > > > > I am planing to use same "operators" array for adding modules as well > > as > > > > operators. The reasoning behind is that user don't have to worry > about > > > the > > > > component being added to the DAG is actually an operator or a module, > > as > > > > the method to specify their properties and to create instance of them > > is > > > > same. And it results in less complex syntax for writing application > > using > > > > json/property file. > > > > > > > > For example the sample Application specified using Json will look > like > > > > below. > > > > ```json > > > > { > > > > "operators": [ > > > > { > > > > "name": "operator1", > > > > "class": "com.datatorrent.lib.operator.Input", > > > > "properties": { > > > > "property1": "value1" > > > > } > > > > }, > > > > { > > > > "name": "module1", > > > > "class": "com.datatorrent.module.Module1", > > > > "properties": { > > > > "property1": "value1" > > > > } > > > > }, > > > > ], > > > > "streams": [ > > > > { > > > > "name": "s1", > > > > "source": { > > > > "operatorName": "operator1", > > > > "portName": "output" > > > > }, > > > > "sinks": [ > > > > { > > > > "operatorName": "module", > > > > "portName": "input" > > > > } > > > > ] > > > > } > > > > ] > > > > } > > > > ``` > > > > > > > > In above example "module1" is of type Module. While processing the > Json > > > if > > > > the > > > > class of component is of type Module then we will use addModule api > to > > > the > > > > DAG. > > > > > > > > Let me know, what do you think about this. > > > > > > > > Regards, > > > > - Tushar. > > > > > > > > On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi < > [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > A module is superset of operator before expansion in logical plan, > > > except > > > > > operator > > > > > attribute are not applicable to modules. As Tim suggested we will > > have > > > to > > > > > add checks > > > > > later to validate these attributes settings, and not allow top > level > > > > > operator and > > > > > module to share same name. > > > > > > > > > > The same question comes when we want to add module to application > > using > > > > > property > > > > > file API and JSON file API. > > > > > > > > > > dt.operator.<operatorname>.classname=<fqcn of operator> > > > > > > > > > > One approach we can take is, if class is of type module, then use > > > > > addModule api > > > > > to add module to the dag, else use addOperator api. The > > initialization > > > of > > > > > object and applying property is generic can be reused by module. > > > > > > > > > > Similarly we have Json API where all operators are specified under > > > > > operators array > > > > > > > > > > { > > > > > "operators": [ > > > > > {.. operators .. } > > > > > ], > > > > > "streams": [ > > > > > {.. streams .. } > > > > > ] > > > > > } > > > > > > > > > > We could add module in the operators array, and based on the object > > > type > > > > > we can > > > > > consider it as module or operator. The mechanism of initializing > > object > > > > > and applying > > > > > properties remains same. > > > > > > > > > > - Tushar. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
