Good point Vlad, if the class implements both Operator and Module interface. We will consider it as a module by default in json API, But also provide a way to specify his intention whether it should be considered as Module or Operator.
If user use a Java API then intention is clear whether the object should be considered as Module or Operator (addOperator v/s addModule), will it be correct to change the intention of the user internally based on the type of component? - Tushar. On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <[email protected]> wrote: > +1. Modules and operators should be interchangeable. It should be possible > for an operator designer to replace it with a module without requirement to > recompile all applications that use the existing operator. From application > designer view both operators and modules may be considered as a black box > with a private implementation that may be monolithic (operator) or > distributed (module). > > Note that a Java class may implement both Operator and Module interface > and be added to a DAG using addOperator() method. At run-time (Logical > Plan->Physical Plan) Apex platform should handle such classes as modules > and may not depend on how a module was added to a DAG, IMO. > > Thank you, > > Vlad > > > On 11/30/15 08:41, Tushar Gosavi wrote: > >> addOperator method in DAG interface takes object of type Operator, unless >> we make Module as subclass of Operator we can not use the same API. >> >> - Tushar. >> >> >> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <[email protected]> >> wrote: >> >> +1 >>> >>> If there are really module specific constructs, they can be added later. >>> >>> Why is the same not supported in the Java API, i.e. why not allow a >>> module >>> to be added like an operator and handle the distinction in the >>> implementation? >>> >>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected]> >>> wrote: >>> >>> Hi All, >>>> >>>> I am going to add a feature to allow use of modules in property file and >>>> json api. >>>> I am planing to use same "operators" array for adding modules as well as >>>> operators. The reasoning behind is that user don't have to worry about >>>> >>> the >>> >>>> component being added to the DAG is actually an operator or a module, as >>>> the method to specify their properties and to create instance of them is >>>> same. And it results in less complex syntax for writing application >>>> using >>>> json/property file. >>>> >>>> For example the sample Application specified using Json will look like >>>> below. >>>> ```json >>>> { >>>> "operators": [ >>>> { >>>> "name": "operator1", >>>> "class": "com.datatorrent.lib.operator.Input", >>>> "properties": { >>>> "property1": "value1" >>>> } >>>> }, >>>> { >>>> "name": "module1", >>>> "class": "com.datatorrent.module.Module1", >>>> "properties": { >>>> "property1": "value1" >>>> } >>>> }, >>>> ], >>>> "streams": [ >>>> { >>>> "name": "s1", >>>> "source": { >>>> "operatorName": "operator1", >>>> "portName": "output" >>>> }, >>>> "sinks": [ >>>> { >>>> "operatorName": "module", >>>> "portName": "input" >>>> } >>>> ] >>>> } >>>> ] >>>> } >>>> ``` >>>> >>>> In above example "module1" is of type Module. While processing the Json >>>> >>> if >>> >>>> the >>>> class of component is of type Module then we will use addModule api to >>>> >>> the >>> >>>> DAG. >>>> >>>> Let me know, what do you think about this. >>>> >>>> Regards, >>>> - Tushar. >>>> >>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <[email protected]> >>>> wrote: >>>> >>>> A module is superset of operator before expansion in logical plan, >>>>> >>>> except >>> >>>> operator >>>>> attribute are not applicable to modules. As Tim suggested we will have >>>>> >>>> to >>> >>>> add checks >>>>> later to validate these attributes settings, and not allow top level >>>>> operator and >>>>> module to share same name. >>>>> >>>>> The same question comes when we want to add module to application using >>>>> property >>>>> file API and JSON file API. >>>>> >>>>> dt.operator.<operatorname>.classname=<fqcn of operator> >>>>> >>>>> One approach we can take is, if class is of type module, then use >>>>> addModule api >>>>> to add module to the dag, else use addOperator api. The initialization >>>>> >>>> of >>> >>>> object and applying property is generic can be reused by module. >>>>> >>>>> Similarly we have Json API where all operators are specified under >>>>> operators array >>>>> >>>>> { >>>>> "operators": [ >>>>> {.. operators .. } >>>>> ], >>>>> "streams": [ >>>>> {.. streams .. } >>>>> ] >>>>> } >>>>> >>>>> We could add module in the operators array, and based on the object >>>>> >>>> type >>> >>>> we can >>>>> consider it as module or operator. The mechanism of initializing object >>>>> and applying >>>>> properties remains same. >>>>> >>>>> - Tushar. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >
