+1 for modules and operators to be used interchangeably. Regards, Sandeep
On Tue, Dec 1, 2015 at 11:54 PM, Vlad Rozov <[email protected]> wrote: > We should either provide Java API that supports interchangeability between > operators and modules or ignore how an operator/module was added to a DAG. > > Thank you, > > Vlad > > > On 11/30/15 20:21, Tushar Gosavi wrote: > >> Good point Vlad, if the class implements both Operator and Module >> interface. We will consider it as a module by default in json API, But >> also >> provide a way to specify his intention whether it should be considered as >> Module or Operator. >> >> If user use a Java API then intention is clear whether the object should >> be >> considered as Module or Operator (addOperator v/s addModule), will it be >> correct to change the intention of the user internally based on the type >> of >> component? >> >> - Tushar. >> >> >> On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <[email protected]> >> wrote: >> >> +1. Modules and operators should be interchangeable. It should be possible >>> for an operator designer to replace it with a module without requirement >>> to >>> recompile all applications that use the existing operator. From >>> application >>> designer view both operators and modules may be considered as a black box >>> with a private implementation that may be monolithic (operator) or >>> distributed (module). >>> >>> Note that a Java class may implement both Operator and Module interface >>> and be added to a DAG using addOperator() method. At run-time (Logical >>> Plan->Physical Plan) Apex platform should handle such classes as modules >>> and may not depend on how a module was added to a DAG, IMO. >>> >>> Thank you, >>> >>> Vlad >>> >>> >>> On 11/30/15 08:41, Tushar Gosavi wrote: >>> >>> addOperator method in DAG interface takes object of type Operator, unless >>>> we make Module as subclass of Operator we can not use the same API. >>>> >>>> - Tushar. >>>> >>>> >>>> On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <[email protected]> >>>> wrote: >>>> >>>> +1 >>>> >>>>> If there are really module specific constructs, they can be added >>>>> later. >>>>> >>>>> Why is the same not supported in the Java API, i.e. why not allow a >>>>> module >>>>> to be added like an operator and handle the distinction in the >>>>> implementation? >>>>> >>>>> On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected] >>>>> > >>>>> wrote: >>>>> >>>>> Hi All, >>>>> >>>>>> I am going to add a feature to allow use of modules in property file >>>>>> and >>>>>> json api. >>>>>> I am planing to use same "operators" array for adding modules as well >>>>>> as >>>>>> operators. The reasoning behind is that user don't have to worry about >>>>>> >>>>>> the >>>>> >>>>> component being added to the DAG is actually an operator or a module, >>>>>> as >>>>>> the method to specify their properties and to create instance of them >>>>>> is >>>>>> same. And it results in less complex syntax for writing application >>>>>> using >>>>>> json/property file. >>>>>> >>>>>> For example the sample Application specified using Json will look like >>>>>> below. >>>>>> ```json >>>>>> { >>>>>> "operators": [ >>>>>> { >>>>>> "name": "operator1", >>>>>> "class": "com.datatorrent.lib.operator.Input", >>>>>> "properties": { >>>>>> "property1": "value1" >>>>>> } >>>>>> }, >>>>>> { >>>>>> "name": "module1", >>>>>> "class": "com.datatorrent.module.Module1", >>>>>> "properties": { >>>>>> "property1": "value1" >>>>>> } >>>>>> }, >>>>>> ], >>>>>> "streams": [ >>>>>> { >>>>>> "name": "s1", >>>>>> "source": { >>>>>> "operatorName": "operator1", >>>>>> "portName": "output" >>>>>> }, >>>>>> "sinks": [ >>>>>> { >>>>>> "operatorName": "module", >>>>>> "portName": "input" >>>>>> } >>>>>> ] >>>>>> } >>>>>> ] >>>>>> } >>>>>> ``` >>>>>> >>>>>> In above example "module1" is of type Module. While processing the >>>>>> Json >>>>>> >>>>>> if >>>>> >>>>> the >>>>>> class of component is of type Module then we will use addModule api to >>>>>> >>>>>> the >>>>> >>>>> DAG. >>>>>> >>>>>> Let me know, what do you think about this. >>>>>> >>>>>> Regards, >>>>>> - Tushar. >>>>>> >>>>>> On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi < >>>>>> [email protected]> >>>>>> wrote: >>>>>> >>>>>> A module is superset of operator before expansion in logical plan, >>>>>> except >>>>>> operator >>>>>> >>>>>>> attribute are not applicable to modules. As Tim suggested we will >>>>>>> have >>>>>>> >>>>>>> to >>>>>> add checks >>>>>> >>>>>>> later to validate these attributes settings, and not allow top level >>>>>>> operator and >>>>>>> module to share same name. >>>>>>> >>>>>>> The same question comes when we want to add module to application >>>>>>> using >>>>>>> property >>>>>>> file API and JSON file API. >>>>>>> >>>>>>> dt.operator.<operatorname>.classname=<fqcn of operator> >>>>>>> >>>>>>> One approach we can take is, if class is of type module, then use >>>>>>> addModule api >>>>>>> to add module to the dag, else use addOperator api. The >>>>>>> initialization >>>>>>> >>>>>>> of >>>>>> object and applying property is generic can be reused by module. >>>>>> >>>>>>> Similarly we have Json API where all operators are specified under >>>>>>> operators array >>>>>>> >>>>>>> { >>>>>>> "operators": [ >>>>>>> {.. operators .. } >>>>>>> ], >>>>>>> "streams": [ >>>>>>> {.. streams .. } >>>>>>> ] >>>>>>> } >>>>>>> >>>>>>> We could add module in the operators array, and based on the object >>>>>>> >>>>>>> type >>>>>> we can >>>>>> >>>>>>> consider it as module or operator. The mechanism of initializing >>>>>>> object >>>>>>> and applying >>>>>>> properties remains same. >>>>>>> >>>>>>> - Tushar. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >
