+1. Modules and operators should be interchangeable. It should be
possible for an operator designer to replace it with a module without
requirement to recompile all applications that use the existing
operator. From application designer view both operators and modules may
be considered as a black box with a private implementation that may be
monolithic (operator) or distributed (module).
Note that a Java class may implement both Operator and Module interface
and be added to a DAG using addOperator() method. At run-time (Logical
Plan->Physical Plan) Apex platform should handle such classes as modules
and may not depend on how a module was added to a DAG, IMO.
Thank you,
Vlad
On 11/30/15 08:41, Tushar Gosavi wrote:
addOperator method in DAG interface takes object of type Operator, unless
we make Module as subclass of Operator we can not use the same API.
- Tushar.
On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <[email protected]>
wrote:
+1
If there are really module specific constructs, they can be added later.
Why is the same not supported in the Java API, i.e. why not allow a module
to be added like an operator and handle the distinction in the
implementation?
On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected]>
wrote:
Hi All,
I am going to add a feature to allow use of modules in property file and
json api.
I am planing to use same "operators" array for adding modules as well as
operators. The reasoning behind is that user don't have to worry about
the
component being added to the DAG is actually an operator or a module, as
the method to specify their properties and to create instance of them is
same. And it results in less complex syntax for writing application using
json/property file.
For example the sample Application specified using Json will look like
below.
```json
{
"operators": [
{
"name": "operator1",
"class": "com.datatorrent.lib.operator.Input",
"properties": {
"property1": "value1"
}
},
{
"name": "module1",
"class": "com.datatorrent.module.Module1",
"properties": {
"property1": "value1"
}
},
],
"streams": [
{
"name": "s1",
"source": {
"operatorName": "operator1",
"portName": "output"
},
"sinks": [
{
"operatorName": "module",
"portName": "input"
}
]
}
]
}
```
In above example "module1" is of type Module. While processing the Json
if
the
class of component is of type Module then we will use addModule api to
the
DAG.
Let me know, what do you think about this.
Regards,
- Tushar.
On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <[email protected]>
wrote:
A module is superset of operator before expansion in logical plan,
except
operator
attribute are not applicable to modules. As Tim suggested we will have
to
add checks
later to validate these attributes settings, and not allow top level
operator and
module to share same name.
The same question comes when we want to add module to application using
property
file API and JSON file API.
dt.operator.<operatorname>.classname=<fqcn of operator>
One approach we can take is, if class is of type module, then use
addModule api
to add module to the dag, else use addOperator api. The initialization
of
object and applying property is generic can be reused by module.
Similarly we have Json API where all operators are specified under
operators array
{
"operators": [
{.. operators .. }
],
"streams": [
{.. streams .. }
]
}
We could add module in the operators array, and based on the object
type
we can
consider it as module or operator. The mechanism of initializing object
and applying
properties remains same.
- Tushar.
