We should either provide Java API that supports interchangeability
between operators and modules or ignore how an operator/module was added
to a DAG.
Thank you,
Vlad
On 11/30/15 20:21, Tushar Gosavi wrote:
Good point Vlad, if the class implements both Operator and Module
interface. We will consider it as a module by default in json API, But also
provide a way to specify his intention whether it should be considered as
Module or Operator.
If user use a Java API then intention is clear whether the object should be
considered as Module or Operator (addOperator v/s addModule), will it be
correct to change the intention of the user internally based on the type of
component?
- Tushar.
On Tue, Dec 1, 2015 at 8:34 AM, Vlad Rozov <[email protected]> wrote:
+1. Modules and operators should be interchangeable. It should be possible
for an operator designer to replace it with a module without requirement to
recompile all applications that use the existing operator. From application
designer view both operators and modules may be considered as a black box
with a private implementation that may be monolithic (operator) or
distributed (module).
Note that a Java class may implement both Operator and Module interface
and be added to a DAG using addOperator() method. At run-time (Logical
Plan->Physical Plan) Apex platform should handle such classes as modules
and may not depend on how a module was added to a DAG, IMO.
Thank you,
Vlad
On 11/30/15 08:41, Tushar Gosavi wrote:
addOperator method in DAG interface takes object of type Operator, unless
we make Module as subclass of Operator we can not use the same API.
- Tushar.
On Mon, Nov 30, 2015 at 9:51 PM, Thomas Weise <[email protected]>
wrote:
+1
If there are really module specific constructs, they can be added later.
Why is the same not supported in the Java API, i.e. why not allow a
module
to be added like an operator and handle the distinction in the
implementation?
On Mon, Nov 30, 2015 at 8:08 AM, Tushar Gosavi <[email protected]>
wrote:
Hi All,
I am going to add a feature to allow use of modules in property file and
json api.
I am planing to use same "operators" array for adding modules as well as
operators. The reasoning behind is that user don't have to worry about
the
component being added to the DAG is actually an operator or a module, as
the method to specify their properties and to create instance of them is
same. And it results in less complex syntax for writing application
using
json/property file.
For example the sample Application specified using Json will look like
below.
```json
{
"operators": [
{
"name": "operator1",
"class": "com.datatorrent.lib.operator.Input",
"properties": {
"property1": "value1"
}
},
{
"name": "module1",
"class": "com.datatorrent.module.Module1",
"properties": {
"property1": "value1"
}
},
],
"streams": [
{
"name": "s1",
"source": {
"operatorName": "operator1",
"portName": "output"
},
"sinks": [
{
"operatorName": "module",
"portName": "input"
}
]
}
]
}
```
In above example "module1" is of type Module. While processing the Json
if
the
class of component is of type Module then we will use addModule api to
the
DAG.
Let me know, what do you think about this.
Regards,
- Tushar.
On Tue, Oct 13, 2015 at 6:32 PM, Tushar Gosavi <[email protected]>
wrote:
A module is superset of operator before expansion in logical plan,
except
operator
attribute are not applicable to modules. As Tim suggested we will have
to
add checks
later to validate these attributes settings, and not allow top level
operator and
module to share same name.
The same question comes when we want to add module to application using
property
file API and JSON file API.
dt.operator.<operatorname>.classname=<fqcn of operator>
One approach we can take is, if class is of type module, then use
addModule api
to add module to the dag, else use addOperator api. The initialization
of
object and applying property is generic can be reused by module.
Similarly we have Json API where all operators are specified under
operators array
{
"operators": [
{.. operators .. }
],
"streams": [
{.. streams .. }
]
}
We could add module in the operators array, and based on the object
type
we can
consider it as module or operator. The mechanism of initializing object
and applying
properties remains same.
- Tushar.
