[
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401883#comment-15401883
]
Tushar Gosavi commented on APEXCORE-408:
----------------------------------------
I have dome some initial prototype which allows stat listener to
specify dag changes, and the dag changes are applied asynchronously.
The changes involved are
- Add DagChangeSet object which is inherited from DAG, supporting
methods to remove operator and streams.
- The stat listener will return this object in Response, and platform
will apply changes specified in response to the DAG.
The Apex changes
https://github.com/apache/apex-core/compare/master...tushargosavi:scheduler?expand=1
The correspondign Demo application, which one operator monitors the
directory for files, and launch the wordcount DAG in
same application master when files are available.
https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/FileStatListenerSameDag.java
Let me know if this type of API is acceptable for modifying dag through java
API.
> Ability to schedule Sub-DAG from running application
> ----------------------------------------------------
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
> Issue Type: Sub-task
> Reporter: Thomas Weise
> Assignee: Tushar Gosavi
>
> Today it is possible to add operators to a running application from the
> CLI/client. It should be possible to do this from within the application
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by
> application specific logic. It will enable use cases such as batch
> applications that perform multiple stages of processing where not all
> resources are required at the same time or the logic of subsequent steps
> depends on the execution of previous steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
