[
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574913#comment-15574913
]
ASF GitHub Bot commented on APEXCORE-408:
-----------------------------------------
GitHub user tushargosavi opened a pull request:
https://github.com/apache/apex-core/pull/410
APEXCORE-408: Ability to schedule Sub-DAG from running application.
Pull request for dynamic dag modification through stats listener. It
provides following
functionality
- StatsListener can access the opearator name for easily detecting which
opearator stats are being processed.
- StatsListener can create a instance of object through which it can submit
dag modifications to the engine.
- StatsListener can return dag changes as a response to engine.
- PlanModifier is modified to take a DAG and apply it on the existing
running DAG and deploy the changes.
The following functionality is not working yet.
- The new opearator does not start from the correct windowId
(https://issues.apache.org/jira/browse/APEXCORE-532)
- Relanched application failed to start when it was killed after dynamic
dag modification.
- There is no support for resuming operator from previous state when they
were removed. This could be achived through
readig state through external storage on setup.
- persist operator support is not present for newly added streams.
- Not all parts are covered through tests.
The demo application using the feature is available at
https://github.com/tushargosavi/apex-dynamic-scheduling
There are two variations of WordCount application. The first variation
detects the presence of
new files start a disconnected DAG to process the data.
(https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/WordCountApp.java)
The second application
(https://github.com/tushargosavi/apex-dynamic-scheduling/blob/master/src/main/java/com/datatorrent/wordcount/ExtendApp.java),
initially only one reader operator is running in the DAG, and provides
pendingFiles as auto-metric to stat listener.
On detecting pending files it attaches splitter counter and output operator
to the read operator. Once files are processed the splitter, counter and
output operators are removed and added back again if new data files are
added into the directory.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tushargosavi/incubator-apex-core APEXCORE-408
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-core/pull/410.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #410
----
commit 5cecd029463de021469afb20650e3afbf75fd088
Author: Tushar R. Gosavi <[email protected]>
Date: 2016-08-08T10:24:41Z
APEXCORE-408: Ability to schedule Sub-DAG from running application.
----
> Ability to schedule Sub-DAG from running application
> ----------------------------------------------------
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
> Issue Type: Sub-task
> Reporter: Thomas Weise
> Assignee: Tushar Gosavi
> Attachments: Dynamic DAG Changes.docx
>
>
> Today it is possible to add operators to a running application from the
> CLI/client. It should be possible to do this from within the application
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by
> application specific logic. It will enable use cases such as batch
> applications that perform multiple stages of processing where not all
> resources are required at the same time or the logic of subsequent steps
> depends on the execution of previous steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
