[
https://issues.apache.org/jira/browse/EAGLE-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610420#comment-15610420
]
ASF GitHub Bot commented on EAGLE-348:
--------------------------------------
GitHub user r7raul1984 opened a pull request:
https://github.com/apache/incubator-eagle/pull/568
[EAGLE-348] Alert engine base on spark streaming
- EAGLE-424
- Support kafka add/remove topic without requiring restart of the alert
engine
- Read spec metadata by restful web app
- Create UnitSparkTopologyRunner to demo spark streaming usage
- EAGLE-425
- Use Spark Accumulator to save and recover state
- Use Spark Accumulator to support Hybird Time Window
https://issues.apache.org/jira/browse/EAGLE-348
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/1haodian/incubator-eagle spark-alert-engine
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-eagle/pull/568.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #568
----
commit 390b8b71da490e80ccbf50956fd455fa046a76f8
Author: r7raul1984 <[email protected]>
Date: 2016-10-25T07:35:13Z
[EAGLE-348] Alert engine base on spark streaming
- EAGLE-424
- Support kafka add/remove topic without requiring restart of the alert
engine
- Read spec metadata by restful web app
- Create UnitSparkTopologyRunner to demo spark streaming usage
- EAGLE-425
- Use Spark Accumulator to save and recover state
- Use Spark Accumulator to support Hybird Time Window
https://issues.apache.org/jira/browse/EAGLE-348
commit c09634d5485b7c3d9a5e70c078ac524467465099
Author: r7raul1984 <[email protected]>
Date: 2016-10-26T07:50:26Z
Add license.
commit 84618e4be58354421d3a6e21829aa7f9510950cc
Author: r7raul1984 <[email protected]>
Date: 2016-10-27T02:15:06Z
Use metaresource instead of specresource to get
spoutSpec,routeSpec,alertSpec,publishSpec.
----
> Alert engine base on spark streaming
> ------------------------------------
>
> Key: EAGLE-348
> URL: https://issues.apache.org/jira/browse/EAGLE-348
> Project: Eagle
> Issue Type: New Feature
> Affects Versions: v0.6.0
> Reporter: JiJun Tang
> Assignee: Hao Chen
> Priority: Minor
> Fix For: v0.6.0
>
>
> {noformat}
> New alert engine architecture:
> 1)Metaservice: a) get meta(policies,topologies,streamdefinitions and so on)
> 2)Coordinator: a) pull policies and figure out if polices are changed by
> Metaservice and notify config consumer of config changes by ConfigBusService
> (powered by zk node cache)
> b) placePolicy depend on policy usage on alertbolt
> 3)UnitTopology: a) connect CorrelationSpout(read all topics data from kafka)
> StreamRouterBolt(find which alertbolt to execute) AlertBolt(execute policy on
> stream) AlertPublisherBolt (publish execute result to db,email and so on)
>
> Let the alert engine to work on spark streaming,I think :
> 1) #1 and #2 can be worked on both storm and spark platform, can use
> broadcast to distribute this service
> 2) we should rewirte
> CorrelationSpout,StreamRouterBolt,AlertBolt,AlertPublisherBolt to Dstream and
> connect them as DAG,then submit into spark platform.
> 3) spark can dynamic allocation executor without restart
> 4) in the future spark streaming can support add or delete kafka topic
> without restart
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
