Re: [Discuss] Typesafe Stream Processing DSL + General GroupByKey

2015-12-14 Thread Hao Chen
Edward, Type-safe API currently is mainly for better development experience, not fully general-purpose monitoring. For example, when the developer is required to implement groupBy "user", the implementation have to override lots of storm-specified code like: https://github.com/haoch/incubator-e

Re: [Discuss] Typesafe Stream Processing DSL + General GroupByKey

2015-12-13 Thread Zhang, Edward (GDI Hadoop)
This is a nice progress. good to see we have type safe fluent API. Could you please explain a little bit more how typesafe would help generic purpose monitoring? In general purpose monitoring, do we need input type/output type defined? How user describe their types from source to each processing

[Discuss] Typesafe Stream Processing DSL + General GroupByKey

2015-12-07 Thread Hao Chen
JIRA: https://issues.apache.org/jira/browse/EAGLE-66 Pull Request: https://github.com/apache/incubator-eagle/pull/17 *Main Changes* 1. *Typesafe API:* Currently the stream processing API is not type-safe (Type info are erased by stream framework), all programming interfaces for developer are face

Re: [Discuss] Typesafe Stream Processing DSL + General GroupByKey

2015-12-07 Thread Hao Chen
Missed another change about *Configuration*: https://github.com/haoch/incubator-eagle/blob/EAGLE-66-TYPESAFE/eagle-core/eagle-data-process/eagle-stream-process-api/src/main/scala/org/apache/eagle/datastream/core/Configuration.scala *val env = ExecutionEnvironments.get[StormExecutionEnvironment](a

Re: [Discuss] Typesafe Stream Processing DSL + General GroupByKey

2015-12-07 Thread Hao Chen
*TODO* ✓ Typesafe API ✓ groupByKey ☐ Test compatibility for old application ☐ Refactor package to seperate core/storm/spark/others ☐ Support KeyValue based grouping for old application and existing executor / spout ☐ Support groupByKey(fields: String*) ☐ Refactor Alert Executor with groupByKey and