Hi, Siyuan mentioned that we definitely want to have some CEP model supported. Translate a CEP model explained here ( https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf) or Oracle CEP to Apex DAG would make many developers' life easier to use Apex.
I also believe this can make the developers' life easier. I see such approach is feasible at a few different levels and would like to hear your input as well. I'm making some assumptions on how Apex work as I'm still getting more familiar with it. Feel free to correct me when I take the wrong direction (and I will do it)! :) What the white paper and Oracle implements - we may want to include Apama and Esper to the list - is an EPL, SQL-like language to work on streams of data, that I like to refer as Event Stream Processing (ESP), it's not really my definition, though. As in Apex, we will have an ordered stream of data/events that we will work with. ESP is a subset of CEP, where we have a "cloud" of data/events, and we infer from it. The ways I see we can add these features to Apex: 1. Malhar Pattern Matching Compute Operator - I found a mention in the documentation, but could not locate it in the source code. We could add an EPL abstraction layer that works within one DAG. 2. EPL for DAG definition - I believe this approach is closer to what Siyuan has in mind. We use an EPL to write a DAG and potentially do pattern matching across DAGs. This abstraction layer would also need to include Malhar operators. 3. Malhar CEP Compute Operator - If it makes sense to attach an in-memory storage to Apex, we could pattern matching against a set of events that happened and are stored for a certain amount of time as well as we could match those with incoming streams. All three ways are non-mutually exclusive, which means all three can co-exist and complement each other. Let me know your thoughts and we can dig deeper into each of them. Thanks, Rodrigo Abreu.
