Thanks a lot for opening the discussion! This is a really interesting idea that has been in our heads since the first implementation of the CEP library.
A big +1 for moving forward with this. And as for the design document, I will definitely have a look and comment there. Kostas > On Jun 7, 2017, at 10:05 AM, Jark Wu <j...@apache.org> wrote: > > Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas > > 2017-06-07 15:42 GMT+08:00 Jark Wu <j...@apache.org <mailto:j...@apache.org>>: > Hi devs, > > Dian and me and our teammates have investigated this for a long time. We > think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll > make SQL more powerful and give users the ability to easily and quickly build > CEP applications. And I find Flink community has also talked about this idea > before, such as the mailing list [1] and [2] and Fabian & Till's talk in > Flink Forward 2016 [3]. > > I think THIS IS THE POINT to bring up this topic again. Because we already > have pattern matching foundation in Flink CEP library, and Stream SQL is > ready now and Calcite has partially supported pattern matching syntax! We > also drafted a design doc about how to integrate SQL and CEP, and how to > support CEP on Table API. > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing > > <https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing> > > @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it would > be great to hear your response. > > > > What do others think? > > > > [1] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787 > > <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787> > [2] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790 > > <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790> > [3] > https://www.slideshare.net/tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin > > <https://www.slideshare.net/tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin> > > Regards, > > Jark Wu > > > > > 2017-06-07 13:50 GMT+08:00 Dian Fu <dia...@apache.org > <mailto:dia...@apache.org>>: > Hi everyone, > > Flink's CEP library is a great library for complex event processing, more > and more customers are expressing their interests in it. But it also has > some limitations that users usually have to write a lot of code even for a > very simple pattern match use case as it currently only supports the Java > API. > We have investigated some popular CEP products such as esper [1] and siddhi > [2] and found that most of these CEP products support SQL-like expressions > such as EPL to describe the match pattern. But these solutions also have > the drawbacks that the pattern match languages are not standard SQL, the > learn curve is steep for users and it's impossible to integrate them into > the Flink Table API & SQL. > We find that Oracle's CEP solution CQL [3] supports a new pattern > recognition clause match_recognize which is a pattern recognition clause > proposed in this paper [4]. It proposes a set of new syntaxes to define > match pattern in sql expression. Calcite already supports part of this > standard [5]. I think it will be of great value to support expressing > pattern recognition clause with match_recognize clause by integrating it > with Flink Table API & SQL and the Flink CEP library. Any thoughts? > > [1] http://www.espertech.com <http://www.espertech.com/> > [2] https://github.com/wso2/siddhi <https://github.com/wso2/siddhi> > [3] > https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531 > > <https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531> > [4] > http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pattern-recogniton-11.pdf > > <http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pattern-recogniton-11.pdf> > [5] https://issues.apache.org/jira/browse/CALCITE-1570 > <https://issues.apache.org/jira/browse/CALCITE-1570> > > Best Regards, > Dian > >
