Thanks for sharing your ideas with the community. I really like the design document and think that it's a good approach to follow Oracle's SQL extension for pattern matching. Looking forward to having support for SQL with CEP capabilities :-)
Cheers, Till On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <j...@apache.org> wrote: > Hi @Kostas, @Fabian, thank you for your support. > > @Fabian, I totally agree with you that we should focus on SQL first. Let's > keep Table API in mind and discuss that later. > > Regarding to the orderBy() clause, I'm not sure about that. I think it > makes sense to make it required in streaming mode(either order by rowtime > or order by proctime). But CEP also works in batch mode, and not necessary > to order by some column. Nevertheless, we can support CEP on batch SQL > later. > > We are estimating how to implement MATCH_RECOGNIZE with CEP library (with > NFA, CEP operator). And we will output a detailed doc and a prototype in > the next days. > > Regards, > Jark Wu > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <fhue...@gmail.com>: > >> Thanks Dian and Jark for this proposal! >> >> As you wrote, Till and I (and Kostas) have been thinking about this for >> some time but haven't had time to work on this feature. >> I think it would be a great addition and value add for Flink's SQL >> support and Table API. >> >> I read the proposal and think it is very good. We might need to add a bit >> more details, esp. when planning the concrete steps of the implementation. >> >> A few comments to the proposal: >> - IMO, the development should start focusing on SQL and its semantics. >> Pattern support for the Table API should be added later. We followed that >> approach for the OVER windows and I think it worked quiet well. >> - We probably want to reuse as much as possible from the CEP library. >> That means we need to check if the semantics of the CEP library and >> Oracle's PATTERN syntax are aligned (or how we can express the PATTERN >> semantics with the CEP library). This should be one of the first steps, IMO. >> - I would make the orderBy() clause required. In regular SQL rows have no >> order, so we need to make that explicit (this would also be consistent with >> the OVER windows). >> >> Let me know what you think. >> >> Best, Fabian >> >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <k.klou...@data-artisans.com>: >> >>> Thanks a lot for opening the discussion! >>> >>> This is a really interesting idea that has been in our heads >>> since the first implementation of the CEP library. >>> >>> A big +1 for moving forward with this. >>> >>> And as for the design document, I will definitely have a look >>> and comment there. >>> >>> Kostas >>> >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <j...@apache.org> wrote: >>> >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas >>> >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <j...@apache.org>: >>> >>>> Hi devs, >>>> >>>> Dian and me and our teammates have investigated this for a long time. >>>> We think consolidating Flink SQL and CEP is an exciting thing for Flink. >>>> It'll make SQL more powerful and give users the ability to easily and >>>> quickly build CEP applications. And I find Flink community has also talked >>>> about this idea before, such as the mailing list [1] and [2] and Fabian & >>>> Till's talk in Flink Forward 2016 [3]. >>>> >>>> I think THIS IS THE POINT to bring up this topic again. Because we >>>> already have pattern matching foundation in Flink CEP library, and Stream >>>> SQL is ready now and Calcite has partially supported pattern matching >>>> syntax! We also drafted a design doc about how to integrate SQL and CEP, >>>> and how to support CEP on Table API. https://docs.google.com/docume >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing >>>> >>>> >>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it >>>> would be great to hear your response. >>>> >>>> >>>> What do others think? >>>> >>>> >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787 >>>> >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790 >>>> >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics- >>>> cep-two-sides-of-the-same-coin >>>> >>>> >>>> Regards, >>>> >>>> Jark Wu >>>> >>>> >>>> >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <dia...@apache.org>: >>>> >>>>> Hi everyone, >>>>> >>>>> Flink's CEP library is a great library for complex event processing, >>>>> more >>>>> and more customers are expressing their interests in it. But it also >>>>> has >>>>> some limitations that users usually have to write a lot of code even >>>>> for a >>>>> very simple pattern match use case as it currently only supports the >>>>> Java >>>>> API. >>>>> We have investigated some popular CEP products such as esper [1] and >>>>> siddhi >>>>> [2] and found that most of these CEP products support SQL-like >>>>> expressions >>>>> such as EPL to describe the match pattern. But these solutions also >>>>> have >>>>> the drawbacks that the pattern match languages are not standard SQL, >>>>> the >>>>> learn curve is steep for users and it's impossible to integrate them >>>>> into >>>>> the Flink Table API & SQL. >>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern >>>>> recognition clause match_recognize which is a pattern recognition >>>>> clause >>>>> proposed in this paper [4]. It proposes a set of new syntaxes to define >>>>> match pattern in sql expression. Calcite already supports part of this >>>>> standard [5]. I think it will be of great value to support expressing >>>>> pattern recognition clause with match_recognize clause by integrating >>>>> it >>>>> with Flink Table API & SQL and the Flink CEP library. Any thoughts? >>>>> >>>>> [1] http://www.espertech.com >>>>> [2] https://github.com/wso2/siddhi >>>>> [3] >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql- >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531 >>>>> [4] >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat >>>>> tern-recogniton-11.pdf >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570 >>>>> >>>>> Best Regards, >>>>> Dian >>>>> >>>> >>>> >>> >>> >> >
