Hi Fabian, We have evaluated the missing features of Flink CEP roughly, it should not be quite difficult to support them. Kostas, Dawid, what's your thought?
For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs and start to work right now or we should wait until the release of calcite 1.13? Btw, could you help to add me(dian.fu) to the contributor list, then I can assign the JIRAs to myself? Thanks a lot. Best regards, Dian On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi Jark, > > Thanks for updating the design doc and sharing your prototype! > I didn't look at the code in detail, but the fact that it is less than 1k > LOC is very promising. It seems that most of the complex CEP logic can be > reused :-) > Adding a dependency on flink-cep should be fine, IMO. It is a very slim > library with almost none external dependencies. > > Regarding the missing features of Flink CEP that you listed in the design > doc, it would be good to get some in put from Kostas and Dawid which are > the main contributors to CEP. > Do you have already plans regarding some of the missing features or can you > assess how hard it would be to integrate them? > > Cheers, Fabian > > Btw. The Calcite community started a discussion about releasing Calcite > 1.13. So, the missing features might soon be available. > > 2017-06-12 14:25 GMT+02:00 Jark Wu <j...@apache.org>: > > > Hi guys, > > > > Good news! We have made a prototype for integrating CEP and SQL. See this > > link > > https://github.com/apache/flink/compare/master... > > wuchong:cep-on-sql?expand=1 > > > > > > You can check CepITCase to try the simple CQL example. > > > > Meanwhile, we updated our design doc with additional implementation > detail, > > including how > > to translate MATCH_RECOGNIZE into CEP API, and the features needed to add > > to Flink CEP, > > and the implementation plan. See the document > > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3 > > > > In the prototype, we make flink-table dependency on flink-cep. Do you > think > > that is fine? > > > > What do you think about the prototype and the design doc ? > > > > Any feedbacks are welcome! > > > > Cheers, > > Jark Wu > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <trohrm...@apache.org>: > > > > > Thanks for sharing your ideas with the community. I really like the > > design > > > document and think that it's a good approach to follow Oracle's SQL > > > extension for pattern matching. Looking forward to having support for > SQL > > > with CEP capabilities :-) > > > > > > Cheers, > > > Till > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <j...@apache.org> wrote: > > > > > > > Hi @Kostas, @Fabian, thank you for your support. > > > > > > > > @Fabian, I totally agree with you that we should focus on SQL first. > > > Let's > > > > keep Table API in mind and discuss that later. > > > > > > > > Regarding to the orderBy() clause, I'm not sure about that. I think > it > > > > makes sense to make it required in streaming mode(either order by > > rowtime > > > > or order by proctime). But CEP also works in batch mode, and not > > > necessary > > > > to order by some column. Nevertheless, we can support CEP on batch > SQL > > > > later. > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP library > > (with > > > > NFA, CEP operator). And we will output a detailed doc and a prototype > > in > > > > the next days. > > > > > > > > Regards, > > > > Jark Wu > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <fhue...@gmail.com>: > > > > > > > >> Thanks Dian and Jark for this proposal! > > > >> > > > >> As you wrote, Till and I (and Kostas) have been thinking about this > > for > > > >> some time but haven't had time to work on this feature. > > > >> I think it would be a great addition and value add for Flink's SQL > > > >> support and Table API. > > > >> > > > >> I read the proposal and think it is very good. We might need to add > a > > > bit > > > >> more details, esp. when planning the concrete steps of the > > > implementation. > > > >> > > > >> A few comments to the proposal: > > > >> - IMO, the development should start focusing on SQL and its > semantics. > > > >> Pattern support for the Table API should be added later. We followed > > > that > > > >> approach for the OVER windows and I think it worked quiet well. > > > >> - We probably want to reuse as much as possible from the CEP > library. > > > >> That means we need to check if the semantics of the CEP library and > > > >> Oracle's PATTERN syntax are aligned (or how we can express the > PATTERN > > > >> semantics with the CEP library). This should be one of the first > > steps, > > > IMO. > > > >> - I would make the orderBy() clause required. In regular SQL rows > have > > > no > > > >> order, so we need to make that explicit (this would also be > consistent > > > with > > > >> the OVER windows). > > > >> > > > >> Let me know what you think. > > > >> > > > >> Best, Fabian > > > >> > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas < > > k.klou...@data-artisans.com > > > >: > > > >> > > > >>> Thanks a lot for opening the discussion! > > > >>> > > > >>> This is a really interesting idea that has been in our heads > > > >>> since the first implementation of the CEP library. > > > >>> > > > >>> A big +1 for moving forward with this. > > > >>> > > > >>> And as for the design document, I will definitely have a look > > > >>> and comment there. > > > >>> > > > >>> Kostas > > > >>> > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <j...@apache.org> wrote: > > > >>> > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas > > > >>> > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <j...@apache.org>: > > > >>> > > > >>>> Hi devs, > > > >>>> > > > >>>> Dian and me and our teammates have investigated this for a long > > time. > > > >>>> We think consolidating Flink SQL and CEP is an exciting thing for > > > Flink. > > > >>>> It'll make SQL more powerful and give users the ability to easily > > and > > > >>>> quickly build CEP applications. And I find Flink community has > also > > > talked > > > >>>> about this idea before, such as the mailing list [1] and [2] and > > > Fabian & > > > >>>> Till's talk in Flink Forward 2016 [3]. > > > >>>> > > > >>>> I think THIS IS THE POINT to bring up this topic again. Because we > > > >>>> already have pattern matching foundation in Flink CEP library, and > > > Stream > > > >>>> SQL is ready now and Calcite has partially supported pattern > > matching > > > >>>> syntax! We also drafted a design doc about how to integrate SQL > and > > > CEP, > > > >>>> and how to support CEP on Table API. > https://docs.google.com/docume > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp= > sharing > > > >>>> > > > >>>> > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, > > it > > > >>>> would be great to hear your response. > > > >>>> > > > >>>> > > > >>>> What do others think? > > > >>>> > > > >>>> > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787 > > > >>>> > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790 > > > >>>> > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics- > > > >>>> cep-two-sides-of-the-same-coin > > > >>>> > > > >>>> > > > >>>> Regards, > > > >>>> > > > >>>> Jark Wu > > > >>>> > > > >>>> > > > >>>> > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <dia...@apache.org>: > > > >>>> > > > >>>>> Hi everyone, > > > >>>>> > > > >>>>> Flink's CEP library is a great library for complex event > > processing, > > > >>>>> more > > > >>>>> and more customers are expressing their interests in it. But it > > also > > > >>>>> has > > > >>>>> some limitations that users usually have to write a lot of code > > even > > > >>>>> for a > > > >>>>> very simple pattern match use case as it currently only supports > > the > > > >>>>> Java > > > >>>>> API. > > > >>>>> We have investigated some popular CEP products such as esper [1] > > and > > > >>>>> siddhi > > > >>>>> [2] and found that most of these CEP products support SQL-like > > > >>>>> expressions > > > >>>>> such as EPL to describe the match pattern. But these solutions > also > > > >>>>> have > > > >>>>> the drawbacks that the pattern match languages are not standard > > SQL, > > > >>>>> the > > > >>>>> learn curve is steep for users and it's impossible to integrate > > them > > > >>>>> into > > > >>>>> the Flink Table API & SQL. > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern > > > >>>>> recognition clause match_recognize which is a pattern recognition > > > >>>>> clause > > > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes to > > > define > > > >>>>> match pattern in sql expression. Calcite already supports part of > > > this > > > >>>>> standard [5]. I think it will be of great value to support > > > expressing > > > >>>>> pattern recognition clause with match_recognize clause by > > integrating > > > >>>>> it > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any > thoughts? > > > >>>>> > > > >>>>> [1] http://www.espertech.com > > > >>>>> [2] https://github.com/wso2/siddhi > > > >>>>> [3] > > > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql- > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm# > CQLLR1531 > > > >>>>> [4] > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat > > > >>>>> tern-recogniton-11.pdf > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570 > > > >>>>> > > > >>>>> Best Regards, > > > >>>>> Dian > > > >>>>> > > > >>>> > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > > > > >
