Hi Dian & Jark,
I checked out your prototype code, but it didn't pass the CEPITCase test in
the flink-table component.
It turns out that in the `MatchCodeGenerator.scala` file, line 74 should
use `${classOf[IterativeCondition.Context[_]].getCanonicalName}` instead of
`${classOf[IterativeCondition.Context[_]]}`.
I've also read your desgin document and it looks fine to me. Actually, I am
working on the same thing recently, I think maybe we can work together to
push this forward.
Thanks,
Yueting Chen
On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <dia...@apache.org> wrote:
> Hi Fabian,
>
> We have evaluated the missing features of Flink CEP roughly, it should not
> be quite difficult to support them. Kostas, Dawid, what's your thought?
>
> For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs and
> start to work right now or we should wait until the release of calcite
> 1.13?
>
> Btw, could you help to add me(dian.fu) to the contributor list, then I can
> assign the JIRAs to myself? Thanks a lot.
>
> Best regards,
> Dian
>
> On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <fhue...@gmail.com> wrote:
>
> > Hi Jark,
> >
> > Thanks for updating the design doc and sharing your prototype!
> > I didn't look at the code in detail, but the fact that it is less than 1k
> > LOC is very promising. It seems that most of the complex CEP logic can be
> > reused :-)
> > Adding a dependency on flink-cep should be fine, IMO. It is a very slim
> > library with almost none external dependencies.
> >
> > Regarding the missing features of Flink CEP that you listed in the design
> > doc, it would be good to get some in put from Kostas and Dawid which are
> > the main contributors to CEP.
> > Do you have already plans regarding some of the missing features or can
> you
> > assess how hard it would be to integrate them?
> >
> > Cheers, Fabian
> >
> > Btw. The Calcite community started a discussion about releasing Calcite
> > 1.13. So, the missing features might soon be available.
> >
> > 2017-06-12 14:25 GMT+02:00 Jark Wu <j...@apache.org>:
> >
> > > Hi guys,
> > >
> > > Good news! We have made a prototype for integrating CEP and SQL. See
> this
> > > link
> > > https://github.com/apache/flink/compare/master...
> > > wuchong:cep-on-sql?expand=1
> > >
> > >
> > > You can check CepITCase to try the simple CQL example.
> > >
> > > Meanwhile, we updated our design doc with additional implementation
> > detail,
> > > including how
> > > to translate MATCH_RECOGNIZE into CEP API, and the features needed to
> add
> > > to Flink CEP,
> > > and the implementation plan. See the document
> > > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > >
> > > In the prototype, we make flink-table dependency on flink-cep. Do you
> > think
> > > that is fine?
> > >
> > > What do you think about the prototype and the design doc ?
> > >
> > > Any feedbacks are welcome!
> > >
> > > Cheers,
> > > Jark Wu
> > >
> > >
> > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <trohrm...@apache.org>:
> > >
> > > > Thanks for sharing your ideas with the community. I really like the
> > > design
> > > > document and think that it's a good approach to follow Oracle's SQL
> > > > extension for pattern matching. Looking forward to having support for
> > SQL
> > > > with CEP capabilities :-)
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <j...@apache.org> wrote:
> > > >
> > > > > Hi @Kostas, @Fabian, thank you for your support.
> > > > >
> > > > > @Fabian, I totally agree with you that we should focus on SQL
> first.
> > > > Let's
> > > > > keep Table API in mind and discuss that later.
> > > > >
> > > > > Regarding to the orderBy() clause, I'm not sure about that. I think
> > it
> > > > > makes sense to make it required in streaming mode(either order by
> > > rowtime
> > > > > or order by proctime). But CEP also works in batch mode, and not
> > > > necessary
> > > > > to order by some column. Nevertheless, we can support CEP on batch
> > SQL
> > > > > later.
> > > > >
> > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP library
> > > (with
> > > > > NFA, CEP operator). And we will output a detailed doc and a
> prototype
> > > in
> > > > > the next days.
> > > > >
> > > > > Regards,
> > > > > Jark Wu
> > > > >
> > > > >
> > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <fhue...@gmail.com>:
> > > > >
> > > > >> Thanks Dian and Jark for this proposal!
> > > > >>
> > > > >> As you wrote, Till and I (and Kostas) have been thinking about
> this
> > > for
> > > > >> some time but haven't had time to work on this feature.
> > > > >> I think it would be a great addition and value add for Flink's SQL
> > > > >> support and Table API.
> > > > >>
> > > > >> I read the proposal and think it is very good. We might need to
> add
> > a
> > > > bit
> > > > >> more details, esp. when planning the concrete steps of the
> > > > implementation.
> > > > >>
> > > > >> A few comments to the proposal:
> > > > >> - IMO, the development should start focusing on SQL and its
> > semantics.
> > > > >> Pattern support for the Table API should be added later. We
> followed
> > > > that
> > > > >> approach for the OVER windows and I think it worked quiet well.
> > > > >> - We probably want to reuse as much as possible from the CEP
> > library.
> > > > >> That means we need to check if the semantics of the CEP library
> and
> > > > >> Oracle's PATTERN syntax are aligned (or how we can express the
> > PATTERN
> > > > >> semantics with the CEP library). This should be one of the first
> > > steps,
> > > > IMO.
> > > > >> - I would make the orderBy() clause required. In regular SQL rows
> > have
> > > > no
> > > > >> order, so we need to make that explicit (this would also be
> > consistent
> > > > with
> > > > >> the OVER windows).
> > > > >>
> > > > >> Let me know what you think.
> > > > >>
> > > > >> Best, Fabian
> > > > >>
> > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > k.klou...@data-artisans.com
> > > > >:
> > > > >>
> > > > >>> Thanks a lot for opening the discussion!
> > > > >>>
> > > > >>> This is a really interesting idea that has been in our heads
> > > > >>> since the first implementation of the CEP library.
> > > > >>>
> > > > >>> A big +1 for moving forward with this.
> > > > >>>
> > > > >>> And as for the design document, I will definitely have a look
> > > > >>> and comment there.
> > > > >>>
> > > > >>> Kostas
> > > > >>>
> > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <j...@apache.org> wrote:
> > > > >>>
> > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> > > > >>>
> > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <j...@apache.org>:
> > > > >>>
> > > > >>>> Hi devs,
> > > > >>>>
> > > > >>>> Dian and me and our teammates have investigated this for a long
> > > time.
> > > > >>>> We think consolidating Flink SQL and CEP is an exciting thing
> for
> > > > Flink.
> > > > >>>> It'll make SQL more powerful and give users the ability to
> easily
> > > and
> > > > >>>> quickly build CEP applications. And I find Flink community has
> > also
> > > > talked
> > > > >>>> about this idea before, such as the mailing list [1] and [2] and
> > > > Fabian &
> > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > >>>>
> > > > >>>> I think THIS IS THE POINT to bring up this topic again. Because
> we
> > > > >>>> already have pattern matching foundation in Flink CEP library,
> and
> > > > Stream
> > > > >>>> SQL is ready now and Calcite has partially supported pattern
> > > matching
> > > > >>>> syntax! We also drafted a design doc about how to integrate SQL
> > and
> > > > CEP,
> > > > >>>> and how to support CEP on Table API.
> > https://docs.google.com/docume
> > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=
> > sharing
> > > > >>>>
> > > > >>>>
> > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> discussion,
> > > it
> > > > >>>> would be great to hear your response.
> > > > >>>>
> > > > >>>>
> > > > >>>> What do others think?
> > > > >>>>
> > > > >>>>
> > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.c
> > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > >>>>
> > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.c
> > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > > > >>>>
> > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> analytics-
> > > > >>>> cep-two-sides-of-the-same-coin
> > > > >>>>
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>>
> > > > >>>> Jark Wu
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <dia...@apache.org>:
> > > > >>>>
> > > > >>>>> Hi everyone,
> > > > >>>>>
> > > > >>>>> Flink's CEP library is a great library for complex event
> > > processing,
> > > > >>>>> more
> > > > >>>>> and more customers are expressing their interests in it. But it
> > > also
> > > > >>>>> has
> > > > >>>>> some limitations that users usually have to write a lot of code
> > > even
> > > > >>>>> for a
> > > > >>>>> very simple pattern match use case as it currently only
> supports
> > > the
> > > > >>>>> Java
> > > > >>>>> API.
> > > > >>>>> We have investigated some popular CEP products such as esper
> [1]
> > > and
> > > > >>>>> siddhi
> > > > >>>>> [2] and found that most of these CEP products support SQL-like
> > > > >>>>> expressions
> > > > >>>>> such as EPL to describe the match pattern. But these solutions
> > also
> > > > >>>>> have
> > > > >>>>> the drawbacks that the pattern match languages are not standard
> > > SQL,
> > > > >>>>> the
> > > > >>>>> learn curve is steep for users and it's impossible to integrate
> > > them
> > > > >>>>> into
> > > > >>>>> the Flink Table API & SQL.
> > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new
> pattern
> > > > >>>>> recognition clause match_recognize which is a pattern
> recognition
> > > > >>>>> clause
> > > > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes
> to
> > > > define
> > > > >>>>> match pattern in sql expression. Calcite already supports part
> of
> > > > this
> > > > >>>>> standard [5]. I think it will be of great value to support
> > > > expressing
> > > > >>>>> pattern recognition clause with match_recognize clause by
> > > integrating
> > > > >>>>> it
> > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any
> > thoughts?
> > > > >>>>>
> > > > >>>>> [1] http://www.espertech.com
> > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > >>>>> [3]
> > > > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
> > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#
> > CQLLR1531
> > > > >>>>> [4]
> > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
> > > > >>>>> tern-recogniton-11.pdf
> > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > >>>>>
> > > > >>>>> Best Regards,
> > > > >>>>> Dian
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> >
>
