Thanks Yunfeng for bringing up this discussion. I have seen that there were many users asking for this feature in the past. So big +1 for this proposal.
Regarding this FLIP, I have the following question: - Does it make sense to rename the method `getLatestRules` into `getRuleChanges`? I assume we want to know the change history of the rules, not only the latest status. Regarding Table API & SQL support: - Generally I agree that we should consider both Table API and DataStream API for new features. It would avoid introducing changes which will be changed immediately. However, I think it depends. For this feature, I just suspect that it could be supported in SQL in reality. There are so many things which may be dynamically changed, e.g. the rule definitions, the partitioning columns, etc. If we want to change them dynamically, we may need to make a lot of extensions to the MATCH_RECOGNIZE statement which comes from SQL standard. Personally I tend to limit the scope of dynamic changing patterns to only DataStream API at least for now. Regarding CEP.rule vs CEP.dynamicPatterns: - I'm also slightly preferring CEP.dynamicPatterns which is more descriptive. Personally I think that it's difficult to tell the difference between CEP.rule and CEP.pattern unless users read the documentation carefully. Regards, Dian On Mon, Dec 13, 2021 at 5:41 PM Nicholas Jiang <nicholasji...@apache.org> wrote: > Hi Konstantin, > > About the renaming for the Rule, I mean that the difference between the > Rule and Pattern is that the Rule not only contains the Pattern, but also > how to match the Pattern, and how to process after matched. If renaming > DynamicPattern, I'm concerned that this name couldn't represent how to > match the Pattern, and how to process after matched but the Rule could > explain this. Therefore I prefer to rename the Rule not the DynamicPattern. > > Best, > Nicholas Jiang > > > On 2021/12/13 08:23:23 Konstantin Knauf wrote: > > Hi Nicholas, > > > > I am not sure I understand your question about renaming. I think the most > > important member of the current Rule class is the Pattern, the > KeySelector > > and PatternProcessFunction are more auxiliary if you will. That's why, I > > think, it would be ok to rename Rule to DynamicPatternHolder although it > > contains more than just a Pattern. > > > > Cheers, > > > > Konstantin > > > > On Mon, Dec 13, 2021 at 9:16 AM Nicholas Jiang <nicholasji...@apache.org > > > > wrote: > > > > > Hi Konstantin, > > > > > > Thanks for your feedback. The point that add a timestamp to each > rule > > > that determines the start time from which the rule makes sense to me. > At > > > present, The timestamp is current time at default, so no timestamp > field > > > represents the start time from which the rule. And about the renaming > rule, > > > your suggestion looks good to me and no any new concept introduces. But > > > does this introduce Rule concept or reuse the Pattern concept for the > > > DynamicPattern renaming? > > > > > > Best, > > > Nicholas Jiang > > > > > > On 2021/12/13 07:45:04 Konstantin Knauf wrote: > > > > Thanks, Yufeng, for starting this discussion. I think this will be a > very > > > > popular feature. I've seen a lot of users asking for this in the > past. > > > So, > > > > generally big +1. > > > > > > > > I think we should have a rough idea on how to expose this feature in > the > > > > other APIs. > > > > > > > > Two ideas: > > > > > > > > 1. In order to make this more deterministic in case of reprocessing > and > > > > out-of-orderness, I am wondering if we can add a timestamp to each > rule > > > > that determines the start time from which the rule should be in > effect. > > > > This can be an event or a processing time depending on the > > > characteristics > > > > of the pipeline. The timestamp would default to Long.MIN_TIMESTAMP > if not > > > > provided, which means effectively immediately. This could also be a > > > follow > > > > up, if you think it will make the implementation too complicated > > > initially. > > > > > > > > 2. I am wondering, if we should name Rule->DynamicPatternHolder or > so and > > > > CEP.rule-> CEP.dynamicPatterns instead (other classes > correspondingly)? > > > > Rule is quite ambiguous and DynamicPattern seems more descriptive to > me. > > > > > > > > On Mon, Dec 13, 2021 at 4:30 AM Nicholas Jiang < > nicholasji...@apache.org > > > > > > > > wrote: > > > > > > > > > Hi Martijn, > > > > > > > > > > IMO, in this FLIP, we only need to introduce the general design > of > > > the > > > > > Table API/SQL level. As for the design details, you can create a > new > > > FLIP. > > > > > And do we need to take into account the support for Batch mode if > you > > > > > expand the MATCH_RECOGNIZE function? About the dynamic rule engine > > > design, > > > > > do you have any comments? This core of the FLIP is about the > multiple > > > rule > > > > > and dynamic rule changing mechanism. > > > > > > > > > > Best, > > > > > Nicholas Jiang > > > > > > > > > > > > > > > > > -- > > > > > > > > Konstantin Knauf > > > > > > > > https://twitter.com/snntrable > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > > >
