Hi Aarti Flink doesn't support connecting multiple streams with heterogeneous schema ,you can try the below solution
a) If stream A is sending some events make the output of that as String/JsonString. b) If stream B is sending some events make the output of that as String/JsonString. c) Now Using union function you can connect all the streams & use FlatMap or process function to evaluate all these streams against your defined rules. d) For Storing your aggregations and rules you can build your cache layer and pass as a argument to the constructor of that flatmap. On Mon, Jul 2, 2018 at 2:38 PM, Aarti Gupta <aagu...@qualys.com> wrote: > Hi, > > We are currently evaluating Flink to build a real time rule engine that > looks at events in a stream and evaluates them against a set of rules. > > The rules are dynamically configured and can be of three types - > 1. Simple Conditions - these require you to look inside a single event. > Example, match rule if A happens. > 2. Aggregations - these require you to aggregate multiple events. Example, > match rule if more than five A's happen. > 3. Complex patterns - these require you to look at multiple events and > detect patterns. Example, match rule if A happens and then B happens. > > Since the rules are dynamically configured, we cannot use CEP. > > As an alternative, we are using connected streams and the CoFlatMap > function to store the rules in shared state, and evaluate each incoming > event against the stored rules. Implementation is similar to what's > outlined here > <https://data-artisans.com/blog/bettercloud-dynamic-alerting-apache-flink> > . > > My questions - > > 1. Since the CoFlatMap function works on a single event, how do we > evaluate rules that require aggregations across events. (Match rule if more > than 5 A events happen) > 2. Since the CoFlatMap function works on a single event, how do we > evaluate rules that require pattern detection across events (Match rule if > A happens, followed by B). > 3. How do you dynamically define a window function. > > > --Aarti > > > -- > Aarti Gupta <https://www.linkedin.com/company/qualys> > Director, Engineering, Correlation > > > aagu...@qualys.com > T > > > Qualys, Inc. – Blog <https://qualys.com/blog> | Community > <https://community.qualys.com> | Twitter <https://twitter.com/qualys> > > > <https://www.qualys.com/email-banner> > -- *Cheers * *Puneet Kinra* *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com <puneet.ki...@customercentria.com>* *e-mail :puneet.ki...@customercentria.com <puneet.ki...@customercentria.com>*
