Hi Sunitha,
what you are describing is a typical streaming enrichment. We need to
enrich the stream with some data from a database. There are different
strategies to handle this:
1) You are querying the database for every record. This is usually not
what you want because it would slow down your pipeline due to the
communication latenties to your database. It would also cause a lot of
pressure to the database in general.
2) You only query database from time to time and store the latest value
in a ProcessFunction ValueState or MapState.
3) You stream in the values as well and use connect() [1].
In any case, I think CEP might not be useful for this case. If you
really want to do option 1, it might make sense to also checkout the SQL
API of Flink because it offers different kind of joins with very good
abstraction. `Join with a Temporal Table` offers a JDBC connector for
lookups in your database.
If you like to use DataStream API, I would also recommend the Pattern
slides here [3] (unfortunately you have to register first).
Regards,
Timo
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/learn-flink/etl.html#connected-streams
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/streaming/joins.html
[3] https://training.ververica.com/decks/patterns/
On 07.09.20 17:25, s_penakalap...@yahoo.com wrote:
Hi All,
I am new to Flink, request your help!!!
My scenario :
1> we receive Json messages at a very high frequency like 10,000
messages / second
2> we need to raise an Alert for a particular user if there is any
breach in threshold value against each attribute in Json.
3> These threshold values are part of my database table and can be
frequently updated by different user.
4> In realtime I would like to know how to get latest data from the
database.
I tried using Flink CEP Pattern approach to generate alert. I would like
to get some inputs on how I can implement the realtime lookup tables in
Flink Java while monitoring alert, any sample code reference.
Also for such scenarios do you recommend to use Flink CEP approach or we
need to use Process function approach.
Regards,
Sunitha.
