Hi Surendra, where does your historical data reside? In a database? Files? Behind a REST API?
Depending on the answer and which API you use, the AsyncIO [1] operator (DataStream API) or a Lookup Table Join [2] might be good options. There are other ways, too. A while back I did a webinar [3] on this. It is a bit outdated, but still covers a few concepts and ideas around this. Cheers, Konstantin [1] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/operators/asyncio/ [2] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/sql/queries/joins/#lookup-join [3] https://www.youtube.com/watch?v=cJS18iKLUIY&t=2s Am Mo., 6. Juni 2022 um 07:36 Uhr schrieb Surendra Lalwani < surendra.lalw...@swiggy.in>: > Hi Team, > > Since Flink does not support Stream-Batch Join, we need to perform a > lookup with some historical data, what could be the best way in Flink to do > that. I am aware of one such approach using Broadcasted stream, not sure if > there are any other proper solutions. Any help in this will be helpful. > > Regards , > Surendra Lalwani > > > ------------------------------ > IMPORTANT NOTICE: This e-mail, including any attachments, may contain > confidential information and is intended only for the addressee(s) named > above. If you are not the intended recipient(s), you should not > disseminate, distribute, or copy this e-mail. Please notify the sender by > reply e-mail immediately if you have received this e-mail in error and > permanently delete all copies of the original message from your system. > E-mail transmission cannot be guaranteed to be secure as it could be > intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > contain viruses. Company accepts no liability for any damage or loss of > confidential information caused by this email or due to any virus > transmitted by this email or otherwise. -- https://twitter.com/snntrable https://github.com/knaufk
