Hi Team,
I'm trying Flink for the first time and encountered an issue that I would
like to discuss and understand if there is a way to achieve my use case
with Flink.
*Use case:* I need to perform unbounded stream joins on multiple data
streams by listening to different Kafka topics. I have a sce
Hi Eva,
I'm not 100% sure if your use case can be solved with SQL. JOIN in SQL
always joins an incoming record with all previous arrived records. Maybe
Jark in CC has some idea?
It might make sense to use the DataStream API instead with a connect()
and CoProcessFunction where you can simply
Hi Eva,
Correct me If i'm wrong. You have an unbounded Task stream and you
want to enrich the User info to the task event. Meanwhile, the User table
is also changing by the time, so you basically want that when task event
comes, join the latest data of User table and emit the results. Even if the
Hi Eva,
If I understand correctly,
1) the user stream is a changelog stream which every record is a upsert
with a primary key, and you only want to join the latest one
2) if the user record is updated, you want to re-trigger the join
(retract&update previous joined result)
If this is your require
Thanks everyone for the replies.
@Jark,
This is helpful, my code is currently in 1.8 version and I'll upgrade the
code to 1.9 and give it a try.
Couple of follow-up questions:
1. I need to perform Deduplication on Task table as well. Would above query
work well on two Deduplicated tables, "Lates
Hi Reva,
I'm glad to see it can help you.
Quick answers for your questions:
1) Yes, it works. You can deduplicate Task table in the same way using
ROW_NUMBER().
2) Yes. It is a stream-stream join which will be triggered for new messages
from both sides.
Best,
Jark
On Sat, 28 Dec 2019 at 01:02,
