Re: Handling stale data enrichment

2020-04-24 Thread Vinay Patil
Hi Konstantin, Thank you for your answer. Yes, we have timestamps in the subscription stream >the disadvantage that you do not make any progress until you see fresh subscription data. Is this the desired behavior for your use case? No, this is not acceptable. Reason being the subscription data m

Re: Handling stale data enrichment

2020-04-23 Thread Konstantin Knauf
Hi Vinay, I assume your subscription updates also have a timestamp and a watermark. Otherwise, there is no way for Flink to tell that the subscription updates are late. If you use a "temporal table "-style join to join the two streams, and you do not receive any subscription updates for 2 hours,

Handling stale data enrichment

2020-04-23 Thread Vinay Patil
Hi, I went through Konstantin webinar on 99 ways you can do enrichment. One thing I am failing to understand is how do we efficiently handle stale data enrichment. Context: Let's say I want to enrich user data with the subscription data. Here subscription data is acting as reference data and will