Re: FlinkKafkaConsumer - Broadcast - Initial Load

2021-03-26 Thread Robert Metzger
Hey Sandeep, (Maybe this thread is also relevant: https://lists.apache.org/thread.html/7d56267d4c2344ccb5a774896682d0a3efb38c1c215ef3500c3569a2%40%3Cuser.flink.apache.org%3E ) > My question is how do I initialise the pipeline for the first set of records in the database? i.e. those that are not

FlinkKafkaConsumer - Broadcast - Initial Load

2021-03-25 Thread Sandeep khanzode
Hi, I have a master/reference data that needs to come in through a FlinkKafkaConsumer to be broadcast to all nodes and subsequently joined with the actual stream for enriching content. The Kafka consumer gets CDC-type records from database changes. All this works well. My question is how do