Re: Exactly once guarantee with beam, flink and kafka

2023-05-18 Thread Cristian Constantinescu
Hi Ifat, I don't work with Beam and Flink anymore so the below is from memory. When I went through what you are going through, I spent quite some time to get EOS working. While it kinda worked, I ran into several issues which I'll describe below. The issue with EOS is that it uses Kafka

Exactly once guarantee with beam, flink and kafka

2023-05-10 Thread Ifat Afek (Nokia)
Hi, I’m trying to configure exactly-once recovery for pipelines that are running on top of flink with kafka sources and sinks. I wrote two simple pipelines that pass messages from one topic to another (for testing purposes): pipeline1 reads messages from topic_a and writes them to topic_b