Hi there It would be helpful if you could include the code for your pipeline. One suggestion, can you disable the "EXACTLY_ONCE" semantics on the producer. Using EXACTLY_ONCE will leverage Kafka transactions and thus add overhead. I would disable it to see if you still get the same situation.
Also, can you look in the Flink UI for this job and see if checkpoints are in fact being taken? Hope that helps -Hector On Tue, Aug 15, 2023 at 11:36 AM Dennis Jung <inylov...@gmail.com> wrote: > Sorry, I've forgot putting title, so sending again. > > 2023년 8월 15일 (화) 오후 6:27, Dennis Jung <inylov...@gmail.com>님이 작성: > >> (this is issue from Flink 1.14) >> >> Hello, >> >> I've set up following logic to consume messages from kafka, and produce >> them to another kafka broker. For producer, I've configured >> `Semantics.EXACTLY_ONCE` to send messages exactly once. (also setup >> 'StreamExecutionEnvironment::enableCheckpointing' as >> 'CheckpointingMode.EXACTLY_ONCE') >> >> >> -------------------------------------------------------------------------------------------- >> kafka A -> FlinkKafkaConsumer -> ... -> FlinkKafkaProducer -> kafka B >> >> -------------------------------------------------------------------------------------------- >> >> But though I've just produced only 1 message to 'kafka A', consumer >> consumes the same message repeatedly. >> >> When I remove `FlinkKafkaProducer` part and make it 'read only', it does >> not happen. >> >> Could someone suggest a way to debug or fix this? >> >> Thank you. >> >
