Re: KafkaConsumer taking time to determine offset

2024-06-12 Thread Greg Harris
Hi Akash, Thanks for your question. When you say: > kafkaconsumer misses some of the records written into the kafka source topic. Do you mean that there are records in the source topic that are never returned by Consumer#poll(), SourceTask#poll(), or written to the target topic by KafkaConnect?

KafkaConsumer taking time to determine offset

2024-06-12 Thread Akash Dhiman
Hello, we have a usecase where we use kafkaConsumer in a SourceTask of a source connector to poll messages from an aws msk. if we try to produce data into the source topic immediately after the connector gets into the running state we sometimes notice that kafkaconsumer misses some of the records