Hi Vishnu, I'd suggest you take a look at the broker configuration value "offsets.retention.minutes".
The consumer offsets are stored in the __consumer_offsets topic. __consumer_offsets is a compacted topic (cleanup.policy=compact), where the key is the combination of the consumer group, the topic, and the partition, and the value is the offset, a timestamp it was committed, and some metadata. Because it is a compacted topic, it would normally never lose data. But imagine if you misconfigured a consumer with the wrong group.id and it started committing offsets before you shut it down and corrected it. Well, that data would "normally" live forever because they're in a compacted topic, but effectively they need to be "garbage collected" at some point. "offsets.retention.minutes" is like a garbage collector for the __consumer_offsets topic. Every "offsets.retention.check.interval.ms", Kafka scans the topic and it adds delete tombstones for any offset that hasn't been seen in "offsets.retention.minutes". Compaction will then eventually remove those values completely. This allows offsets to be cleaned up when they're no longer needed. The default value of "offsets.retention.minutes" is 1440 minutes (24 hours). Mathieu On Mon, Feb 27, 2017 at 4:50 PM, Vishnu Gopal <vishnugo...@live.com> wrote: > Team, > > > I came across an issue where the offset positing is being reset and the > consumer app is receiving the same message in the same offset repeatedly. > this is happing every 3 or 4 days apart. I have purged the topic for now > and re-started the app. but can any body give me a direction on how to > replicate this issue and how to resolve this. Any help is great. > > > Thanks, > > Vishnu >
