Re: Kafka consumer offset info lost

2017-01-12 Thread Mahendra Kariya
Producers were publishing data for the topic. And consumers were also connected, sending heartbeat pings every 100 ms. On Thu, 12 Jan 2017 at 17:15 Michael Freeman wrote: > If the topic has not seen traffic for a while then Kafka will remove the > stored offset. When your consumer reconnects K

Re: Kafka consumer offset info lost

2017-01-12 Thread Michael Freeman
If the topic has not seen traffic for a while then Kafka will remove the stored offset. When your consumer reconnects Kafka no longer has the offset so it will reprocess from earliest. Michael > On 12 Jan 2017, at 11:13, Mahendra Kariya wrote: > > Hey All, > > We have a Kafka cluster hosted

Kafka consumer offset info lost

2017-01-12 Thread Mahendra Kariya
Hey All, We have a Kafka cluster hosted on Google Cloud. There was some network issue on the cloud and suddenly, the offset for a particular consumer group got reset to earliest and all of a sudden the lag was in millions. We aren't able to figure out what went wrong. Has anybody faced the same/si