Re: Is there expiration for committed Offset in the partition
It is an know issue. You can increase the retention time for stored offsets via configs thought. There is already an open PR to fix this issue: https://issues.apache.org/jira/browse/KAFKA-4682 -Matthias On 6/1/18 2:00 AM, Dinesh Subramanian wrote: > Hi M. Manna, > > Am planning to store outside kafka.. will this be a solution ? > > > *Thanks & Regards,* > > *Dinesh S* > > On Fri, Jun 1, 2018 at 2:05 PM, M. Manna wrote: > >> This can happen for two reasons: >> >> 1) Your offsets are expired and removed. So your consumers don't know where >> to start from - earliest means "Start from the beginning" >> 2) You are actually starting as part of a totally new consumer group - in >> which case it's as designed too - start from the beginning. >> >> I would check your offset retention policy by size/time - and tune that if >> necessary. >> >> On 1 June 2018 at 09:03, Dinesh Subramanian >> wrote: >> >>> Hi, >>> >>> Facing duplication in below scenario >>> >>> Last commit is happened in 3 days back in the consumer, after that no >>> messages produced in the topic. so no commits.. >>> so after 3 days am stopping and restarting the consumer.. this time i >>> faced duplication issue in the consumer as i have this consumer >>> property "*auto.offset.reset >>> = earliest*", It is consumed again from the beginning.. any helps will be >>> appreciated. >>> >>> *Thanks & Regards,* >>> >>> *Dinesh S* >>> >> > signature.asc Description: OpenPGP digital signature
Re: Is there expiration for committed Offset in the partition
Hi M. Manna, Am planning to store outside kafka.. will this be a solution ? *Thanks & Regards,* *Dinesh S* On Fri, Jun 1, 2018 at 2:05 PM, M. Manna wrote: > This can happen for two reasons: > > 1) Your offsets are expired and removed. So your consumers don't know where > to start from - earliest means "Start from the beginning" > 2) You are actually starting as part of a totally new consumer group - in > which case it's as designed too - start from the beginning. > > I would check your offset retention policy by size/time - and tune that if > necessary. > > On 1 June 2018 at 09:03, Dinesh Subramanian > wrote: > > > Hi, > > > > Facing duplication in below scenario > > > > Last commit is happened in 3 days back in the consumer, after that no > > messages produced in the topic. so no commits.. > > so after 3 days am stopping and restarting the consumer.. this time i > > faced duplication issue in the consumer as i have this consumer > > property "*auto.offset.reset > > = earliest*", It is consumed again from the beginning.. any helps will be > > appreciated. > > > > *Thanks & Regards,* > > > > *Dinesh S* > > >
Re: Is there expiration for committed Offset in the partition
This can happen for two reasons: 1) Your offsets are expired and removed. So your consumers don't know where to start from - earliest means "Start from the beginning" 2) You are actually starting as part of a totally new consumer group - in which case it's as designed too - start from the beginning. I would check your offset retention policy by size/time - and tune that if necessary. On 1 June 2018 at 09:03, Dinesh Subramanian wrote: > Hi, > > Facing duplication in below scenario > > Last commit is happened in 3 days back in the consumer, after that no > messages produced in the topic. so no commits.. > so after 3 days am stopping and restarting the consumer.. this time i > faced duplication issue in the consumer as i have this consumer > property "*auto.offset.reset > = earliest*", It is consumed again from the beginning.. any helps will be > appreciated. > > *Thanks & Regards,* > > *Dinesh S* >
Re: Is there expiration for committed Offset in the partition
You should just recommit the same offsets sooner than every 24 hours (or whatever your commit topic retention period is set to). The expiry of offsets is based on the timestamp of the commits. -hans > On Jun 1, 2018, at 1:03 AM, Dinesh Subramanian > wrote: > > Hi, > > Facing duplication in below scenario > > Last commit is happened in 3 days back in the consumer, after that no > messages produced in the topic. so no commits.. > so after 3 days am stopping and restarting the consumer.. this time i > faced duplication issue in the consumer as i have this consumer > property "*auto.offset.reset > = earliest*", It is consumed again from the beginning.. any helps will be > appreciated. > > *Thanks & Regards,* > > *Dinesh S*
Is there expiration for committed Offset in the partition
Hi, Facing duplication in below scenario Last commit is happened in 3 days back in the consumer, after that no messages produced in the topic. so no commits.. so after 3 days am stopping and restarting the consumer.. this time i faced duplication issue in the consumer as i have this consumer property "*auto.offset.reset = earliest*", It is consumed again from the beginning.. any helps will be appreciated. *Thanks & Regards,* *Dinesh S*
