Might be easier to handle duplicate messages as opposed to handling long periods of time without messages.
Michael > On 22 Aug 2016, at 15:55, Misra, Rahul <rahul.mi...@altisource.com> wrote: > > Hi, > > Can anybody provide any guidance on the following: > > 1. Given a limited set of groups and consumers, will increasing > 'offsets.retention.minutes' to a high value (say 30 days) cause the > __consumer_offsets topic to bloat unnecessarily or will compaction ensure > that the entries for each key remain limited (which would mean that having a > high 'offsets.retention.minutes' value is not a problem. I would prefer this > option). > > 2. If the consumer calls commitSync() with latest already committed offsets > (which have been committed already but no messages have been received for a > long time after that), will it make an entry to the __consumer_offsets topic > and ensure that the offsets are retained even with a small > 'offsets.retention.minutes'? In our application the dry period (period > without a new message is not well defined in advance). > > > Regards, > Rahul Misra > > > -----Original Message----- > From: Misra, Rahul [mailto:rahul.mi...@altisource.com] > Sent: Sunday, August 21, 2016 12:46 AM > To: Ian Wrigley; users@kafka.apache.org > Subject: RE: Offsets getting lost if no messages sent for a long time > > Hi Ian, > > Thanks for the quick response. Your answer clears things up. > I have some follow up questions though: > > 1. Given a limited set of groups and consumers, will increasing > 'offsets.retention.minutes' to a high value (say 30 days) cause the > __consumer_offsets topic to bloat unnecessarily or will compaction ensure > that the entries for each key remain limited (which would mean that having a > high 'offsets.retention.minutes' value is not a problem. I would prefer this > option). > > 2. If the consumer calls commitSync() with latest already committed offsets > (which have been committed already but no messages have been received for a > long time after that), will it make an entry to the __consumer_offsets topic > and ensure that the offsets are retained even with a small > 'offsets.retention.minutes'? In our application the dry period (period > without a new message is not well defined in advance). > > > Regards, > Rahul Misra > > > > > > -----Original Message----- > From: Ian Wrigley [mailto:i...@confluent.io] > Sent: Sunday, August 21, 2016 12:01 AM > To: users@kafka.apache.org > Subject: Re: Offsets getting lost if no messages sent for a long time > > Since nothing was written to the __consumer_offsets topic for more than its > configured retention period (offsets.retention.minutes, by default 1440 > minutes, or one day), the offset info will be removed. Retention period is > all about when the last offset was written, not the last time a Consumer > looked at a topic. > > You can increase the value of offsets.retention.minutes to ensure that offset > info isn’t cleaned out before more messages are written to a topic and read > by the Consumer (and hence the Consumer updates its offset info in > __consumer_offsets). > > Ian. > > --- > Ian Wrigley > Director, Education Services > Confluent, Inc > >> On Aug 20, 2016, at 11:36 AM, Misra, Rahul <rahul.mi...@altisource.com> >> wrote: >> >> Hi, >> >> I have observed the following scenario (the consumer here has >> 'enable.auto.commit=false' and offsets are committed using commitSync() if >> any messages are received): >> >> 1. Start a consumer (with a specific group.Id) and send some messages >> to its subscribed topic. >> >> 2. The consumer consumes the messages and the group+consumer has an >> entry in the __commit_offsets with the latest offsets for this group and >> consumer. >> >> 3. The consumer will keep polling the topic but don't send any more >> messages to the topic for a long time (longer than one day. The consumer >> keeps polling the topic in a while loop). The default duration for which a >> group's entry is retained in the offsets topic is 1 day. >> >> 4. Now stop the consumer. (there is no other consumer for this group) >> >> 5. Send some more messages to the topic. >> >> 6. Start the consumer (with the same group and consumer id as earlier). >> >> 7. The consumer does not pick up the new messages sent in step 5 as it >> has lost the committed offsets and starts with the 'latest' offsets. >> >> Is this an expected behavior? Or do I have something wrong in the >> configurations? >> Is there a way to ensure that the offsets are retained even if there are no >> messages flowing in? >> I had assumed that if the consumer keeps polling the kafka topic, its >> offsets will be retained even if no messages are received by the >> corresponding topic. >> >> Regards, >> Rahul >> This email message and any attachments are intended solely for the use of >> the addressee. If you are not the intended recipient, you are prohibited >> from reading, disclosing, reproducing, distributing, disseminating or >> otherwise using this transmission. If you have received this message in >> error, please promptly notify the sender by reply email and immediately >> delete this message from your system. This message and any attachments may >> contain information that is confidential, privileged or exempt from >> disclosure. Delivery of this message to any person other than the intended >> recipient is not intended to waive any right or privilege. Message >> transmission is not guaranteed to be secure or free of software viruses. >> *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of the > addressee. If you are not the intended recipient, you are prohibited from > reading, disclosing, reproducing, distributing, disseminating or otherwise > using this transmission. If you have received this message in error, please > promptly notify the sender by reply email and immediately delete this message > from your system. This message and any attachments may contain information > that is confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > *********************************************************************************************************************** > This email message and any attachments are intended solely for the use of the > addressee. If you are not the intended recipient, you are prohibited from > reading, disclosing, reproducing, distributing, disseminating or otherwise > using this transmission. If you have received this message in error, please > promptly notify the sender by reply email and immediately delete this message > from your system. This message and any attachments may contain information > that is confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > ***********************************************************************************************************************
