@Kant I was going through the offset related configurations before setting
offsets.retention.minutes so came accross this configuration and thought to
ask whether this should also be tuned or not.


Regards,
Abhimanyu




On Mon, May 22, 2017 at 2:24 PM, kant kodali <[email protected]> wrote:

> @Abhimanyu Why do you think you need to set that? Did you try setting
> offsets.retention.minutes
> = 1440 * 30 and still seeing duplicates?
>
> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
> [email protected]> wrote:
>
> > Hi Girish ,
> >
> > Do I need to tune this configuration offsets.retention.check.interval.ms
> > also . Please let me know if I need to tune any other configuration.
> >
> >
> > Regards,
> > Abhimanyu
> >
> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <[email protected]>
> wrote:
> >
> > > Yup, exactly as Kant said.
> > > Also make sure that the retention of the offsets topic is an upper
> bound
> > > across all topics. So in this case, don't create any other topics in
> the
> > > future with retention of more than 30 days or otherwise they may have
> the
> > > same problem too.
> > >
> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <[email protected]
> >
> > > wrote:
> > >
> > >> Hi Kant,
> > >>
> > >> Thanks for the suggestion.
> > >>
> > >>
> > >> Regards,
> > >> Abhimanyu
> > >>
> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <[email protected]>
> > wrote:
> > >>
> > >>> @Abhimanyu You can try setting offset.retention = 30 (log.retention).
> > At
> > >>> most, you will have a storage overhead of 5 million msgs per day * 30
> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you
> > have
> > >>> a
> > >>> TB of hard disk)
> > >>>
> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <[email protected]>
> > wrote:
> > >>>
> > >>> > Looking at that ticket and reading the comments it looks like one
> of
> > >>> the
> > >>> > concern is as follows.
> > >>> >
> > >>> > "offsets.retention.minutes is designed to handle the case that a
> > >>> consumer
> > >>> > group goes away forever. In that case, we don't want to store the
> > >>> offsets
> > >>> > for that group forever."
> > >>> >
> > >>> > This can simply be addressed by setting offset.retention ==
> > >>> log.retention
> > >>> > by default right? In which case offset wont be stored forever even
> > when
> > >>> > consumer group goes away forever. When the consumer group goes away
> > >>> forever
> > >>> > the upper bound to clean up offsets would be equal to
> log.retention.
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <[email protected]>
> > >>> wrote:
> > >>> >
> > >>> >> What is your average message size and network speed?
> > >>> >>
> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> > >>> >> [email protected]> wrote:
> > >>> >>
> > >>> >>> Hi Girish,
> > >>> >>>
> > >>> >>> I did not set any value for offsets.retention.minutes so
> therefore
> > >>> what I
> > >>> >>> think is picking its default value i.e 1440 minutes so what do
> you
> > >>> think
> > >>> >>> what should I set if I am keeping my data for 30 days?
> > >>> >>>
> > >>> >>> Regards,
> > >>> >>> Abhimanyu
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> >
>

Reply via email to