@Abhimanyu Why do you think you need to set that? Did you try setting offsets.retention.minutes = 1440 * 30 and still seeing duplicates?
On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > Hi Girish , > > Do I need to tune this configuration offsets.retention.check.interval.ms > also . Please let me know if I need to tune any other configuration. > > > Regards, > Abhimanyu > > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com> wrote: > > > Yup, exactly as Kant said. > > Also make sure that the retention of the offsets topic is an upper bound > > across all topics. So in this case, don't create any other topics in the > > future with retention of more than 30 days or otherwise they may have the > > same problem too. > > > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <abhimanyunagr...@gmail.com> > > wrote: > > > >> Hi Kant, > >> > >> Thanks for the suggestion. > >> > >> > >> Regards, > >> Abhimanyu > >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com> > wrote: > >> > >>> @Abhimanyu You can try setting offset.retention = 30 (log.retention). > At > >>> most, you will have a storage overhead of 5 million msgs per day * 30 > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you > have > >>> a > >>> TB of hard disk) > >>> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> > wrote: > >>> > >>> > Looking at that ticket and reading the comments it looks like one of > >>> the > >>> > concern is as follows. > >>> > > >>> > "offsets.retention.minutes is designed to handle the case that a > >>> consumer > >>> > group goes away forever. In that case, we don't want to store the > >>> offsets > >>> > for that group forever." > >>> > > >>> > This can simply be addressed by setting offset.retention == > >>> log.retention > >>> > by default right? In which case offset wont be stored forever even > when > >>> > consumer group goes away forever. When the consumer group goes away > >>> forever > >>> > the upper bound to clean up offsets would be equal to log.retention. > >>> > > >>> > > >>> > > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth...@gmail.com> > >>> wrote: > >>> > > >>> >> What is your average message size and network speed? > >>> >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < > >>> >> abhimanyunagr...@gmail.com> wrote: > >>> >> > >>> >>> Hi Girish, > >>> >>> > >>> >>> I did not set any value for offsets.retention.minutes so therefore > >>> what I > >>> >>> think is picking its default value i.e 1440 minutes so what do you > >>> think > >>> >>> what should I set if I am keeping my data for 30 days? > >>> >>> > >>> >>> Regards, > >>> >>> Abhimanyu > >>> >>> > >>> >> > >>> >> > >>> > > >>> > >> > >> >
