Thanks Tom,
Thats a lot clearer now.
Michael
On Wed, May 4, 2016 at 3:27 PM, Tom Crayford <tcrayf...@heroku.com> wrote:
> Michael,
>
> Increasing the offset retention period means that old consumers will use up
> space in the __consumer_offsets topic and in the cache the brokers hold on
> that topic. The cache is in memory, so if you churn consumer groups a lot
> that could be problematic. The other issue is that booting new brokers
> requires replaying the whole log to load the cache, and if it's larger then
> that could be problematic.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Wednesday, 4 May 2016, Michael Freeman <mikfree...@gmail.com> wrote:
>
> > Hey Tom,
> > Are there any details on the negative side effects of
> > increasing the offset retention period? I'd like to increase it but want
> to
> > be aware of the risks.
> >
> > Thanks
> >
> > Michael
> >
> > > On 4 May 2016, at 05:06, Tom Crayford <tcrayf...@heroku.com
> > <javascript:;>> wrote:
> > >
> > > Jun,
> > >
> > > Yep, you got it. If there are no offset commits for 24h, then your
> offset
> > > will disappear and be cleaned up. I think this behaviour is at least a
> > > little suboptimal and surprising, but it does prevent a lot of issues
> > that
> > > arise when you rapidly add and remove consumer groups.
> > >
> > > I hope that helps!
> > >
> > > Tom Crayford,
> > > Heroku Kafka
> > >
> > >> On Tue, May 3, 2016 at 7:45 PM, Jun MA <mj.saber1...@gmail.com
> > <javascript:;>> wrote:
> > >>
> > >> I think I figured that out. Based on the explanation on
> > >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka <
> > >> http://www.slideshare.net/jjkoshy/offset-management-in-kafka>,
> > >> offsets.retention.minutes is used for clean up dead consumer group. It
> > >> means if one consumer group hasn’t commit any offset for
> > >> offsets.retention.minutes, kafka will clean up its offset, which make
> > sense
> > >> for my case.
> > >>
> > >> Thanks,
> > >> Jun
> > >>> On May 3, 2016, at 11:46 AM, Jun MA <mj.saber1...@gmail.com
> > <javascript:;>> wrote:
> > >>>
> > >>> Thanks for your reply. I checked the offset topic and the cleanup
> > policy
> > >> is actually compact.
> > >>>
> > >>> Topic:__consumer_offsets PartitionCount:50
> > >> ReplicationFactor:3
> > >>
> >
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
> > >>>
> > >>> And I’m using 0.9.0.1 so the default config for log.cleaner.enable is
> > >> true.
> > >>> In this case theoretically I should not lose any offset, right?
> > >>> I noticed that offsets.retention.minutes by default is 24hours, which
> > is
> > >> correlated to my offset becomes unknown, so I’m wondering if that is
> > >> because of this config? Can someone explain more about this
> > >> config(description: Log retention window for offsets topic) because I
> > don’t
> > >> understand what this does?
> > >>>
> > >>> Another related question is that how can I actually see the data of
> > >> offset topic? I think by reading the actual data might help.
> > >>>
> > >>> Thanks,
> > >>> Jun
> > >>>
> > >>>> On May 3, 2016, at 1:15 AM, Gerard Klijs <gerard.kl...@dizzit.com
> > <javascript:;>
> > >> <mailto:gerard.kl...@dizzit.com <javascript:;>>> wrote:
> > >>>>
> > >>>> Looks like it, you need to be sure the offset topic is using
> > compaction,
> > >>>> and the broker is set to enable compaction.
> > >>>>
> > >>>> On Tue, May 3, 2016 at 9:56 AM Jun MA <mj.saber1...@gmail.com
> > <javascript:;> <mailto:
> > >> mj.saber1...@gmail.com <javascript:;>>> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka
> cleans
> > >> up
> > >>>>> all old log segments(reach delete.retention time), I got unknown
> > >> offset.
> > >>>>>
> > >>>>> bin/kafka-consumer-groups.sh --bootstrap-server server:9092
> > >> --new-consumer
> > >>>>> --group testGroup --describe
> > >>>>> GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> > >>>>> testGroup, test, 0, unknown, 49, unknown, consumer-1_/10.32.241.2
> > >>>>> testGroup, test, 1, unknown, 61, unknown, consumer-1_/10.32.241.2
> > >>>>>
> > >>>>> In this situation, I cannot consume anything using new-consumer
> java
> > >>>>> driver if I disable auto-commit.
> > >>>>> I think this happens because new-consumer driver stores offset in
> > >> broker
> > >>>>> as a topic(not in zookeeper), and after reaching delete.retention
> > >> time, it
> > >>>>> got deleted and becomes unknown. And since I disabled auto-commit,
> it
> > >> can
> > >>>>> never know where it is, then it cannot consume anything.
> > >>>>>
> > >>>>> Is this what happened here? What should I do in this situation?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Jun
> > >>
> > >>
> >
>
