This is actually a really good question. If you change the retention policy of the offsets topic, then in the worst case, consumer groups could lose their last committed positions and fall back to the auto reset behavior. However, if your consumers are not down for a long time and you set the retention to a reasonably long value, maybe you can get away with it? One downside is that broker reads the entire offset log into an in-memory cache when it takes over leadership of one of the __consumer_offsets partitions. Hence the longer your retention time, the longer it will take for the new leader to read to the end of the log. There may be other consequences as well that I haven't thought of...
Can you describe in a little more detail the problem that you found enabling the cleaner? -Jason On Sun, Mar 6, 2016 at 3:09 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > We tested this on our stage environment and works fine if we change the > policy to delete from compact. Will there be any side effects if we change > it to delete for the __consumer_offsets topic? > > On Wed, Mar 2, 2016 at 4:43 PM, Achanta Vamsi Subhash < > achanta.va...@flipkart.com> wrote: > > > Hi all, > > > > We have a __consumer_offsets topic has cleanup.policy=compact and > > log.cleaner.enable=false. What would happen if we change the > cleanup.policy > > to delete? Will that treat the offsets topic as same as any other topic? > > > > We currently have a setup without log.cleaner.enable=false and we have > > offset topics hosting brokers using a lot of disk as they are never > > cleaned/compacted. We tried enabling log.cleaner.enable=true for the > > brokers with offsets topic and that is leading to lot of replicated data > > and is taking hours to finish. > > > > What is a better way to clean up the old segments of __consumer_offsets > > topic? > > > > -- > > Regards > > Vamsi Subhash > > > > > > -- > Regards > Vamsi Subhash >
