Hi Sachin, I’ve found this useful: http://stackoverflow.com/questions/34281643/how-to-test-whether-log-compaction-is-working-or-not-in-kafka, <http://stackoverflow.com/questions/34281643/how-to-test-whether-log-compaction-is-working-or-not-in-kafka,as> as well as this http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ <http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/> (the log compaction part).
Thanks Eno > On Apr 1, 2017, at 6:17 AM, Sachin Mittal <sjmit...@gmail.com> wrote: > > Ok I have gone through the docs and looked the log.cleaner properties. > The use case we have is that say for a change log topic we have at a point > of time > (k, v1). > When for same key we now have > (k, v2) we really don't want (k, v1) to be retained and get cleaned up as > soon as possible. > > So I see this: > 1. log.cleaner.delete.retention.ms 86400000 > I suppose we need to reduce this value to much less say 600000 (10 minutes > or so from default 1 day) > > 2. log.cleaner.min.compaction.lag.ms > I think default value of 0 is OK given we want old record be clean as soon > as possible? > > 3. log.cleaner.min.cleanable.ratio > I have not understood this one. What do we mean by dirty log? > > What we have observed is that our log segment retention time for change log > topic is 30 minutes. > So after that time old log segment does get deleted, but what we see is all > the segment file sizes are 1GB. > Very rarely we see an old segment size less than 1 GB. > So does this mean that logs are not getting compacted or the messages are > marked for deletion but somehow log cleaner is not cleaning up those > messages. > > Please let me know is these are the properties we need to adjust for a more > efficient cleanup of old messages for same key. > > Thanks > Sachin > > > > > On Sat, Apr 1, 2017 at 4:54 AM, Matthias J. Sax <matth...@confluent.io> > wrote: > >> That's a topic config you need to set at the broker side: >> >> See config parameter `log.cleaner.*` in >> http://kafka.apache.org/documentation/#brokerconfigs >> >> >> -Matthias >> >> On 3/31/17 11:49 AM, Sachin Mittal wrote: >>> Hi, >>> I have noticed that many times change log topics don't get compacted. The >>> segment log file is always 1 GB. >>> >>> So I would like to know how and when compaction comes into play. >>> is there a way we can get the topic compacted say trigger compaction >> after >>> x seconds of a given message or a given segment file creation. >>> >>> Also what could be the reason for change log topic not getting compacted. >>> >>> Thanks >>> Sachin >>> >> >>
