We can enable both compaction and retention for a topic by setting cleanup.policy="delete,compact" http://kafka.apache.org/documentation/#topicconfigs
Does this handle your requirement? On Wed, Mar 21, 2018 at 2:36 PM, Kopacki, Tomasz (Nokia - PL/Wroclaw) < tomasz.kopa...@nokia.com> wrote: > Hi, > I've been recently exploring log handling in kafka and I wonder if/how can > I mixed log compaction with log rotation. > A little background first: > I have an application that uses kafka topics as a backend for event > sourcing. Messages represents change of state of my 'resources'. Each > resource has UID that is also used as a key for the messages. > My resources have a lifecycle and when their life ends I don't need them > anymore and there is no point in keeping their history. Having said that I > thought that best choice for me will be log compaction with tombstone > feature but I also would like to have a possibility to keep history of > changes for the resources(only until they die). > With those requirements I'd love to have a possibility to use tombstone > feature for log rotation but I guess it ain't working like that. > > Does anyone here had similar requirements and solve that somehow ? > > > Sincerely, > Tomasz Kopacki > DevOps Engineer @ Nokia > >
