alt1) if you can store a generation counter in the value of the "latest value" topic you could do as follows
topic latest_value key [id] topic full_history key[id, generation] on delete get the latest_value.generation_counter and issue deletes on full_history key[id, 0..generation_counter] alt2) if you cannot store a generation_counter in "latest_value" store a timestamp or uuid to make each key unique topic latest_value key [id] topic full_history key[id, timestamp/uuid] on delete of "id" scan full_history topic from beginning and issue deletes on full_history key[id, timestamp] you could optimized this by having another topic that contains a "to be purged ids" /svante 2018-03-21 11:16 GMT+01:00 Manikumar <manikumar.re...@gmail.com>: > Sorry, I was wrong. For history topic, we can use regular topic with > sufficient retention period. > maybe others can give more ideas. > > On Wed, Mar 21, 2018 at 3:34 PM, Kopacki, Tomasz (Nokia - PL/Wroclaw) < > tomasz.kopa...@nokia.com> wrote: > > > Do you mean I can use tombstone if my clean policy is 'delete' and it > > still work ? > > > > Sincerely, > > Tomasz Kopacki > > DevOps Engineer @ Nokia > > > > -----Original Message----- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: Wednesday, March 21, 2018 11:03 AM > > To: users@kafka.apache.org > > Subject: Re: log compaction v log rotation - best of the two worlds > > > > Not sure if understood requirement correctly. one option is to use two > > compacted topic topics. one is for current state of the resource and one > is > > for history. and use tombstones whenever you want to clear them. > > > > On Wed, Mar 21, 2018 at 2:53 PM, Kopacki, Tomasz (Nokia - PL/Wroclaw) < > > tomasz.kopa...@nokia.com> wrote: > > > > > Almost, > > > Consider this example: > > > > > > |R1|R2|R1|R3|R2|R4|R4|R1|R2| <- this is an example of a stream where > > > |R1|R2|R1|R3|R2|R4|R4|R1|R2| RX > > > represents updates of a particular resource. I need to keep the > > > history of changes forever for all the resources but only until > > > resource is alive. If resource expires/dies I'd like to remove it > > > completely. In this example consider that resource R2 dies but others > > > are still alive. In such case I'd like to able to transform this into: > > > |R1|R1|R3|R4|R4|R1| <- so, it's not compaction because I need the > > > |R1|R1|R3|R4|R4|R1| history > > > of changes but neither I can simply remove 'old' messages because I > > > need to do this based of the lifecycle of the resource not just their > > age. > > > > > > > > > > > > Sincerely, > > > Tomasz Kopacki > > > DevOps Engineer @ Nokia > > > > > > -----Original Message----- > > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > > Sent: Wednesday, March 21, 2018 10:17 AM > > > To: users@kafka.apache.org > > > Subject: Re: log compaction v log rotation - best of the two worlds > > > > > > We can enable both compaction and retention for a topic by setting > > > cleanup.policy="delete,compact" > > > http://kafka.apache.org/documentation/#topicconfigs > > > > > > Does this handle your requirement? > > > > > > On Wed, Mar 21, 2018 at 2:36 PM, Kopacki, Tomasz (Nokia - PL/Wroclaw) > > > < tomasz.kopa...@nokia.com> wrote: > > > > > > > Hi, > > > > I've been recently exploring log handling in kafka and I wonder > > > > if/how can I mixed log compaction with log rotation. > > > > A little background first: > > > > I have an application that uses kafka topics as a backend for event > > > > sourcing. Messages represents change of state of my 'resources'. > > > > Each resource has UID that is also used as a key for the messages. > > > > My resources have a lifecycle and when their life ends I don't need > > > > them anymore and there is no point in keeping their history. Having > > > > said that I thought that best choice for me will be log compaction > > > > with tombstone feature but I also would like to have a possibility > > > > to keep history of changes for the resources(only until they die). > > > > With those requirements I'd love to have a possibility to use > > > > tombstone feature for log rotation but I guess it ain't working like > > > that. > > > > > > > > Does anyone here had similar requirements and solve that somehow ? > > > > > > > > > > > > Sincerely, > > > > Tomasz Kopacki > > > > DevOps Engineer @ Nokia > > > > > > > > > > > > > >
