Hello all, @Tom - thank you for your answer :)
Here's the link to a gist with a minimal example: https://gist.github.com/anonymous/f78184eaeec3ee82b15182aec24a432a Here's the commands to create the compacted topic on a local kafka: kafka-topics.sh -zookeeper 127.0.0.1:2181 --create --topic compaction-test --partitions 1 --replication-factor 1 kafka-topics.sh -zookeeper 127.0.0.1:2181 --alter --topic compaction-test --config min.cleanable.dirty.ratio=0.01 --config cleanup.policy=compact --config segment.ms=100 --config delete.retention.ms=100 Here is the relevant part of the server.properties log.retention.check.interval.ms=100 log.cleaner.delete.retention.ms=100 log.cleaner.enable=true log.cleaner.min.cleanable.ratio=0.01 I am not sure what a tombstone would actually look like if consumed. I still see all the messages, just with null values. I believe the log.cleaner.delete.retention.ms should be responsible for how log a tombstone is kept - I set it to a very small time for both topic and server config, to be sure. My hope was that the message would be deleted during the test. Any ideas still appreciated :) Best, Christiane On 10 August 2016 at 12:05, Tom Crayford <tcrayf...@heroku.com> wrote: > I'd possibly check the log segments in question, by using the > DumpLogSegments tool. > > Note that Kafka keeps a deleted tombstone for 24h by default after a > deletion. Are you checking for the key and the value being present during > this testing? > > Sample code for the producer messages would be useful as well. > > Thanks > > Tom Crayford > Heroku Kafka > > On Wednesday, 10 August 2016, Christiane Lemke <christiane.le...@gmail.com > > > wrote: > > > Hi all, > > > > I am trying to set up a minimal example understanding log compaction > > behaviour using kafka-clients-0.10.0.0.jar. I got the compaction > behaviour > > working fine, however, when trying to delete a message explicitly by > > writing a null value, the message seems to not be deleted. > > > > These are my settings for my topic for compaction and deletion kicking in > > as soon as possible: Configs:min.cleanable.dirty.ratio=0.01, > > delete.retention.ms=100,retention.ms=100,segment.ms > > =100,cleanup.policy=compact > > > > The offending consumer record looks like this: > > [ConsumerRecord(topic = compation-test, partition = 0, offset = 33, > > CreateTime = 1470812816735, checksum = 3859648886, serialized key size = > > 16, serialized value size = -1, key = 9c9bde71-29ec-4687-ab24- > > 9459f5fc0d34, > > value = null)] > > > > I can see the cleaner threads running fine, producing output like this: > > [2016-08-10 08:24:51,601] INFO Cleaner 0: Cleaning segment 37 in log > > tns_ticket-0 (last modified Wed Aug 10 07:52:26 UTC 2016) into 0, > retaining > > deletes. (kafka.log.LogCleaner) > > (retaining deletes?) > > > > I am running out of ideas of settings to try - are there any ideas about > > what I might have missed or misunderstood? > > > > Any hint greatly appreciated :) > > > > Best, Christiane > > >
