[
https://issues.apache.org/jira/browse/KAFKA-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918985#comment-16918985
]
Richard Yu commented on KAFKA-8522:
-----------------------------------
I've some thoughts to get around this. (the best approach I can think of is
once a topic partition and its respective checkpoint file is created, we don't
remove the entry from the partition to checkpoint map). Once the
LogCleanerManager instance is destroyed (with the exception of when
{{partitions}} are having their checkpoint files being moved from one directory
to the other), then we remove all files.
> Tombstones can survive forever
> ------------------------------
>
> Key: KAFKA-8522
> URL: https://issues.apache.org/jira/browse/KAFKA-8522
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
> Reporter: Evelyn Bayes
> Priority: Minor
>
> This is a bit grey zone as to whether it's a "bug" but it is certainly
> unintended behaviour.
>
> Under specific conditions tombstones effectively survive forever:
> * Small amount of throughput;
> * min.cleanable.dirty.ratio near or at 0; and
> * Other parameters at default.
> What happens is all the data continuously gets cycled into the oldest
> segment. Old records get compacted away, but the new records continuously
> update the timestamp of the oldest segment reseting the countdown for
> deleting tombstones.
> So tombstones build up in the oldest segment forever.
>
> While you could "fix" this by reducing the segment size, this can be
> undesirable as a sudden change in throughput could cause a dangerous number
> of segments to be created.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
