[
https://issues.apache.org/jira/browse/KAFKA-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Cheng closed KAFKA-3137.
------------------------------
> Delete tombstones in log compacted topics may never get removed.
> ----------------------------------------------------------------
>
> Key: KAFKA-3137
> URL: https://issues.apache.org/jira/browse/KAFKA-3137
> Project: Kafka
> Issue Type: Bug
> Reporter: James Cheng
>
> I spoke about this with [~junrao]. I haven't tried to reproduce this, but Jun
> said that it looks like this is possible, so I'm filing it.
> Delete tombstones in log compacted topics are deleted after
> delete.retention.ms (at the topic level) or log.cleaner.delete.retention.ms
> (at the broker level).
> However, we don't have per-message timestamps (at least until KIP-32 is
> implemented). So the timestamp of the log segment file is used as a proxy.
> However, the modification time of the log segment changes whenever a
> compaction run happens.
> It's possible then that if log compaction happens very frequently that
> delete.retention.ms will never be reached. In that case, the delete
> tombstones would stay around longer than the user expected.
> I believe that means that log compaction would have to happen more frequently
> than delete.retention.ms. The frequency of log compaction is some calculation
> based on segment size, the criteria for segment roll (time or bytes), the
> min.cleanable.dirty.ratio, as well as the amount of traffic coming into the
> log compacted topic. So it's possible, but I'm not sure how likely.
> And I would imagine that this can't be fixed until KIP-32 is available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
