[
https://issues.apache.org/jira/browse/KAFKA-10760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249409#comment-17249409
]
Jun Rao commented on KAFKA-10760:
---------------------------------
[~kbrajesh176]: For max.compaction.lag.ms, we simply set segment.ms to that
value to trigger a segment roll. Currently, time-based rolling is achieved by
comparing an incoming record's timestamp with the timestamp of the first
record. So, a segment will only be rolled if there is a new record coming in.
Not sure what's the best way to improve this since the straight-forward
approach of just comparing the record's timestamp and current time for log
rolling has the potential issue that sending records with old timestamp
triggers frequent log rolling.
> In compacted topic with max.compaction.lag.ms, the segments are not rolled
> until new messages arrive
> ----------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10760
> URL: https://issues.apache.org/jira/browse/KAFKA-10760
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Sarwar Bhuiyan
> Assignee: Brajesh Kumar
> Priority: Major
>
> Currently, if a compacted topic has min.cleanable.dirty.ratio set to
> something low and max.compaction.lag.ms set to a small time, according to KIP
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-354] the expectation
> is that the active segment will be rolled regardless or segment.ms or whether
> new data has come in to "advance" the time. However, in practice, the current
> implementation only rolls the segment when new data which means that there
> are situations where the topic is not fully compacted until new data arrives
> which may not be until a while later. The implementation can be improved by
> rolling the segment just purely based on the max.compaction.lag.ms setting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
