[
https://issues.apache.org/jira/browse/KAFKA-12889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366024#comment-17366024
]
Guozhang Wang commented on KAFKA-12889:
---------------------------------------
Thanks [~iamgd67] for reporting the issue and for the fix too!
> log clean group consider empty log segment to avoid empty log left
> ------------------------------------------------------------------
>
> Key: KAFKA-12889
> URL: https://issues.apache.org/jira/browse/KAFKA-12889
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 0.10.1.1, 2.8.0, 3.1.0
> Reporter: qiang Liu
> Priority: Trivial
> Fix For: 3.0.0
>
>
> to avoid log index 4 byte relative offset overflow, log cleaner group check
> log segments offset to make sure group offset range not exceed Int.MaxValue.
> this offset check currentlly not cosider next is next log segment is empty,
> so there will left empty log files every about 2^31 messages.
> the left empty logs will be reprocessed every clean cycle, which will rewrite
> it with same empty content, witch cause little no need io.
> for __consumer_offsets topic, normally we can set cleanup.policy to
> compact,delete to get rid of this.
> my cluster is 0.10.1.1, but after aylize trunk code, it should has same
> problem too.
>
> some of my left empty logs,(run ls -l)
> -rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.index
> -rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.log
> -rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.timeindex
> -rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.index
> -rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.log
> -rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.timeindex
> -rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.index
> -rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.log
> -rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.timeindex
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
