qiang Liu created KAFKA-12889:
---------------------------------
Summary: log clean group consider empty log segment to avoid empty
log left
Key: KAFKA-12889
URL: https://issues.apache.org/jira/browse/KAFKA-12889
Project: Kafka
Issue Type: Improvement
Components: log cleaner
Affects Versions: 0.10.1.1
Reporter: qiang Liu
to avoid log index 4 byte relative offset overflow, log cleaner group check log
segments offset to make sure group offset range not exceed Int.MaxValue.
this offset check currentlly not cosider next is next log segment is empty, so
there will left empty log files every about 2^31 messages.
the left empty logs will be reprocessed every clean cycle, which will rewrite
it with same empty content, witch cause little no need io.
for __consumer_offsets topic, normally we can set cleanup.policy to
compact,delete to get rid of this.
my cluster is 0.10.1.1, but after aylize trunk code, it should has same problem
too.
some of my left empty logs,(run ls -l)
-rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.index
-rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.log
-rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.timeindex
-rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.index
-rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.log
-rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.timeindex
-rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.index
-rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.log
-rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.timeindex
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
