Hello, I've been experimenting with Kafka and I've run into the following issue:
- Our log segment retention period is 2 hours - A broker in the ISR for partition A went - The broker stayed offline for several days - The broker was brought online - Log segments for partition A (at this point, are several days old) exist on the broker - The broker is no longer part of the ISR for partition A - The broker proceeds to delete log segments that it is part of the ISR of now, but doesn't seem to be deleting log segments from partition A - When we reassigned partition A to the broker, it deleted the old log segments. Here are the log segments: date;ls -al /data/disk6/kafka/loadtest_1-0 Thu Dec 14 18:19:59 UTC 2017 ... -rw-r--r-- 1 kafka kafka 222688 Dec 11 23:50 00000000017831621928.index -rw-r--r-- 1 kafka kafka 1073740824 Dec 11 21:52 00000000017831621928.log -rw-r--r-- 1 kafka kafka 314640 Dec 11 23:50 00000000017831621928.timeindex -rw-r--r-- 1 kafka kafka 203408 Dec 11 23:50 00000000017846956788.index -rw-r--r-- 1 kafka kafka 1073741323 Dec 11 21:58 00000000017846956788.log Also, the log messages from the broker indicate that it is "aware" of this partition: [2017-12-13 23:12:20,977] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (/data/disk6/kafka/loadtest_1-0/00000000018140082770.index) has non-zero size but the last offset is 18140082770 which is no larger than the base offset 18140082770.}. deleting /data/disk6/kafka/loadtest_1-0/00000000018140082770.timeindex, /data/disk6/kafka/loadtest_1-0/00000000018140082770.index, and /data/disk6/kafka/loadtest_1-0/00000000018140082770.txnindex and rebuilding index... (kafka.log.Log) [2017-12-13 23:12:20,977] INFO Loading producer state from snapshot file '/data/disk6/kafka/loadtest_1-0/00000000018140082770.snapshot' for partition loadtest_1-0 (kafka.log.ProducerStateManager) [2017-12-13 23:12:26,147] INFO Recovering unflushed segment 18140082770 in log loadtest_1-0. (kafka.log.Log) [2017-12-13 23:12:26,148] INFO Loading producer state from snapshot file '/data/disk6/kafka/loadtest_1-0/00000000018140082770.snapshot' for partition loadtest_1-0 (kafka.log.ProducerStateManager) [2017-12-13 23:12:27,121] INFO Loading producer state from offset 18146479928 for partition loadtest_1-0 with message format version 2 (kafka.log.Log) [2017-12-13 23:12:27,122] INFO Loading producer state from snapshot file '/data/disk6/kafka/loadtest_1-0/00000000018146479928.snapshot' for partition loadtest_1-0 (kafka.log.ProducerStateManager) [2017-12-13 23:12:27,122] INFO Completed load of log loadtest_1-0 with 21 log segments, log start offset 17831621928 and log end offset 18146479928 in 6298 ms (kafka.log.Log) Should Kafka brokers be cleaning up old log segments for a partition, even if the broker is not part of the ISR for that partition? Thanks in advance, Philip Su
