Jun Rao created KAFKA-17076:
-------------------------------
Summary: logEndOffset could be lost due to log cleaning
Key: KAFKA-17076
URL: https://issues.apache.org/jira/browse/KAFKA-17076
Project: Kafka
Issue Type: Bug
Components: core
Reporter: Jun Rao
It's possible for the log cleaner to remove all records in the suffix of the
log. If the partition is then reassigned, the new replica won't be able to see
the true logEndOffset since there is no record batch associated with it. If
this replica becomes the leader, it will assign an already used offset to a
newly produced record, which is incorrect.
It's relatively rare to trigger this issue since the active segment is never
cleaned and typically is not empty. However, the following is one possibility.
# records with offset 100-110 are produced and fully replicated to all ISR.
All those records are delete records for certain keys.
# record with offset 111 is produced. It forces the roll of a new segment in
broker b1 and is added to the log. The record is not committed and is later
truncated from the log, leaving an empty active segment in this log. b1 at some
point becomes the leader.
# log cleaner kicks in and removes records 100-110.
# The partition is reassigned to another broker b2. b2 replicates all records
from b1 up to offset 100 and marks its logEndOffset at 100. Since there is no
record to replicate after offset 100 in b1, b2's logEndOffset stays at 100 and
b2 can join the ISR.
# b2 becomes the leader and assign offset 100 to a new record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
