Luke Chen created KAFKA-16711:
---------------------------------
Summary: highestOffsetInRemoteStorage is not updated after logDir
altering within broker
Key: KAFKA-16711
URL: https://issues.apache.org/jira/browse/KAFKA-16711
Project: Kafka
Issue Type: Sub-task
Affects Versions: 3.7.0
Reporter: Luke Chen
Assignee: Luke Chen
We use topicIdPartition as the key for each RLM task. It will cause
highestOffsetInRemoteStorage in log not updated after logDir alter completion.
The reproducing flow is like this:
# tp-0 locating in dirA is the leader of the partition
# tp-0 is altering logDir to dirB
# tp-0 is copying segments to remote storage (note: the log in dirA)
# The logDir altering for tp-0 is completed
# remoteLogManager#onLeadershipChange is invoked, copiedOffsetOption is reset
to Optional.empty()
# The copying segments to remote storage in step 3 for tp-0 is completed,
updating copiedOffsetOption to new offset, as well as the
log#highestOffsetInRemoteStorage. (note: the log in dirA)
# In the next run of RLMTask, the log will be the one in target dir (dirB),
and the log#highestOffsetInRemoteStorage (dirB) will be the default value (-1),
which will block the log cleanup operation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
