[
https://issues.apache.org/jira/browse/KAFKA-8362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135400#comment-17135400
]
Ming Liu commented on KAFKA-8362:
---------------------------------
In 2.5 Kafka, this still easily repro for me.
With compacted topic, let's say topic Test1 partition 100 is disk 1, in
"cleaner-offset-checkpoint", you can see
Test1 100 2344432
Then if you move that partition to disk 2, you will see the same thing in
"cleaner-offset-checkpoint", but the disk 1 "cleaner-offset-checkpoint" still
contains that partition.
Test1 100 2344543
This actually caused the problem during logclean due to how
allCleanerCheckpoints is implemented.
> LogCleaner gets stuck after partition move between log directories
> ------------------------------------------------------------------
>
> Key: KAFKA-8362
> URL: https://issues.apache.org/jira/browse/KAFKA-8362
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Reporter: Julio Ng
> Priority: Major
>
> When a partition is moved from one directory to another, their checkpoint
> entry in cleaner-offset-checkpoint file is not removed from the source
> directory.
> As a consequence when we read the last firstDirtyOffset, we might get a stale
> value from the old checkpoint file.
> Basically, we need clean up the entry from the check point file in the source
> directory when the move is completed
> The current issue is that the code in LogCleanerManager:
> {noformat}
> /**
> * @return the position processed for all logs.
> */
> def allCleanerCheckpoints: Map[TopicPartition, Long] = {
> inLock(lock) {
> checkpoints.values.flatMap(checkpoint => {
> try {
> checkpoint.read()
> } catch {
> case e: KafkaStorageException =>
> error(s"Failed to access checkpoint file ${checkpoint.file.getName}
> in dir ${checkpoint.file.getParentFile.getAbsolutePath}", e)
> Map.empty[TopicPartition, Long]
> }
> }).toMap
> }
> }{noformat}
> collapses the offsets when multiple entries exist for the topicPartition
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
