[
https://issues.apache.org/jira/browse/KAFKA-20805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Bejeck updated KAFKA-20805:
--------------------------------
Affects Version/s: 4.3.1
4.3.0
> State directory cleaned prematurely under KIP-1035, forcing from-scratch
> restore
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-20805
> URL: https://issues.apache.org/jira/browse/KAFKA-20805
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.0, 4.3.1, 4.4.0
> Reporter: Bill Bejeck
> Assignee: Bill Bejeck
> Priority: Blocker
> Fix For: 4.4.0
>
>
> After a StreamThread dies and releases its task locks, the background
> state-directory cleaner can delete a still-valid task directory whose task
> committed only seconds earlier. The reassigned task then finds no local
> offsets, restores its changelog from the beginning, and — for a
> compacted/retention-deleted changelog — possibly the restore is overtaken by
> retention mid-flight, producing \{{OffsetOutOfRangeException}} →
> \{{TaskCorruptedException}} → rebalance.
> The cleaner decides obsolescence from the task directory's filesystem mtime:
> \{{now - taskDir.lastModified() > state.cleanup.delay.ms}}
> (StateDirectory.cleanRemovedTasks). This relied on the pre-KIP-1035 behavior
> where the per-commit \{{.checkpoint}} file (a direct child of the task dir)
> refreshed the dir mtime on every commit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
