Matthias J. Sax created KAFKA-21576:
---------------------------------------
Summary: Streams client reports a restored but not yet running
active task without an end offset, so its task moves cold
Key: KAFKA-21576
URL: https://issues.apache.org/jira/browse/KAFKA-21576
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax
Fix For: 4.5.0, 4.4.1
Under the streams group protocol, a client reports the changelog offsets of the
active tasks it is restoring. The coordinator treats an active task with a
reported offset as restoring, and with an end offset, it can tell whether the
task is caught up.
An active task that has finished restoring but is not running yet is reported
with an offset but without an end offset:
* TaskManager.maybeUpdateTaskOffsetSumSnapshot() reports the offset of every
task that is not RUNNING.
* DefaultStateUpdater.updateTaskEndOffsetSumSnapshot() computes end offsets
only for the tasks the state updater is still restoring. A restored task
waiting in restoredActiveTasks is not among them.
The coordinator therefore sees the owner as restoring with an unknown lag. The
warm-up refiner then counts the owner's state as not usable, and moves a task
the target assignment gives to another member without warming it up first.
The window lasts as long as the group is not ready: StreamThread.runOnce()
returns before draining restored tasks while streamsGroupReady is false. A
member whose first tasks arrive together with ASSIGNMENT_DELAYED (for example,
because another member joined within group.streams.assignment.interval.ms)
stays not ready until the next assignment, and reports its restored active
tasks with offsets but without end offsets until then. If the coordinator
refines the assignment in that time, it moves the member's tasks without
warming them up.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)