aliehsaeedii commented on code in PR #22808:
URL: https://github.com/apache/kafka/pull/22808#discussion_r3599118339
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/StreamsGroup.java:
##########
@@ -1308,7 +1308,11 @@ public StreamsGroupDescribeResponseData.DescribedGroup
asDescribedGroup(
members.entrySet(committedOffset).forEach(
entry -> describedGroup.members().add(
entry.getValue().asStreamsGroupDescribeMember(
- targetAssignment.get(entry.getValue().memberId(),
committedOffset)
+ targetAssignment.get(entry.getValue().memberId(),
committedOffset),
+ // Task (end-)offsets are transient, unpersisted
telemetry, so unlike the rest of the member state
+ // they are read from the latest in-memory value rather
than at committedOffset. Members that have
+ // not reported any yield MemberTaskOffsets.EMPTY.
+ taskOffsets(entry.getValue().memberId())
Review Comment:
>So you are saying the returned targetAssignment and the task-offsets might
not match up, ie, we might return task-offsets for tasks that are not in
targetAssignment?
No, even for a task that is in the target assignment, if the heartbeat write
that updated its offset later fails and rolls back, everything else reverts to
the committed snapshot, but the taskOffsets value stays at the failed write's
value until the next heartbeat overwrites it (it's a plain in-memory map, not
on the snapshot timeline). Net effect: describe could momentarily report an
offset that was never actually committed. I think it's tiny and will be fixed
by itself in the next HB.
I don't get what you are suggesting here? you mean we filter on target
assignement? This is orthogonal to my concern. I do not recommend doing that.
targetAssignment is where the group is heading. These diverge exactly during
rebalances/warm-up. If you filter by target assignment, you'd hide offsets for
a task the member currently owns but is being moved off. So filtering trades
occasionally shows an offset for a
not-yet/no-longer-assigned task for can omit offsets for tasks actively
being worked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
