mjsax commented on code in PR #22808:
URL: https://github.com/apache/kafka/pull/22808#discussion_r3599460287
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/streams/StreamsGroup.java:
##########
@@ -1308,7 +1308,11 @@ public StreamsGroupDescribeResponseData.DescribedGroup
asDescribedGroup(
members.entrySet(committedOffset).forEach(
entry -> describedGroup.members().add(
entry.getValue().asStreamsGroupDescribeMember(
- targetAssignment.get(entry.getValue().memberId(),
committedOffset)
+ targetAssignment.get(entry.getValue().memberId(),
committedOffset),
+ // Task (end-)offsets are transient, unpersisted
telemetry, so unlike the rest of the member state
+ // they are read from the latest in-memory value rather
than at committedOffset. Members that have
+ // not reported any yield MemberTaskOffsets.EMPTY.
+ taskOffsets(entry.getValue().memberId())
Review Comment:
> No, even for a task that is in the target assignment, if the heartbeat
write that updated its offset later fails and rolls back, everything else
reverts to the committed snapshot, but the taskOffsets value stays at the
failed write's value until the next heartbeat overwrites it (it's a plain
in-memory map, not on the snapshot timeline).
Yes, I understand this part.
> Net effect: describe could momentarily report an offset that was never
actually committed.
When you say "offset" here, what offset are you referring too? I am not sure
if I understand what the issue is. Can you provide an example (mostly for my
own education)?
> don't get what you are suggesting here? you mean we filter on target
assignement? This is orthogonal to my concern. I do not recommend doing that.
targetAssignment is where the group is heading. These diverge exactly during
rebalances/warm-up. If you filter by target assignment, you'd hide offsets for
a task the member currently owns but is being moved off. So filtering trades
occasionally shows an offset for a not-yet/no-longer-assigned task for can omit
offsets for tasks actively being worked.
Ah. This make sense. We have the target assignment, not current assignment
that we report back. For this case, all bets are off anyway. As you said,
task-offset of for currently assigned tasks which may be different to target
tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
