[
https://issues.apache.org/jira/browse/KAFKA-21111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116422#comment-18116422
]
Matthias J. Sax commented on KAFKA-21111:
-----------------------------------------
Yes, this would be the fall out for the simple solution of not running the
refiner at all. Not sure how often we expect a GC failover, plus having a KS
group in the middle of a rebalance at the same time?
For the more complex approach, when we pass in the RPC version for every member
into the refiner, the refiner could keep the status quo (ie let the
intermediate assignment match the current assignment, including already
assigned warm-up tasks) for all members as long as we don't know the version of
all members (which is only a small delay of max 2 x heartbeat-interval), and we
only start to assign new warm-ups after every member did heartbeat at least
once.
> Ensure refiner backward compatibility
> -------------------------------------
>
> Key: KAFKA-21111
> URL: https://issues.apache.org/jira/browse/KAFKA-21111
> Project: Kafka
> Issue Type: Sub-task
> Components: group-coordinator, streams
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Priority: Critical
>
> We need to ensure that the refiner assigns warm-up task only to new clients,
> RPC version 1+, but not to old RPC version 0 clients (4.2/4.3 release).
> Older clients don't report task-offsets, and thus we cannot use warmup tasks
> because w/o the ability to observe their restore progress, we could never
> promote them as active later.
> Note one corner case: after a GC fail-over, a member might not have send a
> heartbeat yet, and thus we don't know it's RPC version yet. Atm, the code
> treats this incorrectly, and would optimistically assume that it's a v1
> client and that we can put warmup task on the client. This must be change to
> being conservative and treating "unknown" as don't put warmup tasks on the
> client.
> The high level idea is, to add new transient field to `StreamsGroup` to track
> the RPC version. The tricky part is, that the refiner is computing an
> intermediate assignment for the whole group, ie, if might be invoked on a v1
> heartbeat, that there could still be v0 members in the group. So we could
> give the individual member's RPC version into the refiner, and the refiner
> would avoid to put a warmup on older client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
