[
https://issues.apache.org/jira/browse/KAFKA-21111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-21111:
------------------------------------
Description:
We need to ensure that the refiner is only executed for new clients, RPC
version 1+, but not for old RPC version 0 clients (4.2/4.3 release).
Older clients don't report task-offsets, and thus we cannot use warmup tasks
because w/o the ability to observe their restore progress, we could never
promote them as active later.
Note one corner case: after a GC fail-over, a member might not have send a
heartbeat yet, and thus we don't know it's RPC version yet. Atm, the code
treats this incorrectly, and would optimistically assume that it's a v1 client
and that we can put warmup task on the client. This must be change to being
conservative and treating "unknown" as don't put warmup tasks on the client.
The high level idea is, to add new transient field to `StreamsGroup` to track
the RPC version. The tricky part is, that the refiner is computing an
intermediate assignment for the whole group, ie, if might be invoked on a v1
heartbeat, that there could still be v0 members in the group. So we could give
the individual member's RPC version into the refiner, and the refiner would
avoid to put a warmup on older client.
was:
We need to ensure that the refiner is only executed for new clients, RPC
version 1+, but not for old RPC version 0 clients (4.2/4.3 release).
Older clients don't report task-offsets, and thus we cannot use warmup tasks
because w/o the ability to observe their restore progress, we could never
promote them as active later.
Note one corner case: after a GC fail-over, a member might not have send a
heartbeat yet, and thus we don't know it's RPC version yet. Atm, the code
treats this incorrectly, and would optimistically assume that it's a v1 client
and that the refiner can run. This must be change to being conservative and
treating "unknown" as don't run the refiner.
The high level idea is, to add new transient field to `StreamsGroup` to track
the RPC version. The tricky part is, that the refiner is computing an
intermediate assignment for the whole group, ie, if might be invoked on a v1
heartbeat, that there could still be v0 members in the group. So we could give
the individual member's RPC version into the refiner, and the refiner would
avoid to put a warmup on older client. – A more conservative (maybe simpler?)
approach would be, to only invoke the refiner at all, if _all_ member in the
group are on v1+.
> Ensure refiner backward compatibility
> -------------------------------------
>
> Key: KAFKA-21111
> URL: https://issues.apache.org/jira/browse/KAFKA-21111
> Project: Kafka
> Issue Type: Sub-task
> Components: group-coordinator, streams
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Priority: Critical
>
> We need to ensure that the refiner is only executed for new clients, RPC
> version 1+, but not for old RPC version 0 clients (4.2/4.3 release).
> Older clients don't report task-offsets, and thus we cannot use warmup tasks
> because w/o the ability to observe their restore progress, we could never
> promote them as active later.
> Note one corner case: after a GC fail-over, a member might not have send a
> heartbeat yet, and thus we don't know it's RPC version yet. Atm, the code
> treats this incorrectly, and would optimistically assume that it's a v1
> client and that we can put warmup task on the client. This must be change to
> being conservative and treating "unknown" as don't put warmup tasks on the
> client.
> The high level idea is, to add new transient field to `StreamsGroup` to track
> the RPC version. The tricky part is, that the refiner is computing an
> intermediate assignment for the whole group, ie, if might be invoked on a v1
> heartbeat, that there could still be v0 members in the group. So we could
> give the individual member's RPC version into the refiner, and the refiner
> would avoid to put a warmup on older client.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
