Guozhang Wang created KAFKA-8512:
------------------------------------
Summary: Looking into the Future: Assignor Version
Key: KAFKA-8512
URL: https://issues.apache.org/jira/browse/KAFKA-8512
Project: Kafka
Issue Type: Sub-task
Reporter: Guozhang Wang
I'd propose to modify the JoinGroup protocol in this KIP as well to take the
read `protocol version` from the PartitionAssignor.
And then on the broker side, when choosing the leader it will pick the one with
the highest protocol version instead of picking it "first come first serve".
Although this change will not benefit the upgrade path at this time, in the
future if we need to upgrade the assignor again, as long as they would not
change the rebalance semantics (e.g. like we did in this KIP, from "eager" to
"cooperative") we can actually use one rolling bounce instead since as long as
there's one member on the newer version, that consumer will be picked.
For example, this can also help saving "version probing" cost on Streams as
well: suppose we augment the join group schema with `protocol version` in Kafka
version 2.3, and then with both brokers and clients being in version 2.3+, on
the first rolling bounce where subscription and assignment schema and / or user
metadata has changed, this protocol version will be bumped. On the broker side,
when receiving all member's join-group request, it will choose the one that has
the highest protocol version (also it assumes higher versioned protocol is
always backward compatible, i.e. the coordinator can recognize lower versioned
protocol as well) and select it as the leader. Then the leader can decide,
based on its received and deserialized subscription information, how to assign
partitions and how to encode the assignment accordingly so that everyone can
understand it. With this, in Streams for example, no version probing would be
needed since we are guaranteed the leader knows everyone's version -- again it
is assuming that higher versioned protocol is always backward compatible -- and
hence can successfully do the assignment at that round.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
