Thanks for the KIP. This has the potential to be a great improvement. A few initial questions/comments:
1. I think it's premature to talk about target versions for deprecation and removal of the existing group protocol. Unlike KRaft, this affects a core client protocol and hence deprecation/removal will be heavily dependent on how quickly applications migrate to the new protocol. 2. The KIP says we intend to release this in 4.x, but it wasn't made clear why. If we added that as a way to estimate when we'd deprecate and remove the group protocol, I also suggest removing this part. 3. We need to flesh out the details of the migration story. It sounds like we're saying we will support online migrations. Is that correct? We should explain this in detail. It could also be done as a separate KIP, if it's easier. 4. I am happy that we are pushing the pattern subscriptions to the server, but it seems like there could be some tricky compatibility issues. Will we have a mechanism for users to detect that they need to update their regex before switching to the new protocol? 5. Related to the last question, will the Java client allow the users to stick with the current regex engine for compatibility reasons? For example, it may be handy to keep using client based regex at first to keep migrations simple and then migrate to server based regexes as a second step. 6. When we say that the group coordinator will be responsible for storing the configurations and that the configurations will be deleted when the group is deleted. Will a transition to DEAD trigger deletion of configurations? 7. Will the choice to store the configs in the group coordinator make it harder to list all cluster configs and their values? 8. How would someone configure a group before starting the consumers? Have we considered allowing the explicit creation of groups? Alternatively, the configs could be decoupled from the group lifecycle. 9. Will the Consumer.subscribe method for the Java client still take a `java.util.regex.Pattern` of do we have to introduce an overload? 10. I agree with Justine that we should be clearer about the reason to switch to IBP/metadata.version from the feature flag. Maybe we mean that we can switch the default for the feature flag to true based on the metadata.version once we want to make it the default. 11. Some of the protocol APIs don't mention the required ACLs, it would be good to add that for consistency. 12. It is a bit odd that ConsumerGroupHeartbeat requires "Read Group" even though it seems to do more than reading. 13. How is topic recreation handled by the consumer with the new group protocol? It would be good to have a section on this. 14. The KIP mentions we will write the new coordinator in Java. Even though this is an implementation detail, do we plan to have a new gradle module for it? 15. Do we have a scalability goal when it comes to how many members the new group protocol can support? 16. Did we consider having SubscribedTopidIds instead of SubscribedTopicNames in ConsumerGroupHeartbeatRequest? Is the idea that since we have to resolve the regex on the server, we can do the same for the topic name? The difference is that sending the regex is more efficient whereas sending the topic names is less efficient. Furthermore, delete and recreation is easier to handle if we have topic ids. Thanks, Ismael On Wed, Jul 6, 2022 at 10:45 AM David Jacot <dja...@confluent.io.invalid> wrote: > Hi all, > > I would like to start a discussion thread on KIP-848: The Next > Generation of the Consumer Rebalance Protocol. With this KIP, we aim > to make the rebalance protocol (for consumers) more reliable, more > scalable, easier to implement for clients, and easier to debug for > operators. > > The KIP is here: https://cwiki.apache.org/confluence/x/HhD1D. > > Please take a look and let me know what you think. > > Best, > David > > PS: I will be away from July 18th to August 8th. That gives you a bit > of time to read and digest this long KIP. >
