Konstantine, this is exciting work! Couple questions: I understand that, overall, rebalances would require less work and less time acquiring and releasing resources. But OTOH I believe individual consumers might see successive revokes/assigns while a rebalance settles. Is that right? And if so, would the rebalance actually take longer in terms of wall time vs stop-the-world?
Ryanne On Tue, Oct 2, 2018 at 6:22 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > Hey everyone, > > I'd like to bring to your attention a general design document that was just > published in Apache Kafka's wiki space: > > > https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies > > It deals with the subject of Rebalancing of groups in Kafka and proposes > basic infrastructure to support improvements on the current rebalancing > protocol as well as a set of policies that can be implemented to optimize > rebalancing under a number of real-world scenarios. > > Currently, this wiki page is meant to serve as a reference to the > proposition of Incremental Cooperative Rebalancing overall. Specific KIPs > will follow in order to describe in more detail - using the standard KIP > format - the basic infrastructure and the first policies that will be > proposed for implementation in components such as Connect, the Kafka > Consumer and Streams. > > Stay tuned! > Konstantine >
