Thanks Guozhang. Is the below true?
1. There is no central controller for the consuemrs which assigns the partitions to the active consumers. 2. Hence, the consumers each fight for the allocation by registering the watch on active consumers (/ids path in zk). But how do the consumers finally arrive at concensus? Who ensures that the consumer who took ownership actually is consuming data? We are noticing scenarios where the rebalances are taking a lot of time and we have (backoff-time * rebalance-retries) > zk-session-timeout. Would it be better to have a controller pattern followed by the consumers? 1. Have a consumer controller which is leader elected 2. The consumers then register their active state in zk 3. The controller allocates the consumers the partitions depending on strategies - ex: ceil(no-of-partitions/active-consumers) 4. There is a background thread at the controller to monitor if the allocated consumer is actually consuming the messages out of the partition and if not, re-allocates them to others On Tue, Feb 2, 2016 at 10:46 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Vamsi, > > Consumer partition assignment is always within a single group. You can find > the java docs for the PartitionAssignor interface here: > > > https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/RangeAssignor.html > > StreamPartitionAssignor is one implementation of the interface which is > used exclusively in Kafka Streams. > > Guozhang > > On Tue, Feb 2, 2016 at 1:37 AM, Achanta Vamsi Subhash < > achanta.va...@flipkart.com> wrote: > > > Hi all, > > > > I am going through the partition assignment logic in the Kafka new > > consumer. Is there any documentation for the logic implemented? It would > be > > great if some one can write the algorithm for consumer partition > assignment > > (within a group and across groups). > > > > -- > > Regards > > Vamsi Subhash > > > > > > -- > -- Guozhang > -- Regards Vamsi Subhash
