I asked about this same issue in a previous thread. Thanks for reminding me, I've added this Jira: https://issues.apache.org/jira/browse/KAFKA-2172
I think this is a great new feature, but it's unfortunately the "all consumers must be the same" is just a bit too restrictive. Jason On Tue, May 5, 2015 at 5:20 PM, Bryan Baugher <bjb...@gmail.com> wrote: > Hi everyone, > > We recently switched to round robin partition assignment after we noticed > that range partition assignment (default) will only make use of the first X > consumers were X is the number of partitions for a topic our consumers are > interested in. We then noticed the caveat in round robin, > > "Round-robin assignment is permitted only if: (a) Every topic has the same > number of streams within a consumer instance (b) The set of subscribed > topics is identical for every consumer instance within the group." > > We tried this out and found if all consumers don't agree on topic > subscription they basically stop consuming until things get figured out. > This is bad for us since our consumers change their topic subscription > based on config they load from a REST service periodically. > > Is there something we can do on our side to avoid this? The best thing I > can think of is to try and use something like zookeeper to coordinate > changing the topic filters. > > Would it be possible to see the round robin assignment updated to not > require identical topic subscriptions? > > Bryan >
