Managed to figure this one out myself. This is due to the range partition assignment in 0.8.1.1 and the fact each of our topics has 8 partitions so only the first 8 consumers get assigned anything. Looks like 0.8.2.0 has a round robin assignment which is what we want.
On Fri, Apr 24, 2015 at 11:13 AM Bryan Baugher <bjb...@gmail.com> wrote: > Hi everyone, > > We are running Kafka 0.8.1.1 with Storm. We wrote our own spout which uses > the high level consumer API. Our setup is to create 4 spouts per worker. If > your not familiar with Storm its basically 4 kafka consumers per java > process. This particular consumer group is interested in 20 topics and ~150 > partitions. When we increased the parallelism to 6 workers and 24 consumers > we noticed certain consumers did not own any partitions in the group. These > consumers were on certain hosts. We see their ephemeral nodes in zookeeper > under their consumer group. We have also verified connectivity with kafka > from those nodes. I also found if I can get those workers/consumers to run > on the hosts with consumers that do own partitions they too will start > owning partitions. > > I'm also finding nothing in our logs in Kafka or on the consumer side > which indicates any kind of problem. > > Any suggestions on what to try? Is it possible to force a consumer > rebalance and who handles the partition assignment for a consumer group? > > Bryan >
