Do each of your topics have only 1 partition? The default partition assignor is org.apache.kafka.clients.consumer.RangeAssignor which will give the behavior you describe if each topic only has a single partition.
You probably want to switch it to RoundRobinAssignor which guarantees a more even distribution of topic partitions to consumers. -Ewen On Wed, Dec 21, 2016 at 10:15 AM, Robert Quinlivan <[email protected]> wrote: > Hello, > > We are using a multi-topic approach for partitioning data. Each topic will > follow a pattern naming convention. We want to design our consumer > configuration so that a slow topic will not block a fast topic, as each > topic will consume at a different rate. > > However I am seeing an issue where all topics matching the pattern are > assigned to a single consumer. When there are several consumers in the > group with a single topic with many partitions, the partitions are > automatically distributed among the available consumers in the group. I > expected that calling KafkaConsumer#subscribe(Pattern, > ConsumerRebalanceListener) would follow a similar behavior by distributing > the assigned topics among all consumers in the group. > > Is this not the case? What is the expected behavior and how would you > recommend implementing this design? > > Thank you > > -- > Robert Quinlivan > Software Engineer, Signal >
