Satarupa, it sounds like you are conflating some concepts here. Some clarifying points:
- Only one consumer in a consumer group receives any given record from a topic. So in your scenario of 1 million consumers, they could not be members of the same group. You'd need 1 million consumer "groups" to achieve this behavior. - You don't need 1 million partitions, unless you have a consumer group with 1 million consumers. You could do this with a single partition, since each consumer group is essentially a group of one. As hinted at above, one way to sorta achieve this is: have each consumer use a distinct consumer group, i.e. use a UUID as group id, s.t. each consumer is in a group of one. Then each consumer will receive every record in the topic. Kafka stores client state for each consumer -- an architecture which really isn't designed for millions of consumers. But it sounds like your clients are ephemeral, so perhaps they don't actually need to preserve state in Kafka. Maybe turn off auto-commit. Ryanne On Wed, Aug 29, 2018 at 12:47 AM Pal, Satarupa <satarupa_...@intuit.com> wrote: > Hi, > > I am from Intuit. We want to use Kafka as message bus where Single > Producer produces message and 1 Million Consumer listens it. > > > Requirement – > > > 1. Single producer and 1 Million Consumer and one particular Topic with > message. > 2. When Pushed Message thru producer, should be received by all > consumers > 3. Consumers can be added any time and may be removed any time. > > Query – > > > 1. Can I use a Single Consumer Group for the above requirement? > 2. Do I need to config 1 Million Partitions for all the Consumers > manually? Or Kafka will automatically do load balancing? > 3. Should Consumer need to subscribe every time, it listens? > 4. Or should consumer need to assign itself for the particular topic? > 5. Can all consumer listen to same host with post 9092 of Zoo Keeper? > > Need help to finalize my design. I just did a POC with One topic and One > consumer. > > Thank you, > Satarupa > >
