Hi Ryanne,
Thank you so much for detailed explanation.
Here is couple of more asks -
1) Here Consumers are not short lived but we want to listen to the message and
become idle. Is there a way to notify the Kafka server that message is reached
to Consumer? So that post which server does not preserve the state.
2) If we turn off Auto-commit , and client is offline, then next time, when
client is up, consumer will not get the message, right?
3) What about the performance here when there are 1 million Consumers are
listening to the same topic?
Is it a good design to suggest ? To give some background, we want to push
message to all the 1million consumers (each installed application will act as
consumer here) when there is a Hit fix/ Critical fix is released. So that all
our customers are notified to take the update.
Thank you,
Satarupa
On 8/29/18, 11:38 PM, "Ryanne Dolan" <ryannedo...@gmail.com> wrote:
Satarupa, it sounds like you are conflating some concepts here. Some
clarifying points:
- Only one consumer in a consumer group receives any given record from a
topic. So in your scenario of 1 million consumers, they could not be
members of the same group. You'd need 1 million consumer "groups" to
achieve this behavior.
- You don't need 1 million partitions, unless you have a consumer group
with 1 million consumers. You could do this with a single partition, since
each consumer group is essentially a group of one.
As hinted at above, one way to sorta achieve this is: have each consumer
use a distinct consumer group, i.e. use a UUID as group id, s.t. each
consumer is in a group of one. Then each consumer will receive every record
in the topic.
Kafka stores client state for each consumer -- an architecture which really
isn't designed for millions of consumers. But it sounds like your clients
are ephemeral, so perhaps they don't actually need to preserve state in
Kafka. Maybe turn off auto-commit.
Ryanne
On Wed, Aug 29, 2018 at 12:47 AM Pal, Satarupa <satarupa_...@intuit.com>
wrote:
> Hi,
>
> I am from Intuit. We want to use Kafka as message bus where Single
> Producer produces message and 1 Million Consumer listens it.
>
>
> Requirement –
>
>
> 1. Single producer and 1 Million Consumer and one particular Topic with
> message.
> 2. When Pushed Message thru producer, should be received by all
> consumers
> 3. Consumers can be added any time and may be removed any time.
>
> Query –
>
>
> 1. Can I use a Single Consumer Group for the above requirement?
> 2. Do I need to config 1 Million Partitions for all the Consumers
> manually? Or Kafka will automatically do load balancing?
> 3. Should Consumer need to subscribe every time, it listens?
> 4. Or should consumer need to assign itself for the particular topic?
> 5. Can all consumer listen to same host with post 9092 of Zoo Keeper?
>
> Need help to finalize my design. I just did a POC with One topic and One
> consumer.
>
> Thank you,
> Satarupa
>
>
