Satarupa, Glad I could help, and thanks for the additional context. It doesn't sound like this use-case requires real-time notification. Why not just poll a web service periodically, say every 5 minutes?
If you need something more real-time, I'd suggest using a more traditional publish-subscribe model, not Kafka. Start with a web service that just returns the most recent message(s). Then have clients connect to a central "hub" which broadcasts notifications to all connected clients. That way, clients can poll the web server periodically and/or connect to the hub to be notified when the message changes. Ryanne On Wed, Aug 29, 2018 at 9:31 PM Pal, Satarupa <satarupa_...@intuit.com> wrote: > Hi Ryanne, > > Thank you so much for detailed explanation. > > Here is couple of more asks - > > 1) Here Consumers are not short lived but we want to listen to the message > and become idle. Is there a way to notify the Kafka server that message is > reached to Consumer? So that post which server does not preserve the state. > 2) If we turn off Auto-commit , and client is offline, then next time, > when client is up, consumer will not get the message, right? > 3) What about the performance here when there are 1 million Consumers are > listening to the same topic? > > Is it a good design to suggest ? To give some background, we want to push > message to all the 1million consumers (each installed application will act > as consumer here) when there is a Hit fix/ Critical fix is released. So > that all our customers are notified to take the update. > > Thank you, > Satarupa > > On 8/29/18, 11:38 PM, "Ryanne Dolan" <ryannedo...@gmail.com> wrote: > > Satarupa, it sounds like you are conflating some concepts here. Some > clarifying points: > > - Only one consumer in a consumer group receives any given record from > a > topic. So in your scenario of 1 million consumers, they could not be > members of the same group. You'd need 1 million consumer "groups" to > achieve this behavior. > > - You don't need 1 million partitions, unless you have a consumer group > with 1 million consumers. You could do this with a single partition, > since > each consumer group is essentially a group of one. > > As hinted at above, one way to sorta achieve this is: have each > consumer > use a distinct consumer group, i.e. use a UUID as group id, s.t. each > consumer is in a group of one. Then each consumer will receive every > record > in the topic. > > Kafka stores client state for each consumer -- an architecture which > really > isn't designed for millions of consumers. But it sounds like your > clients > are ephemeral, so perhaps they don't actually need to preserve state in > Kafka. Maybe turn off auto-commit. > > Ryanne > > On Wed, Aug 29, 2018 at 12:47 AM Pal, Satarupa < > satarupa_...@intuit.com> > wrote: > > > Hi, > > > > I am from Intuit. We want to use Kafka as message bus where Single > > Producer produces message and 1 Million Consumer listens it. > > > > > > Requirement – > > > > > > 1. Single producer and 1 Million Consumer and one particular > Topic with > > message. > > 2. When Pushed Message thru producer, should be received by all > > consumers > > 3. Consumers can be added any time and may be removed any time. > > > > Query – > > > > > > 1. Can I use a Single Consumer Group for the above requirement? > > 2. Do I need to config 1 Million Partitions for all the Consumers > > manually? Or Kafka will automatically do load balancing? > > 3. Should Consumer need to subscribe every time, it listens? > > 4. Or should consumer need to assign itself for the particular > topic? > > 5. Can all consumer listen to same host with post 9092 of Zoo > Keeper? > > > > Need help to finalize my design. I just did a POC with One topic and > One > > consumer. > > > > Thank you, > > Satarupa > > > > > > >
