David, If the consumers never commit() or subscribe(), I don't think the group.id is relevant.
Ryanne On Wed, Oct 17, 2018 at 10:04 AM David Baraff <[email protected]> wrote: > I’m using Kafka for what I think is a somewhat non-standard purpose. We > have multiple producers which send messages to a topic. We have say as > many as 500 or 1000 consumers which each want to read every message posted > to the topic, but *only* from the point in time that the consumer came > alive and started listening. (We’re using Kafka to replace a UDB based > message system.) > > Given the above, I set things up as: > > + topic has only 1 partition > + retention time is very short (10 seconds) > + every consumer uses the same group.id <http://group.id/> > + consumers DO NOT commit offsets ever > + new consumers set the offset to the “latest available” when they > start > + consumers don’t subscribe: they assign themselves to partition 0 > of the topic > > This works fine: each consumer appears to get a copy of all the (new) > messages that come in, after they start consuming. > > Question: Is using the same group.id <http://group.id/> for every > consumer likely to wreck anything? I could generate a random group.id < > http://group.id/> so that it appears that every consumer is distinct, but > given that each consumer does an assign to partition 0, and never commits > offsets, is what I’m doing sound? > > I’m slightly worried I’m misusing Kafka somehow: will this likely continue > working in the future? > As I said, I’m slightly worried about reusing the same group.id < > http://group.id/> each time. But it seems bad to generate like 1,000 > different consumer records on the broker if there is no need… > > Also, I want to make sure I don’t bog the brokers up with needless > logfiles for messages that nobody will see. (Once a message gets to be 10 > seconds old, nobody should see it, because if you weren’t listening when it > was produced, you don’t care about.) > > Thanks for any insights/opinions. > > David Baraff > [email protected] <mailto:[email protected]> > > >
