Re: multiple consumers for a same topic in multiple copies

2016-10-31 Thread Luca Pertile
Hi Patrick, As far as I understand you can achieve that level of isolation by assigning the BI/Marketing consumers to distinct consumer *groups *for the topic/s, that you can for instance name accordingly. Within each consumer group you can then have a Marketing and a BI consumer that will be abl

Re: multiple consumers for a same topic in multiple copies

2016-10-31 Thread Hans Jespersen
Just make sure they are not in the same consumer group by creating a unique value for group.id for each independent consumer. /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 31, 2016 at 9:42 AM, Patrick Viet wrote: > Hi, > > I

Re: multiple consumers for a same topic in multiple copies

2016-10-31 Thread Patrick Viet
Yes! This was what I was looking for: " If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances. If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer p

Re: multiple consumers for a same topic in multiple copies

2016-10-31 Thread Samuel Taylor
Hi Patrick, Can you use separate consumer groups to accomplish what you're looking for? Scroll to the "consumers" section on this document for a more detailed description: https://kafka.apache.org/documentation.html#introduction On Mon, Oct 31, 2016 at 11:42 AM, Patrick Viet wrote: > Hi, > >

multiple consumers for a same topic in multiple copies

2016-10-31 Thread Patrick Viet
Hi, I've done some searching and can't find much on this topic. Here is what I'm looking at: So a usual usage of kafka is to push messages in a topic, and then have one/several consumers consume it (one per partition), in a "unique" way: a message is consumed only once, and if it's consumed multi