Yes! This was what I was looking for: " If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances. If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes. "
I can't believe I missed it the first time. Thanks again --Patrick On Mon, Oct 31, 2016 at 12:46 PM, Samuel Taylor <[email protected]> wrote: > > Hi Patrick, > > Can you use separate consumer groups to accomplish what you're looking for? > > Scroll to the "consumers" section on this document for a more detailed > description: > > https://kafka.apache.org/documentation.html#introduction > > On Mon, Oct 31, 2016 at 11:42 AM, Patrick Viet <[email protected]> > wrote: > > > Hi, > > > > I've done some searching and can't find much on this topic. > > Here is what I'm looking at: > > > > So a usual usage of kafka is to push messages in a topic, and then have > > one/several consumers consume it (one per partition), in a "unique" way: a > > message is consumed only once, and if it's consumed multiple times it's a > > mistake, which we try to avoid. > > > > What if I'd like several consumers to get the same messages? > > Here is a practical example: > > > > - Some app produces user action messages > > - One entity (say, marketing) would like to consume them to do analysis > > - Another entity (say, BI) would like to consume them too, to do some other > > analysis > > > > So effectively both would get the full (duplicate) messages > > > > It seems that in the current high level consumer model, the messages only > > go to one entity at a time, commits to zookeeper where it's at every X > > messages read from partition Y, etc. My understanding is that if I want to > > be able to read the messages in any other way, I'd have to make my own > > consumer based on raw info from kafka and zookeeper. > > > > But maybe I'm wrong and it's totally possible in the current model to have > > several consumers subscribe independently on a same topic, and consume then > > with their own progression markers. If yes, a pointer to any documentation > > / examples / discussions would be greatly appreciated > > > > -- > > Best regards, > > Patrick Viet > > > > > > -- > *Samuel Taylor* > Data Science > > *Square Root, Inc. <http://square-root.com/>* > <http://square-root.com/>Square-Root.com <http://square-root.com/>
