Hi Patrick, As far as I understand you can achieve that level of isolation by assigning the BI/Marketing consumers to distinct consumer *groups *for the topic/s, that you can for instance name accordingly. Within each consumer group you can then have a Marketing and a BI consumer that will be able to commit their offset in kafka independently as they move along the topic.
Luca On 31 Oct 2016 16:42, "Patrick Viet" <patrick.v...@gmail.com> wrote: > Hi, > > I've done some searching and can't find much on this topic. > Here is what I'm looking at: > > So a usual usage of kafka is to push messages in a topic, and then have > one/several consumers consume it (one per partition), in a "unique" way: a > message is consumed only once, and if it's consumed multiple times it's a > mistake, which we try to avoid. > > What if I'd like several consumers to get the same messages? > Here is a practical example: > > - Some app produces user action messages > - One entity (say, marketing) would like to consume them to do analysis > - Another entity (say, BI) would like to consume them too, to do some other > analysis > > So effectively both would get the full (duplicate) messages > > It seems that in the current high level consumer model, the messages only > go to one entity at a time, commits to zookeeper where it's at every X > messages read from partition Y, etc. My understanding is that if I want to > be able to read the messages in any other way, I'd have to make my own > consumer based on raw info from kafka and zookeeper. > > But maybe I'm wrong and it's totally possible in the current model to have > several consumers subscribe independently on a same topic, and consume then > with their own progression markers. If yes, a pointer to any documentation > / examples / discussions would be greatly appreciated > > -- > Best regards, > Patrick Viet >
