Hi, I've done some searching and can't find much on this topic. Here is what I'm looking at:
So a usual usage of kafka is to push messages in a topic, and then have one/several consumers consume it (one per partition), in a "unique" way: a message is consumed only once, and if it's consumed multiple times it's a mistake, which we try to avoid. What if I'd like several consumers to get the same messages? Here is a practical example: - Some app produces user action messages - One entity (say, marketing) would like to consume them to do analysis - Another entity (say, BI) would like to consume them too, to do some other analysis So effectively both would get the full (duplicate) messages It seems that in the current high level consumer model, the messages only go to one entity at a time, commits to zookeeper where it's at every X messages read from partition Y, etc. My understanding is that if I want to be able to read the messages in any other way, I'd have to make my own consumer based on raw info from kafka and zookeeper. But maybe I'm wrong and it's totally possible in the current model to have several consumers subscribe independently on a same topic, and consume then with their own progression markers. If yes, a pointer to any documentation / examples / discussions would be greatly appreciated -- Best regards, Patrick Viet
