Multiple topics is the model I would recommend for what you have described.
LinkedIn has an environment where we have a wide mix, in a lot of different
clusters. We have some topics that have one producer and one consumer
(queuing). We have some topics that are multi-producer (tracking and
Gwen,
Thanks for your reply. I understand all of the points you've made. I
think the challenge for us is that we have some consumers that are
interested in messages of one type, but we also have a rules engine that is
checking for events of many types and acting on them.
If we put discrete
Hello,
At my organization we are already using kafka in a few areas, but we're
looking to expand our use and we're struggling with how best to distribute
our events on to topics.
We have on the order of 30 different kinds of events that we'd like to
distribute via kafka. We have one or two
I usually approach this questions by looking at possible consumers.
You usually want each consumer to read from relatively few topics, use most
of the messages it receives and have fairly cohesive logic for using these
messages.
Signs that things went wrong with too few topics:
* Consumers that
