Hi Luke, thanks for the hints. This helps a lot already.
We already use assign as we manage offsets on the consumer side. Currently we only have one partition and simply assign a stored offset on partition 0. For multiple partitions is it the correct behaviour to simply assign to partition number:offset or do I have to provide offsets for the other partitions too? I only want to listen to one partition. You mentioned custom producer partitioner. We currently use a random consumer group name for each consumer as we want each consumer to receive all messages of the environment. In this case do we still need a custom producer partitioner or is it enough to simply assign to the topic like described above? Christian Am Mi., 8. Dez. 2021 um 11:19 Uhr schrieb Luke Chen <show...@gmail.com>: > Hi Christian, > Answering your question below: > > > Let's assume we just have one topic with 10 partitions for simplicity. > We can now use the environment id as a key for the messages to make sure > the messages of each environment arrive in order while sharing the load on > the partitions. > > > Now we want each environment to only read the minimal number of messages > while consuming. Ideally we would like to to only consume its own messages. > Can we somehow filter to only > receive messages with a certain key? Can we maybe only listen to a certain > partition at least? > > > Unfortunately, Kafka doesn't have the feature to filter the messages on > broker before sending to consumer. > But for your 2nd question: > > Can we maybe only listen to a certain partition at least? > > Actually, yes. Kafka has a way to just fetch data from a certain partition > of a topic. You can use Consumer#assign API to achieve that. So, to do > that, I think you also need to have a custom producer partitioner for your > purpose. Let's say, in your example, you have 10 partitions, and 10 > environments. Your partitioner should send to the specific partition based > on the environment ID, ex: env ID 1 -> partition 1, env ID 2 -> partition > 2.... So, in your consumer, you can just assign to the partition containing > its environment ID. > > And for the idea of encrypting the messages to achieve isolation, it's > interesting! I've never thought about it! :) > > Hope it helps. > > Thank you. > Luke > > > On Wed, Dec 8, 2021 at 4:48 PM Christian Schneider < > ch...@die-schneider.net> > wrote: > > > We have a single tenant application that we deploy to a kubernetes > cluster > > in many instances. > > Every customer has several environments of the application. Each > > application lives in a separate namespace and should be isolated from > other > > applications. > > > > We plan to use kafka to communicate inside an environment (between the > > different pods). > > As setting up one kafka cluster per such environment is a lot of overhead > > and cost we would like to just use a single multi tenant kafka cluster. > > > > Let's assume we just have one topic with 10 partitions for simplicity. > > We can now use the environment id as a key for the messages to make sure > > the messages of each environment arrive in order while sharing the load > on > > the partitions. > > > > Now we want each environment to only read the minimal number of messages > > while consuming. Ideally we would like to to only consume its own > messages. > > Can we somehow filter to only > > receive messages with a certain key? Can we maybe only listen to a > certain > > partition at least? > > > > Additionally we ideally would like to have enforced isolation. So each > > environment can only see its own messages even if it might receive > messages > > of other environments from the same partition. > > I think in worst case we can make this happen by encrypting the messages > > but it would be great if we could filter on broker side. > > > > Christian > > > > -- > > -- > > Christian Schneider > > http://www.liquid-reality.de > > > > Computer Scientist > > http://www.adobe.com > > > -- -- Christian Schneider http://www.liquid-reality.de Computer Scientist http://www.adobe.com
