The default partition assignment for the consumer should guarantee this as long as the number of partitions in each topic is the same.
The term used for this in the Streams doc is co-partitioning. The link below is referring specifically to the KStreams Join method, but the concept is similar to what you're asking to do. https://kafka.apache.org/20/documentation/streams/developer-guide/dsl-api.html#join-co-partitioning-requirements RangeAssignor Javadoc: https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/RangeAssignor.html On Fri, Aug 24, 2018 at 1:55 AM Oliver Kindernay <kinder...@dispecer.sk> wrote: > Hello, > > I need to parallel-process messages with per key ordering guarantees so > I have kafka topic with keyed messages and multiple partitions. The need > has arised to process some messages with higher priority, so I thought > of adding a second topic where the messages with higher priority will be > sent. > > The consumer maintains per key state in memory and also publishes > changes to compacted topic. Now, if I add the second topic for the > higher priority messages, is there some way to guarantee that when one > instance of consumer subscribes to both topics, it will get all messages > for some particular key, from both topics? I want to avoid the situation > when message for some particular key arrive at one consumer from the low > priority topic, and another consumer from the high priority topic. > > Hope its clear, thank you > > > >
