Thank you,
this is the documentation I've been looking for.
On 08/24/2018 07:46 PM, Eric Azama wrote:
The default partition assignment for the consumer should guarantee this as
long as the number of partitions in each topic is the same.
The term used for this in the Streams doc is co-partitioning. The link
below is referring specifically to the KStreams Join method, but the
concept is similar to what you're asking to do.
https://kafka.apache.org/20/documentation/streams/developer-guide/dsl-api.html#join-co-partitioning-requirements
RangeAssignor Javadoc:
https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/RangeAssignor.html
On Fri, Aug 24, 2018 at 1:55 AM Oliver Kindernay <kinder...@dispecer.sk>
wrote:
Hello,
I need to parallel-process messages with per key ordering guarantees so
I have kafka topic with keyed messages and multiple partitions. The need
has arised to process some messages with higher priority, so I thought
of adding a second topic where the messages with higher priority will be
sent.
The consumer maintains per key state in memory and also publishes
changes to compacted topic. Now, if I add the second topic for the
higher priority messages, is there some way to guarantee that when one
instance of consumer subscribes to both topics, it will get all messages
for some particular key, from both topics? I want to avoid the situation
when message for some particular key arrive at one consumer from the low
priority topic, and another consumer from the high priority topic.
Hope its clear, thank you
