M. Manna created KAFKA-7358: ------------------------------- Summary: Extended Partitioner to Support "Always Round Robin" Selection Key: KAFKA-7358 URL: https://issues.apache.org/jira/browse/KAFKA-7358 Project: Kafka Issue Type: Wish Components: clients Reporter: M. Manna Assignee: M. Manna
In my organisation, we have been using kafka as the basic publish-subscribe messaging system provider. Our goal is the send event-based messages reliably and securely, and perform data synchronisation based on the messages. For us, the message keys represent some metadata which we use to either ignore messages (if a loopback) or log some information. We have the following use case for messaging: 1) A Database transaction event takes place 2) The event is captured and messaged across 10 data centres all around the world. 3) A group of consumers (for each data centre with a unique consumer-group ID) are will process messages from their respective partitions. 1 consumer per partition. Under the circumstances, we only need a guarantee that same message won't be sent to multiple partitions. Using DefaultPartitioner, we can achieve this only with NULL keys. But since we need keys for metadata, we cannot maintain "Round-robin" selection of partitions because a key hash will determine which partition to choose. We need both non-null key with round-robin partition selection for KafkaProducer. We believe this solution is achievable and stable, because multiple consumer will not be consuming from the same partition as per Kafka's fundamental architecture. Hence, the ask for an extended partitioner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)