NEERAJ VAIDYA created KAFKA-13383:
-------------------------------------
Summary: Consumer Group partition management across clusters
(Active-Active)
Key: KAFKA-13383
URL: https://issues.apache.org/jira/browse/KAFKA-13383
Project: Kafka
Issue Type: New Feature
Components: consumer, core, mirrormaker
Affects Versions: 3.0.0, 2.8.1
Reporter: NEERAJ VAIDYA
When using MM2 (Mirror Maker 2) in an Active-Active setup, we would like
consumers to consume from the same topic name in both datacentres, rather than
consume from 1) The topic without the site prefix 2) and the topic with the
site prefix.
MM2 replicates a topic to a remote site by prefixing the site name.
Confluent replicator prevents this topic name prefixing by using Provenance
headers in the message.
However, if we start 2 consumers (1 on each site) and both of them consume from
the topic with the same name, the partitions are not balanced across sites.
So, the consumer on both sites process all messages and hence duplication of
processing occurs. This might not be a big deal for consumers in some business
domains, but in the Telecom domain, where duplicates are not tolerated, it
becomes an issue.
It would be great if the partition allocation can be done across clusters.
So that when a consumer is started on 1 site, it always also takes into
consideration if there are any other consumers for the same topic on the other
site.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
