Hi,
Code snippet below. This creates two consumers with same group id
"consumer-group", they consume from "common-topic" which has 6 partitions. Each
group has 3 consumers.
However only one of the groups will ever run. Unless the producer is always
putting the messages into one partition, I would expect both consumer groups to
read from the topic.
Thanks,
Phill
Properties consumerProps = new Properties();
consumerProps.put("zookeeper.connect", zookeeperNode);
consumerProps.put("group.id", "consumer-group");
final ConsumerConnector consumerConnector =
Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps));
Map<String, Integer> topicCount = new HashMap<String, Integer>();
topicCount.put("common-topic", new Integer(3));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =
consumerConnector.createMessageStreams(topicCount);
// put each stream into its own thread
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
for (final KafkaStream stream : streams) {
executorService1.submit(new CcsKafkaStreamConsumer(stream,
("common-topic", messageWrapper));
}
Properties consumerProps = new Properties();
consumerProps.put("zookeeper.connect", zookeeperNode);
consumerProps.put("group.id", "consumer-group");
final ConsumerConnector consumerConnector =
Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps));
Map<String, Integer> topicCount = new HashMap<String, Integer>();
topicCount.put("common-topic", new Integer(3));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap =
consumerConnector.createMessageStreams(topicCount);
// put each stream into its own thread
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
for (final KafkaStream stream : streams) {
executorService2.submit(new CcsKafkaStreamConsumer(stream,
("common-topic", messageWrapper));
}
-----Original Message-----
From: Kevin Scaldeferri [mailto:[email protected]]
Sent: 09 March 2015 18:13
To: [email protected]
Subject: Re: Multiple consumer groups with same group id on a single topic
On Mon, Mar 9, 2015 at 10:38 AM, Phill Tomlinson <[email protected]>
wrote:
> Hi,
>
> I have two separate consumer groups on different JVM processes, but
> both have the same "group.id".
You've said this twice, and I think it's creating some confusion, because the
group.id is exactly what determines the members of a consumer group.
How many consumer threads are you providing total? Could you provide code
snippets to clarify how you're setting things up?
This email and any files transmitted with it are confidential, proprietary and
intended solely for the individual or entity to whom they are addressed. If you
have received this email in error please delete it immediately.
