Hi, " Unless the producer is always putting the messages into one partition, I would expect both consumer groups to read from the topic."
This was the issue. I thought the producer would round robin messages to different partitions but it must have just been writing to a single partition. I re-wrote the producer to use the new API and target set partitions and I can now see both consumer groups consuming from different partitions. Phill -----Original Message----- From: Phill Tomlinson [mailto:philltomlin...@fico.com] Sent: 10 March 2015 08:18 To: users@kafka.apache.org Subject: RE: Multiple consumer groups with same group id on a single topic Hi, Code snippet below. This creates two consumers with same group id "consumer-group", they consume from "common-topic" which has 6 partitions. Each group has 3 consumers. However only one of the groups will ever run. Unless the producer is always putting the messages into one partition, I would expect both consumer groups to read from the topic. Thanks, Phill Properties consumerProps = new Properties(); consumerProps.put("zookeeper.connect", zookeeperNode); consumerProps.put("group.id", "consumer-group"); final ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps)); Map<String, Integer> topicCount = new HashMap<String, Integer>(); topicCount.put("common-topic", new Integer(3)); Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector.createMessageStreams(topicCount); // put each stream into its own thread List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); for (final KafkaStream stream : streams) { executorService1.submit(new CcsKafkaStreamConsumer(stream, ("common-topic", messageWrapper)); } Properties consumerProps = new Properties(); consumerProps.put("zookeeper.connect", zookeeperNode); consumerProps.put("group.id", "consumer-group"); final ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps)); Map<String, Integer> topicCount = new HashMap<String, Integer>(); topicCount.put("common-topic", new Integer(3)); Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector.createMessageStreams(topicCount); // put each stream into its own thread List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); for (final KafkaStream stream : streams) { executorService2.submit(new CcsKafkaStreamConsumer(stream, ("common-topic", messageWrapper)); } -----Original Message----- From: Kevin Scaldeferri [mailto:ke...@scaldeferri.com] Sent: 09 March 2015 18:13 To: users@kafka.apache.org Subject: Re: Multiple consumer groups with same group id on a single topic On Mon, Mar 9, 2015 at 10:38 AM, Phill Tomlinson <philltomlin...@fico.com> wrote: > Hi, > > I have two separate consumer groups on different JVM processes, but > both have the same "group.id". You've said this twice, and I think it's creating some confusion, because the group.id is exactly what determines the members of a consumer group. How many consumer threads are you providing total? Could you provide code snippets to clarify how you're setting things up? This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately. This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.
