The default partitioner of old producer API is a sticky partitioner that keeps sending messages to the same partition for 10 secs (I don't remember the exact length duration) before switch to another partition if no key is specified in message. You can easily override this by setting partitioner.class
On Tue, Mar 10, 2015 at 4:32 PM, Phill Tomlinson <philltomlin...@fico.com> wrote: > Hi, > > " Unless the producer is always putting the messages into one partition, I > would expect both consumer groups to read from the topic." > > This was the issue. I thought the producer would round robin messages to > different partitions but it must have just been writing to a single > partition. I re-wrote the producer to use the new API and target set > partitions and I can now see both consumer groups consuming from different > partitions. > > Phill > > -----Original Message----- > From: Phill Tomlinson [mailto:philltomlin...@fico.com] > Sent: 10 March 2015 08:18 > To: users@kafka.apache.org > Subject: RE: Multiple consumer groups with same group id on a single topic > > Hi, > > Code snippet below. This creates two consumers with same group id > "consumer-group", they consume from "common-topic" which has 6 partitions. > Each group has 3 consumers. > > However only one of the groups will ever run. Unless the producer is > always putting the messages into one partition, I would expect both > consumer groups to read from the topic. > > Thanks, > Phill > > Properties consumerProps = new Properties(); > consumerProps.put("zookeeper.connect", zookeeperNode); consumerProps.put(" > group.id", "consumer-group"); final ConsumerConnector consumerConnector = > Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps)); > Map<String, Integer> topicCount = new HashMap<String, Integer>(); > topicCount.put("common-topic", new Integer(3)); Map<String, > List<KafkaStream<byte[], byte[]>>> consumerMap = > consumerConnector.createMessageStreams(topicCount); > > // put each stream into its own thread > List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); for > (final KafkaStream stream : streams) { > executorService1.submit(new CcsKafkaStreamConsumer(stream, > ("common-topic", messageWrapper)); } > > > > Properties consumerProps = new Properties(); > consumerProps.put("zookeeper.connect", zookeeperNode); consumerProps.put(" > group.id", "consumer-group"); final ConsumerConnector consumerConnector = > Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps)); > Map<String, Integer> topicCount = new HashMap<String, Integer>(); > topicCount.put("common-topic", new Integer(3)); Map<String, > List<KafkaStream<byte[], byte[]>>> consumerMap = > consumerConnector.createMessageStreams(topicCount); > > // put each stream into its own thread > List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); for > (final KafkaStream stream : streams) { > executorService2.submit(new CcsKafkaStreamConsumer(stream, > ("common-topic", messageWrapper)); } > > > -----Original Message----- > From: Kevin Scaldeferri [mailto:ke...@scaldeferri.com] > Sent: 09 March 2015 18:13 > To: users@kafka.apache.org > Subject: Re: Multiple consumer groups with same group id on a single topic > > On Mon, Mar 9, 2015 at 10:38 AM, Phill Tomlinson <philltomlin...@fico.com> > wrote: > > > Hi, > > > > I have two separate consumer groups on different JVM processes, but > > both have the same "group.id". > > > > You've said this twice, and I think it's creating some confusion, because > the group.id is exactly what determines the members of a consumer group. > > How many consumer threads are you providing total? Could you provide code > snippets to clarify how you're setting things up? > > This email and any files transmitted with it are confidential, proprietary > and intended solely for the individual or entity to whom they are > addressed. If you have received this email in error please delete it > immediately. > > This email and any files transmitted with it are confidential, proprietary > and intended solely for the individual or entity to whom they are > addressed. If you have received this email in error please delete it > immediately. > -- Regards, Tao
