I would suggest to use the new java producer if possible. It is more efficient and does the round robin by default.
Jiangjie (Becket) Qin On 3/10/15, 3:28 AM, "tao xiao" <xiaotao...@gmail.com> wrote: >The default partitioner of old producer API is a sticky partitioner that >keeps sending messages to the same partition for 10 secs (I don't remember >the exact length duration) before switch to another partition if no key is >specified in message. You can easily override this by setting >partitioner.class > >On Tue, Mar 10, 2015 at 4:32 PM, Phill Tomlinson <philltomlin...@fico.com> >wrote: > >> Hi, >> >> " Unless the producer is always putting the messages into one >>partition, I >> would expect both consumer groups to read from the topic." >> >> This was the issue. I thought the producer would round robin messages to >> different partitions but it must have just been writing to a single >> partition. I re-wrote the producer to use the new API and target set >> partitions and I can now see both consumer groups consuming from >>different >> partitions. >> >> Phill >> >> -----Original Message----- >> From: Phill Tomlinson [mailto:philltomlin...@fico.com] >> Sent: 10 March 2015 08:18 >> To: users@kafka.apache.org >> Subject: RE: Multiple consumer groups with same group id on a single >>topic >> >> Hi, >> >> Code snippet below. This creates two consumers with same group id >> "consumer-group", they consume from "common-topic" which has 6 >>partitions. >> Each group has 3 consumers. >> >> However only one of the groups will ever run. Unless the producer is >> always putting the messages into one partition, I would expect both >> consumer groups to read from the topic. >> >> Thanks, >> Phill >> >> Properties consumerProps = new Properties(); >> consumerProps.put("zookeeper.connect", zookeeperNode); >>consumerProps.put(" >> group.id", "consumer-group"); final ConsumerConnector consumerConnector >>= >> Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps)); >> Map<String, Integer> topicCount = new HashMap<String, Integer>(); >> topicCount.put("common-topic", new Integer(3)); Map<String, >> List<KafkaStream<byte[], byte[]>>> consumerMap = >> consumerConnector.createMessageStreams(topicCount); >> >> // put each stream into its own thread >> List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); for >> (final KafkaStream stream : streams) { >> executorService1.submit(new CcsKafkaStreamConsumer(stream, >> ("common-topic", messageWrapper)); } >> >> >> >> Properties consumerProps = new Properties(); >> consumerProps.put("zookeeper.connect", zookeeperNode); >>consumerProps.put(" >> group.id", "consumer-group"); final ConsumerConnector consumerConnector >>= >> Consumer.createJavaConsumerConnector(new ConsumerConfig(consumerProps)); >> Map<String, Integer> topicCount = new HashMap<String, Integer>(); >> topicCount.put("common-topic", new Integer(3)); Map<String, >> List<KafkaStream<byte[], byte[]>>> consumerMap = >> consumerConnector.createMessageStreams(topicCount); >> >> // put each stream into its own thread >> List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic); for >> (final KafkaStream stream : streams) { >> executorService2.submit(new CcsKafkaStreamConsumer(stream, >> ("common-topic", messageWrapper)); } >> >> >> -----Original Message----- >> From: Kevin Scaldeferri [mailto:ke...@scaldeferri.com] >> Sent: 09 March 2015 18:13 >> To: users@kafka.apache.org >> Subject: Re: Multiple consumer groups with same group id on a single >>topic >> >> On Mon, Mar 9, 2015 at 10:38 AM, Phill Tomlinson >><philltomlin...@fico.com> >> wrote: >> >> > Hi, >> > >> > I have two separate consumer groups on different JVM processes, but >> > both have the same "group.id". >> >> >> >> You've said this twice, and I think it's creating some confusion, >>because >> the group.id is exactly what determines the members of a consumer group. >> >> How many consumer threads are you providing total? Could you provide >>code >> snippets to clarify how you're setting things up? >> >> This email and any files transmitted with it are confidential, >>proprietary >> and intended solely for the individual or entity to whom they are >> addressed. If you have received this email in error please delete it >> immediately. >> >> This email and any files transmitted with it are confidential, >>proprietary >> and intended solely for the individual or entity to whom they are >> addressed. If you have received this email in error please delete it >> immediately. >> > > > >-- >Regards, >Tao
