Nilesh, I recommend you to take a look at this code https://github.com/HolmesNL/kafka-spout/blob/develop/src/main/java/nl/minvenj/nfi/storm/kafka/KafkaSpout.java . It uses high level consumer api. In your code , in nextTuple() you just emit all the tuples from your topic. This never ends until your topic is exhausted. My suspicion is when your spout initializes and one of the instances will just go through your topic since nextTuple never ends ( until your topic is exhausted). https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/consumer/StormKafkaConsumer.java#L71
This is handled properly in HolmesNL kafka-spout. -Harsha. On Tue, Dec 23, 2014, at 10:22 AM, Nilesh Chhapru wrote: > Hi Harsha, > > PFB the link for git. > > https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer > > Also there is the caller program in with method : > setHighLevelKafkaSpout > https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java > > > *Regards*, > *Nilesh Chhapru.* > > *From:* Harsha [mailto:st...@harsha.io] > > *Sent:* 18 December 2014 10:34 PM *To:* user@storm.apache.org > *Subject:* Re: Issues With Parallelism In Kafka Spout > > Its going to be difficult without looking your code. If its on github > do give us a link to it. > -Harsha > > > On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote: >> Hi Harsha, >> >> The partitions and topics are having the data and the zookeeper also >> shows the group id of all the consumer in its logs. >> >> *Regards*, >> *Nilesh Chhapru.* >> >> *From:* Harsha [mailto:st...@harsha.io] >> >> *Sent:* 18 December 2014 09:13 PM *To:* user@storm.apache.org >> *Subject:* Re: Issues With Parallelism In Kafka Spout >> >> Hi Nilesh, >> Did you check if your kafka topic partitions all being writtten to. >> -Harsha >> >> >> On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote: >>> Hi All, >>> >>> Please give some inputs as this is pending since long and need to >>> meet the deadlines >>> >>> *Regards*, >>> *Nilesh Chhapru.* >>> >>> *From:* Nilesh Chhapru [mailto:nilesh.chha...@ugamsolutions.com] >>> >>> *Sent:* 18 December 2014 01:24 PM *To:* user@storm.apache.org; >>> us...@kafka.apache.org *Subject:* Issues With Parallelism In >>> Kafka Spout >>> >>> Hi All, >>> >>> I have implemented a high level Kafka consumer in Storm but looks >>> like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data. >>> PFB the screen grab for number of task of spout and data emitted by >>> only one of them. >>> >>> Please assist on how to achieve parallelism using high level Kafka >>> spout. >>> >>> >>> >>> *Regards*, >>> *Nilesh Chhapru.* >>> >>> >>> >>> >>> >>> ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- >>> >>> ****Opinions expressed in this e-mail are those of the author and do >>> not necessarily represent those of Ugam. Ugam does not accept any >>> responsibility or liability for it. This e-mail message may contain >>> proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. >>> >>> Warning: Sufficient measures have been taken to scan any presence of >>> viruses however the recipient should check this email and any >>> attachments for the presence of viruses. Ugam accepts no liability >>> for any damage caused by any virus transmitted by this email. **** >>> >>> >>> >>> ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- >>> >>> ****Opinions expressed in this e-mail are those of the author and do >>> not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. >>> >>> Warning: Sufficient measures have been taken to scan any presence of >>> viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. **** >> >> >> >> ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- >> >> ****Opinions expressed in this e-mail are those of the author and do >> not necessarily represent those of Ugam. Ugam does not accept any >> responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. >> >> Warning: Sufficient measures have been taken to scan any presence of >> viruses however the recipient should check this email and any >> attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. **** > > > ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- > > ****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. > > Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
