Harsha asked a similar question, but I would definitely make certain that messages being written to kafka are being partitioned the way you are expecting. If you are trying to consume live events coming from a lone kafka producer using default configuration, then messages are only going to appear in one partition at a time.
See the Kafka FAQ for details https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? On Tue, Dec 23, 2014 at 10:22 AM, Nilesh Chhapru < [email protected]> wrote: > Hi Harsha, > > > > PFB the link for git. > > > > > https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer > > > > Also there is the caller program in with method : setHighLevelKafkaSpout > > > https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java > > > > > > *Regards*, > > *Nilesh Chhapru.* > > > > *From:* Harsha [mailto:[email protected] <[email protected]>] > *Sent:* 18 December 2014 10:34 PM > > *To:* [email protected] > *Subject:* Re: Issues With Parallelism In Kafka Spout > > > > Its going to be difficult without looking your code. If its on github do > give us a link to it. > > -Harsha > > > > > > On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote: > > Hi Harsha, > > > > The partitions and topics are having the data and the zookeeper also shows > the group id of all the consumer in its logs. > > > > *Regards*, > > *Nilesh Chhapru.* > > > > *From:* Harsha [mailto:[email protected] <[email protected]>] > *Sent:* 18 December 2014 09:13 PM > *To:* [email protected] > *Subject:* Re: Issues With Parallelism In Kafka Spout > > > > Hi Nilesh, > > Did you check if your kafka topic partitions all being > writtten to. > > -Harsha > > > > > > On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote: > > Hi All, > > > > Please give some inputs as this is pending since long and need to meet the > deadlines > > > > *Regards*, > > *Nilesh Chhapru.* > > > > *From:* Nilesh Chhapru [mailto:[email protected] > <[email protected]>] > *Sent:* 18 December 2014 01:24 PM > *To:* [email protected]; [email protected] > *Subject:* Issues With Parallelism In Kafka Spout > > > > Hi All, > > > > I have implemented a high level Kafka consumer in Storm but looks like the > parallelism isn’t getting achieved as I have 3 partitions and 2 task for > the spout, but only one of it is emitting the data. > > PFB the screen grab for number of task of spout and data emitted by only > one of them. > > > > Please assist on how to achieve parallelism using high level Kafka spout. > > > > > > > > *Regards*, > > *Nilesh Chhapru.* > > > > > ------------------------------ > > > > > > > ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- > > ****Opinions expressed in this e-mail are those of the author and do not > necessarily represent those of Ugam. Ugam does not accept any > responsibility or liability for it. This e-mail message may contain > proprietary, confidential or legally privileged information for the sole > use of the person or entity to whom this message was originally addressed. > Any review, re-transmission, dissemination or other use of or taking of any > action in reliance upon this information by persons or entities other than > the intended recipient is prohibited. If you have received this e-mail in > error, please delete it and all attachments from any servers, hard drives > or any other media. > > Warning: Sufficient measures have been taken to scan any presence of > viruses however the recipient should check this email and any attachments > for the presence of viruses. Ugam accepts no liability for any damage > caused by any virus transmitted by this email. **** > > > ------------------------------ > > > > > ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- > > > > ****Opinions expressed in this e-mail are those of the author and do not > necessarily represent those of Ugam. Ugam does not accept any > responsibility or liability for it. This e-mail message may contain > proprietary, confidential or legally privileged information for the sole > use of the person or entity to whom this message was originally addressed. > Any review, re-transmission, dissemination or other use of or taking of any > action in reliance upon this information by persons or entities other than > the intended recipient is prohibited. If you have received this e-mail in > error, please delete it and all attachments from any servers, hard drives > or any other media. > > > > Warning: Sufficient measures have been taken to scan any presence of > viruses however the recipient should check this email and any attachments > for the presence of viruses. Ugam accepts no liability for any damage > caused by any virus transmitted by this email. **** > > > > > ------------------------------ > > > ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- > > > > ****Opinions expressed in this e-mail are those of the author and do not > necessarily represent those of Ugam. Ugam does not accept any > responsibility or liability for it. This e-mail message may contain > proprietary, confidential or legally privileged information for the sole > use of the person or entity to whom this message was originally addressed. > Any review, re-transmission, dissemination or other use of or taking of any > action in reliance upon this information by persons or entities other than > the intended recipient is prohibited. If you have received this e-mail in > error, please delete it and all attachments from any servers, hard drives > or any other media. > > > > Warning: Sufficient measures have been taken to scan any presence of > viruses however the recipient should check this email and any attachments > for the presence of viruses. Ugam accepts no liability for any damage > caused by any virus transmitted by this email. **** > > > > ------------------------------ > > ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- > > ****Opinions expressed in this e-mail are those of the author and do not > necessarily represent those of Ugam. Ugam does not accept any > responsibility or liability for it. This e-mail message may contain > proprietary, confidential or legally privileged information for the sole > use of the person or entity to whom this message was originally addressed. > Any review, re-transmission, dissemination or other use of or taking of any > action in reliance upon this information by persons or entities other than > the intended recipient is prohibited. If you have received this e-mail in > error, please delete it and all attachments from any servers, hard drives > or any other media. > > Warning: Sufficient measures have been taken to scan any presence of > viruses however the recipient should check this email and any attachments > for the presence of viruses. Ugam accepts no liability for any damage > caused by any virus transmitted by this email. **** >
