Hi Kobi, In standard implementation we have the offsets maintained by kafka which causes issues, need to implement high level consumer which maintains in the zookeeper.
Moreover the partition with standard one isn’t working as desired as it stops the process in middle for some time if the partition is empty(eg partitions one stops even if it has data if partitions 2 is empty) Regards, Nilesh Chhapru. From: Kobi Salant [mailto:[email protected]] Sent: 18 December 2014 02:02 PM To: [email protected] Subject: Re: Issues With Parallelism In Kafka Spout Hi Nilesh, Why don't you use the standard storm-contrib kafka spout? It has built in mechanism for parallelism that split the partitions according to the task number of the spout, Kobi On Thu, Dec 18, 2014 at 8:57 AM, Nilesh Chhapru <[email protected]<mailto:[email protected]>> wrote: Hi All, I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data. PFB the screen grab for number of task of spout and data emitted by only one of them. Please assist on how to achieve parallelism using high level Kafka spout. [cid:[email protected]] Regards, Nilesh Chhapru. ________________________________ ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- ****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. **** This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you. ________________________________ ---------------------------------------------------------------------------------------Disclaimer---------------------------------------------------------------------------------------------- ****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media. Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
