I do not know how these mechanisms you mentioned (SQS, S3 and database), but Kafka is recommended to set the parallelism hit number according to the amount of Kafka partitions <https://kafka.apache.org/documentation/>. The number of parallelism hit there is in addition to the amount of Kafka partitions the executors have become idle. I have seen recommendations in hortonworks <http://hortonworks.com/blog/> that the number of executors does not exceed the number of cores.
-- Thomas Cristanis 2017-02-27 15:05 GMT-03:00 pradeep s <sreekumar.prad...@gmail.com>: > Hi, > I am running a 5 worker node storm cluster on ec2 m4.2x large machines. > Each machine is having 8 cores . I have a spout which is consuming from > SQS and two bolts ,one to S3 and another to a database . > What will be the maximum parallelism value i can assign for Spout and Bolt. > Since i have 8*5= 40 cores in total, is 40 the max parallelism i can give > for spout and bolt . > Also if i assign 40 parallelism hint for spout, will the bolt parallelism > value can be the same? > Regards > Pradeep S >
