Number of executors < number of tasks?
I've been looking at the parallelism of a storm topology. In what situations is it useful to have more than one task running on the same executor? An executor is a thread, so if there's several tasks on that executor only one task can run at a time. So why would you want more than one task on the same thread, if only one can run at once? SimonC
Re: Number of executors < number of tasks?
If you see practically number of workers should always be less than or equal to the number of tasks. This is not only valid for Comp sci but for any field. If we have more no. of workers than tasks available then we are just wasting our resources. That is we are not using our resources optimally. So in most cases there should at least be one task or more running on an executor. Now, consider an executor which normalizes a sentence into words, now when sentence arrives the executor normalizes that sentence after it is finished it is ready to receive another tuple. With storm there could be several executors doing the same job with different sentences simultaneously but that doesn't mean we have to have as many no. of executors as there are sentences. I hope this makes sense. On Mon, Feb 17, 2014 at 11:11 PM, Simon Cooper < simon.coo...@featurespace.co.uk> wrote: > I've been looking at the parallelism of a storm topology. In what > situations is it useful to have more than one task running on the same > executor? An executor is a thread, so if there's several tasks on that > executor only one task can run at a time. So why would you want more than > one task on the same thread, if only one can run at once? > > > > SimonC > -- *Abhishek Bhattacharjee* *Pune Institute of Computer Technology*
Re: Number of executors < number of tasks?
This SO answer seems to answer your question: http://stackoverflow.com/a/17454586/234901 On Mon, Feb 17, 2014 at 7:30 PM, Abhishek Bhattacharjee < abhishek.bhattacharje...@gmail.com> wrote: > If you see practically number of workers should always be less than or > equal to the number of tasks. This is not only valid for Comp sci but for > any field. If we have more no. of workers than tasks available then we are > just wasting our resources. That is we are not using our resources > optimally. > So in most cases there should at least be one task or more running on an > executor. > Now, consider an executor which normalizes a sentence into words, now when > sentence arrives the executor normalizes that sentence after it is finished > it is ready to receive another tuple. > With storm there could be several executors doing the same job with > different sentences simultaneously but that doesn't mean we have to have as > many no. of executors as there are sentences. I hope this makes sense. > > > On Mon, Feb 17, 2014 at 11:11 PM, Simon Cooper < > simon.coo...@featurespace.co.uk> wrote: > >> I've been looking at the parallelism of a storm topology. In what >> situations is it useful to have more than one task running on the same >> executor? An executor is a thread, so if there's several tasks on that >> executor only one task can run at a time. So why would you want more than >> one task on the same thread, if only one can run at once? >> >> >> >> SimonC >> > > > > -- > *Abhishek Bhattacharjee* > *Pune Institute of Computer Technology* >
Re: Number of executors < number of tasks?
Hello! Unfortunately you cannot change the number of tasks when doing rebalancing. Considering the following scenario: 1. A storm topology with maximum parallelism of 10 tasks. 2. Your storm topology is deployed on cloud 3. Your application should scale horizontally depending on the incoming volume. 4. if the volume is high you can scale more with more machines but unfortunately you they cannot be used due to the task limit (meaning that you are adding machines that will overlap the maximum parallelism supported by your topology) I found this limitation as an inconvenience for the cloud application, where theoretically, you can scale horizontally without limit. You can check my complains about the examples from the wiki page regarding the parallelism, here : http://mail-archives.apache.org/mod_mbox/storm-user/201402.mbox/%3CCAOy42CtEz6P0kRVOEff=u83bo_317ld3zrmywgkt2yjx6qr...@mail.gmail.com%3E Hope that these help. Regards, Florin On Mon, Feb 17, 2014 at 11:58 PM, Enno Shioji wrote: > This SO answer seems to answer your question: > http://stackoverflow.com/a/17454586/234901 > > > On Mon, Feb 17, 2014 at 7:30 PM, Abhishek Bhattacharjee < > abhishek.bhattacharje...@gmail.com> wrote: > >> If you see practically number of workers should always be less than or >> equal to the number of tasks. This is not only valid for Comp sci but for >> any field. If we have more no. of workers than tasks available then we are >> just wasting our resources. That is we are not using our resources >> optimally. >> So in most cases there should at least be one task or more running on an >> executor. >> Now, consider an executor which normalizes a sentence into words, now >> when sentence arrives the executor normalizes that sentence after it is >> finished it is ready to receive another tuple. >> With storm there could be several executors doing the same job with >> different sentences simultaneously but that doesn't mean we have to have as >> many no. of executors as there are sentences. I hope this makes sense. >> >> >> On Mon, Feb 17, 2014 at 11:11 PM, Simon Cooper < >> simon.coo...@featurespace.co.uk> wrote: >> >>> I've been looking at the parallelism of a storm topology. In what >>> situations is it useful to have more than one task running on the same >>> executor? An executor is a thread, so if there's several tasks on that >>> executor only one task can run at a time. So why would you want more than >>> one task on the same thread, if only one can run at once? >>> >>> >>> >>> SimonC >>> >> >> >> >> -- >> *Abhishek Bhattacharjee* >> *Pune Institute of Computer Technology* >> > >
