I would expect that it depends on how many executors you have. In storm, an executor corresponds to an OS thread while a task is more of a logical unit of work. The only situation where I would personally use more tasks than executors is if I wanted to over provision the tasks so that I can rebalance to use more executors in the future (you cannot change number of tasks in rebalance).
When you have more tasks than executors, the spout thread does the same logic, it just does it for more tasks during its main loop. I'm not sure why that would increase your per thread throughput. On May 13, 2015 10:13 PM, "yang...@bupt.edu.cn" <yang...@bupt.edu.cn> wrote: > hi,Nathan > actually I tried many ways to make my program fit the Storm > 1: a 'while(true)' in the nextTuple() > 2: execute n times in one nextTuple. > I don't need to batch messages because what I really care is the speed > it processes(emit phase is not the bottleneck). > I want to mention this: I only created one single spout task in one > machine node. > and I read some papers about storm evaluation, they did some parrallelism > to some extent. So I tried to add some parrallelism(10tasks per executor > per node),and I got a pretty good result(the same throughout with the java > program). > I wonder if this is the design pattern we should pick in storm? > > ------------------------------ > yang...@bupt.edu.cn > > *From:* Nathan Leung <ncle...@gmail.com> > *Date:* 2015-05-12 20:57 > *To:* user <user@storm.apache.org> > *Subject:* Re: How much is the overhead of time to deploy a system on > Storm ? > > I'm not very surprised. See for example published single machine > benchmarks (iirc 1.6 million tuples / s is the official figure from Nathan > Marz though that figure is a little old). This is spout to bolt and matches > my observations for trivial cases. With some processing logic and only one > spout I can see how it's lower. > > You can reduce the overhead by batching your work differently, eg by doing > more work in each call to nextTuple. > On May 12, 2015 4:56 AM, "Matthias J. Sax" <mj...@informatik.hu-berlin.de> > wrote: > >> Can you share your code? >> >> Do you process a single tuple each time nextTuple() is called? If a >> spout does not emit anything, Storm applies a waiting-penalty to avoid >> busy waiting. That might slow down your code. >> >> You can configure the waiting strategy: >> https://storm.apache.org/2012/09/06/storm081-released.html >> >> -Matthias >> >> >> On 05/12/2015 09:31 AM, Daniel Compton wrote: >> > I'm also interested on the answers to this question, but to add to the >> > discussion, take a look at >> > >> http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html >> . >> > I suspect Storm is still introducing coordination overhead even running >> > on a single machine. >> > On Tue, 12 May 2015 at 1:39 pm yang...@bupt.edu.cn >> > <mailto:yang...@bupt.edu.cn> <yang...@bupt.edu.cn >> > <mailto:yang...@bupt.edu.cn>> wrote: >> > >> > __ >> > Hi and thanks . >> > >> > I'm working on a parrallel algorithm, which is to count massive >> > items in data streams. The previous researches on the parallelism of >> > this algorithm were focusing on muti-core CPU, however, I want to >> > take advantage of Storm. >> > >> > Processing latency is extremly important for this algorithm, and I >> > did some evaluation of the perfomance. >> > >> > Firstly, I implemented the algorithm in java(one thread, with no >> > parallelism) and I get the performance : it could process 3 million >> > items per second. >> > >> > Secondly, I wrapped this implement of the algorithm into Storm(just >> > one Spout to process) and I get the perfomance: it could process >> > only 0.75 million items per second. I changes a little bit of my >> > impletment to adapt Storm structure, but in the end the perfomance >> > is still not good.... >> > >> > ps. I didn't take the network overhead into consideration because I >> > just run the program in the single Spout node so that there is no >> > emit or transfer.(so I don't care how storm emits messages between >> > nodes for now ) The program on Spout is actually doing the same >> > thing as the former one.(I just copy the program into the >> > NextTuple() method with some necessary changes) >> > >> > 1. The degration(1/4 of the speed) is inevitable? >> > 2. What incurred the degration? >> > 3. How can I reduce the degration? >> > >> > Thank you all. >> > >> > >> ------------------------------------------------------------------------ >> > yang...@bupt.edu.cn <mailto:yang...@bupt.edu.cn> >> > >> >>
