Hi Andrew, Thanks for your response, well I am already keeping the number of workers equal to the number of worker nodes as I have already set that property,
Config config=new Config(); config.put(Config.TOPOLOGY_WORKERS, 2); config.setNumAckers(2); config.setNumWorkers(2); config.setMaxSpoutPending(20); config.setMaxTaskParallelism(20); What I am achieving right now with two nodes(worker) cluster is around ~1 Million msg/min which is around 16k msg/sec, which seems to be a good number with this k/o architecture. I was wondering to go par the existing performance(since the CPU is highly utilized ~> 320% on 4 core) would it be better to add more nodes or still some config. tweaking can enhance the existing performance? Thanks! On Thu, Feb 26, 2015 at 1:24 PM, Andrew Neilson <arsneil...@gmail.com> wrote: > There is a configuration setting that you should be submitting to your > topology, "topology.workers". You can see how many workers each topology is > configured to use by running "storm list" under the "num workers" column. > To confirm what Nathan suspects, you might see that this is set to 1 for > your topology. You can also see how many workers your topology is > configured to use from the Storm UI if you navigate to your running > topology and just search the page for "topology.workers". > > Changing the number of workers can be done through the configuration that > you pass to the topology like (in Java): > > Config conf = new Config(); > conf.setNumWorkers(2); > > By the way, you can find this info in the Storm tutorial: > http://storm.apache.org/documentation/Tutorial.html. There is also some > great documentation on understanding parallelism in Storm here: > http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html > > On Wed, Feb 25, 2015 at 9:39 PM, Vineet Mishra <clearmido...@gmail.com> > wrote: > >> Hi Nathan, >> >> I guess you missed the first mail of this thread. >> >> As mentioned before I m having 3 node cluster out of which node 1 is for >> nimbus and ui while other 2 nodes are for worker and supervisor. >> >> The specification of machine is 4 core intel xeon processor which >> currently is running with more than 300% cpu utilization. >> >> At this case how can I Increase number of workers >> >> Thanks! >> On Feb 26, 2015 3:48 AM, "Nathan Leung" <ncle...@gmail.com> wrote: >> >>> If you have only one node on the cluster then yes I would recommend to >>> add more nodes. If you have more than one I would recommend you increase >>> the worker processes in the topology configuration to use the other >>> hardware. >>> On Feb 25, 2015 10:38 AM, "Vineet Mishra" <clearmido...@gmail.com> >>> wrote: >>> >>>> If I am taking you right, by adding more workers you mean adding more >>>> nodes to the existing cluster or/and enhancing the existing configuration >>>> with perhaps 8 or 16 core cpu? >>>> >>>> On Wed, Feb 25, 2015 at 6:59 PM, Nathan Leung <ncle...@gmail.com> >>>> wrote: >>>> >>>>> In practice it's very difficult to fully maximize cpu usage, >>>>> especially when each thread has to do things like wait for i/o in any way >>>>> (network, disk, etc). Since you're cpu limited, or very close to it, you >>>>> should try adding more workers. >>>>> On Feb 25, 2015 7:48 AM, "Vineet Mishra" <clearmido...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Nathan, >>>>>> >>>>>> You are right, I guess that was the case. >>>>>> >>>>>> But I am not sure why even after increasing the spouts to 20 I >>>>>> couldn't end up with max throughput, I hold 4 core Xeon processor >>>>>> Intel(R) >>>>>> Xeon(R) CPU E5-2680 v2 @ 2.80GHz, although I could see the high CPU >>>>>> utilization reaching more than 350% but is it the only reason behind the >>>>>> less performance gain than expected or do I need to add some >>>>>> configuration >>>>>> tuning. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> On Wed, Feb 25, 2015 at 5:53 PM, Nathan Leung <ncle...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> It looks like you configured your topology to only use one worker >>>>>>> process. >>>>>>> >>>>>>> On Wed, Feb 25, 2015 at 6:39 AM, Vineet Mishra < >>>>>>> clearmido...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> I am running a 3 node storm cluster with a Topology which is >>>>>>>> subscribed to Kafka using Kafka-Storm Spout. >>>>>>>> >>>>>>>> The Cluster is hosting Nimbus on a node and left with 2 other nodes >>>>>>>> for the workers and supervisor to function. >>>>>>>> >>>>>>>> While running the Topology I could see that all the spouts(20) and >>>>>>>> bolts(5) tasks are running on the same machine leaving the other node >>>>>>>> instance without any CPU utilization. >>>>>>>> >>>>>>>> [1-1] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2264180 2264180 >>>>>>>> 11.519 2262620 0 [2-2] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 841540 841540 >>>>>>>> 19.717 840940 0 [3-3] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1080080 1080080 >>>>>>>> 18.105 1079240 0 [4-4] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1113900 1113900 >>>>>>>> 17.966 1105320 0 [5-5] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854220 854220 >>>>>>>> 19.342 856240 0 [6-6] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1249640 1249640 >>>>>>>> 17.071 1251320 0 [7-7] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2266520 2266520 >>>>>>>> 11.63 2267260 0 [8-8] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 629760 629760 >>>>>>>> 20.8 629120 0 [9-9] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854360 854360 >>>>>>>> 19.513 851800 0 [10-10] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1417380 1417380 >>>>>>>> 15.78 1418200 0 [11-11] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 858620 858620 >>>>>>>> 20.897 858180 0 [12-12] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1242980 1242980 >>>>>>>> 16.766 1241100 0 [13-13] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 352300 352300 >>>>>>>> 19.975 349420 0 [14-14] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 627920 627920 >>>>>>>> 20.329 625440 0 [15-15] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1061880 1061880 >>>>>>>> 18.263 1061300 0 [16-16] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 706480 706480 >>>>>>>> 20.297 702720 0 [17-17] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1518800 1518800 >>>>>>>> 15.154 1514640 0 [18-18] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 622640 622640 >>>>>>>> 20.743 626240 0 [19-19] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 3376960 3376960 >>>>>>>> 8.361 3377060 0 [20-20] 31m 7s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1044940 1044940 >>>>>>>> 18.569 1043760 0 >>>>>>>> >>>>>>>> Same is the case for bolts, >>>>>>>> >>>>>>>> [22-22]} 32m 13s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.002 0.05 >>>>>>>> 1376560 0.042 1376540 0 [23-23]} 32m 13s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.001 0.021 >>>>>>>> 347180 0.02 347180 0 [24-24]} 32m 13s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.023 >>>>>>>> 34600 0.023 34600 0 [25-25]} 32m 13s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.017 >>>>>>>> 8380 0.014 8380 0 [26-26]} 32m 13s ip-20-0-0-75 6703 >>>>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.082 0.039 >>>>>>>> 22217940 0.041 22217920 0 >>>>>>>> >>>>>>>> ip-20-0-0-75 is only utilized while their is another host which is >>>>>>>> not utilized at all. Please find the supervisor summary below >>>>>>>> >>>>>>>> 8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h 7m 32s 4 >>>>>>>> 0 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h 57m 53s >>>>>>>> 4 1 >>>>>>>> This is not the regular behavior of Storm as I have seen both the >>>>>>>> nodes operating while running the jobs, but how we can comment on the >>>>>>>> stability of the storm with this kind off behavior. >>>>>>>> >>>>>>>> Urgent Issue, Request for expert intervention. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >