At that point you have to think about what makes sense for your system right now. For example, maybe it makes sense to have # tasks = 4 times what you need right now, and then reload the topology when you outgrow that.
Alternatively, you can consider bringing up a larger replacement topology, and then killing the older one. In this case you will have to be more careful with names, and possibly things like resource (worker) allocation. On Thu, Jun 9, 2016 at 9:30 AM, Adrien Carreira <[email protected]> wrote: > So let's say one day I would like to have 100 machine, > > I should set 100 on setNumTask ? > > 2016-06-09 15:20 GMT+02:00 Nathan Leung <[email protected]>: > >> You can create your topology with more tasks than executors, then when >> the rebalance happens you can add executors. However at the moment you >> cannot add more tasks to a running topology. >> >> On Thu, Jun 9, 2016 at 8:58 AM, Adrien Carreira <[email protected]> >> wrote: >> >>> I've just create a topology like this : >>> >>> builder.setBolt("fetcher", new Fetch()) >>> .shuffleGrouping("spout"); >>> >>> builder.setBolt("extract", new Extract()) >>> .shuffleGrouping("fetcher"); >>> >>> builder.setBolt("indexer", new Indexer()) >>> .shuffleGrouping("extract"); >>> >>> >>> Means that I've three bolt with One Worker and parrallelism_hint of 1. >>> >>> Now, Let's say that I've another machine available, or that I've too many >>> tuple to process and I need another machine. >>> >>> >>> I've executed this command : >>> >>> storm rebalance kairos-who -n 2 -e indexer=2 -e fetcher=2 -e extract=2 >>> >>> >>> But what I've is two worker with : >>> >>> worker 1 => Spout + extract >>> >>> worker 2 => fetcher + indexer >>> >>> >>> What I would love : >>> >>> Worker 1 => Spout + fetcher + extract + indexer >>> >>> Worker 2 => Same... >>> >>> >>> I hope I'm clear... >>> >>> >>> >>> >>> >>> >>> >>> 2016-06-09 14:47 GMT+02:00 Andrew Xor <[email protected]>: >>> >>>> Hello, >>>> >>>> I am sorry, but I don't know why you cannot emulate those scale up >>>> factors by using rebalance; after all it spawns the requested amount of >>>> workers (in topology) and executors (in spouts/bolts) only bounded by the >>>> topology_max_task_parallelism. Have you read the article in order to >>>> understand how parallelism works in storm? >>>> >>>> Regards. >>>> >>>> On Thu, Jun 9, 2016 at 3:34 PM, Adrien Carreira <[email protected]> >>>> wrote: >>>> >>>>> Yes, >>>>> >>>>> But the rebalance command doesn't do what I would like. >>>>> >>>>> >>>>> Let's suppose that I've : >>>>> >>>>> SPOUT A (1) => BOLT 1 (1) => BOLT2 (1) => BOLT3 (3) >>>>> >>>>> (number is the parallelism hint) >>>>> It means that If I scale to n worker I would like : >>>>> >>>>> SPOUT A (1*n) => BOLT 1 (1*n) => BOLT2 (1*n) => BOLT3 (3*n) >>>>> >>>>> >>>>> But, the storm rebalance keeps the parralisme_hint :/ >>>>> >>>>> >>>>> >>>>> 2016-06-09 14:29 GMT+02:00 Andrew Xor <[email protected]>: >>>>> >>>>>> Hello, >>>>>> >>>>>> Why not use the rebalance command? It's well documented here >>>>>> <http://storm.apache.org/releases/current/Understanding-the-parallelism-of-a-Storm-topology.html> >>>>>> . >>>>>> >>>>>> Regards. >>>>>> >>>>>> On Thu, Jun 9, 2016 at 3:22 PM, Adrien Carreira <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> After a month building a topology on storm. I've one question about >>>>>>> parallelism that I can't answer. >>>>>>> >>>>>>> I've developed my topology and tested on a cluster with two nodes. >>>>>>> >>>>>>> My parallelism_hint are ok, everything are fine. >>>>>>> >>>>>>> My question is, if I need to scale the number of worker in the >>>>>>> topology to have more worker dooing the same thing how can I achieve >>>>>>> that >>>>>>> without kill/restart the topology >>>>>>> >>>>>>> Thanks for your reply >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
