Topologies, Slots & Memory limitations on Storm

Hello, In my application, I need to run several 'topologies' in parallel. I understand I can have a number of 'slots', which can be assigned. Once the number of slots available are used, no topology actually runs, even if they can load. If i wanted to run many topologies, I need to set a hi

supervisor uptime

Hi, All I didn't notice before, but just see the uptime of storm supervisor node in storm UI are 7h32m42s 1d 21h 53m 10s 9h 3m 31s 8h 10m 2s However, they should be up for few weeks, since I've been running topologies on the cluster. I actually use upstart to control storm

BCP for nimbus?

Hello, Are there any place for BCP on Nimbus hosts? Also may I know what is stopping that considering all the state is in zookeeper? Thanks Sid

RE: configuring topology.workers

Thank you for the clarification and pointing me to the JIRA case "So in the above pull request, I implemented a solution in the way similar to watershed algorithm. It would firstly pick the slots from least used host, until that host uses the same number of slots as the second least used slots.

Re: configuring topology.workers

My understudying of the DefaultScheduler isn’t complete, so take this with a grain of salt. As of 0.9.3, if I understand correctly: The list of available slots are sorted based on number of free slots on the same supervisor as the given slot then on port number. If there are two or more slots t

Re: Trident read from Kafka borkers, processes multiple times

Looks like you’re using OpaqueTridentKafkaSpout? topology.max.spout.pending, for a Trident spout, refers to the number of in-flight batches. When you set this number too high, OpaqueTridentKafkaSpout sort of freaks out and emits the same tuples in several batches. The batches aren’t failing; th

Re: Storm UI metrics

You should probably use a parallelism value of 1, to have a single global *LoggingMetricsConsumer.* conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 1); It will write the metrics into $STORM_HOME/logs/metrics.log 2015-03-13 15:29 GMT+01:00 Nick R. Katsipoulakis : > Thanks a lot Marti

Re: Storm UI metrics

Thanks a lot Martin. I will take a look at both of the links. Cheers, Nikos 2015-03-13 10:27 GMT-04:00 Martin Illecker : > I believe the *LoggingMetricsConsumer* [1] will do the job. > > conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 2); > > > Have a look at this tutorial [2]. > > [1

Re: Storm UI metrics

I believe the *LoggingMetricsConsumer* [1] will do the job. conf.registerMetricsConsumer(LoggingMetricsConsumer.class, 2); Have a look at this tutorial [2]. [1] https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/metric/LoggingMetricsConsumer.java [2] https://www.e

Storm UI metrics

Hello all, In Storm UI I can see the following metrics for each bolt/spout in an active topology: Capacity and Latency. Is there a way to get those metrics from the bolt's code and store them in a custom log file. Thank you, Nikos -- Nikolaos Romanos Katsipoulakis, University of Pittsburgh, Ph

Re: Understanding parallelism

Hi all, Thanks a lot for great clarification. So if I have multiple tasks in one executor, no of executors will increase upon re-balancing. But no change will happen with default one task per executor concept. Thanks, Tishan On Fri, Mar 13, 2015 at 12:49 PM, Kosala Dissanayake wrote: > I am pr

RE: Trident read from Kafka borkers, processes multiple times

After I remove the storm configuration "topology.max.spout.pending", the trident workload runs well. But I still get a little confused if I should set this parameter to improve parallelism when processing trident topology. From: Qian, Shilei [mailto:shilei.q...@intel.com] Sent: Tuesday, March 1

Re: Understanding parallelism

I am pretty sure Nathan is referring to rebalancing in that response. *'When you set the parallelism to 'x', you will have 'x' executors forever*.' No. The number of *tasks *is static. You can change the number of *executors* using the rebalance command. Since from 0.8.0 'parallelism' refers to t

Re: Understanding parallelism

On the fly change is by stome rebalance command. storm rebalance topology-name> -e component=parallelism On 3/13/15, tishan pubudu kanishka dahanayakage wrote: > Hi Kosala, > > Thanks for the response. Yeah. I came across that. But that was written in > 2012 whereas [1] is more recently. it says

Re: Understanding parallelism

Hi Kosala, Thanks for the response. Yeah. I came across that. But that was written in 2012 whereas [1] is more recently. it says "Note that as of Storm 0.8 the parallelism_hint parameter now specifies the ​'​ initial ​'​ number of executors (not tasks!) for that bolt". Also in here[2] Nathan says