Re: Max Spout Pending - Question

Thanks Nathan. Will go with assumption that if latency at spout is much higher than combined latencies at rest of bolts, then the additional latencies at spout are due to messages piling on out bound queue.. Regards Kashyap On Jul 29, 2015 1:49 PM, "Nathan Leung" wrote: > Roughly speaking (disre

Re: Max Spout Pending - Question

Roughly speaking (disregarding any network latencies and any benefits from having multiple threads servicing the output queues vs having 1), having 1 spout with high max pending should be similar to many spouts with low max pending. Your total number of tuples pending in the topology is the same e

Fwd: Trident DRPC not distributing task to all workers

Hi, I am using storm 0.8.2 , trident DRPC for my log processing. We have a very heavy task and trying to do it with drpc. Input to drpc is a file name , output is usually 1GB string data. Problem : DRPC is not assigning task to all workers. Currently running 5 nodes.Other 4 nodes using given conf

Re: Max Spout Pending - Question

Nathan, So the following is true? - Spout Latency = (Time spent output queues *[A]*)+(time difference between emit from NextTuple and time spent in acking *[B]*) So does it mean that if the complete latency at spout level is high but the bolts have very low latencies, then instead of increasing th

Re: Clarification regarding packaging of storm-core jar

Thank you Taylor and Grant for your replies. I will try the approaches suggested by both of you. Richards Peter.

Re: Max Spout Pending - Question

1 second is too short. Spout latency includes time spent in the output queue from the spout (increasing max spout pending potentially increases your end-to-end latency, depending on whether you have anything buffered in the spout output queues). On Wed, Jul 29, 2015 at 12:40 PM, Kashyap Mhaisekar

Re: Max Spout Pending - Question

Thanks Nathan. But in this case how should the Spout Latency be interpreted. In the same example you quoted above - spout a -> bolt b (emits 10 tuples per msg) -> bolt c I see the process latency and execute latencies under 5 ms both for B and C. While the spout is at 1500ms. The Bolts dont don an

Re: Max Spout Pending - Question

No. You need to consider your system more carefully. As a trivial example, imagine you have spout a -> bolt b -> bolt c, with bolt b splitting tuple into 10 tuples. Each component has 1 task. If each component takes 1ms, your latency will not be the sum of the per bolt latency because of your f

Re: Clarification regarding packaging of storm-core jar

Hi Richards, The transitive dependencies that are now included in the storm-core jar file have their package names “relocated” so users can include their own versions of those dependencies without encountering dependency conflicts. The best practice is to include your own version of those depen

Re: Max Spout Pending - Question

Thanks Nathan. If I see the complete latency at spout is greater than the process latencies of all bolts put together, does it mean that the ACKERS are a problem and need to be increased? thanks kashyap On Tue, Jul 28, 2015 at 7:30 PM, Nathan Leung wrote: > The count is tracked from each spout

Re: Clarification regarding packaging of storm-core jar

I’d leave storm/lib vanilla and package the higher versions of those libs with your topology jar using maven shade to relocate the necessary packages to avoid conflict. Here is an example from a storm topology that relocates guava (com.google.common). org.apache.maven.plugins maven-s

Clarification regarding packaging of storm-core jar

Hi, I was using storm-0.8.2 so far in my project. I am starting to evaluate storm-0.9.*. I found a difference in the way storm-core(0.9.4) /storm(0.8.2) jar has been packaged. I am not an expert in this packaging concept. So I thought of taking some advice from the experts. In storm 0.8.* jar, I

Recommended Scheduler--Isolation Scheduler or Mesos

Hi Everyone, For a Storm cluster running multiple topologies, what is the recommended scheduler--Mesos or the Isolation Scheduler? The reasons I ask are per this post back in 2013: https://groups.google.com/forum/#!topic/storm-user/GQEH-6x6oyI and the fact we are seeing some issues where a partic

RE: Unsuscribe

storm use logback，so you do not need to restart storm。 > Date: Tue, 28 Jul 2015 17:48:05 +0200 > From: i...@lambdoop.com > To: user@storm.apache.org > Subject: Unsuscribe > > Unsuscribe > > - Mensaje original - > De: "张炜" > Para: user@storm.apache.org > Enviados: Jueves, 2 de Julio 20

Re: Storm execution

Did you package all your classes correctly in your jar file that is submitted to Nimbus? -Matthias On 07/29/2015 09:39 AM, Vamsikrishna Vinjam wrote: > i can run the topologies which are in storm examples..but i can the > topologies which i have written if i try to run them i getting error like >

Storm execution

i can run the topologies which are in storm examples..but i can the topologies which i have written if i try to run them i getting error like "could not find or load class"