JVM heap options for Supervisor

2014-04-06 Thread Srinath C
Hi, I'm trying to figure out a reasonable amount of heap to grant to the supervisor process on the storm worker machines. What are the factors that must be considered? I'm thinking - size of the topology jar and number of slots should be considered. But would like to hear if anyone was able to

Re: Replace Netty with ZeroMQ in Storm 0.9.1

2014-04-06 Thread Daria Mayorova
Hi Neelesh, We are having some issues with the running topology, I described it here - https://www.mail-archive.com/user@storm.incubator.apache.org/msg01655.html We want to try using 0mq to check if replacing netty with it will make any change. On Sun, Apr 6, 2014 at 3:28 PM, Neelesh wrote: >

Re: Replace Netty with ZeroMQ in Storm 0.9.1

2014-04-06 Thread Daria Mayorova
Thank you for the link, Svend. But I'm afraid this doesn't work "out of box" with the latest version. On Sun, Apr 6, 2014 at 9:49 AM, Svend Vanderveken < svend.vanderve...@gmail.com> wrote: > Daria, > > Here are the old installation instructions of Storm (pre Netty), that > describe how to insta

Re: ACK performance hit & Loggly abandoning Storm

2014-04-06 Thread Benjamin Black
There is also no indication they are configuring Kafka such that it won't lose data in the event of a broker failure: http://aphyr.com/posts/293-call-me-maybe-kafka http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen The performance hit to Kafka is similar to Storm when you

Re: Topology is stuck

2014-04-06 Thread Jason Jackson
This could happen if you have corrupted input that always causes a batch to fail and be retried. I have seen this behaviour before and I didn't see corrupted input. It might be a bug in trident, I'm not sure. If you figure it out please update this thread and/or submit a patch. On Mon, Mar 31,

Re: Emitting/Acking in a Bolt

2014-04-06 Thread Jason Jackson
you only need to call ack once per input tuple, after all your emits are done. so at the end. On Mon, Mar 31, 2014 at 7:31 AM, Phil Burress wrote: > Quick question aboutacking. If I had a bolt that had a loop; and inside > that loop it emits a message to the stream for each iteration, should I a

Re: monitoring queuing delay and end-to-end latency?

2014-04-06 Thread Jason Jackson
end-to-end processing latency is the shown when acking is enabled and the "complete latency" for the spout. https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing On Thu, Apr 3, 2014 at 1:48 PM, Dong Mo wrote: > Dear list, > > Is there a way to monitor the end to end proces

Re: ACK performance hit & Loggly abandoning Storm

2014-04-06 Thread Jason Jackson
It would of been far more useful if they measured the systems in terms of dollars, as each system makes different tradeoffs. Certainly when you enable acking you may become bottlenecked on CPU at that point instead of being bottlenecked on disk/kafka. So one thing you can do is move to hardware wit

Re: Basic storm question

2014-04-06 Thread Jason Jackson
zookeeper doesn't retain a version history. On Fri, Apr 4, 2014 at 12:22 AM, M Tarkeshwar Rao < m.tarkeshwar@ericsson.com> wrote: > Hi, > > > > Is it possible to get older version of the data of znode? > > > > Regards > > Tarkeshwar >

Re: hiding storm parameters in Storm UI

2014-04-06 Thread Jason Jackson
no. However there is a button at the bottom of the Topology UI "show/hide system stats". Likely these are already hidden for you. On Sun, Apr 6, 2014 at 3:38 AM, Vladi Feigin wrote: > Hi All, > > Is there way to hide the exposing of some parameters in the storm UI? > > Thank you, > Vlad >

Re: Replace Netty with ZeroMQ in Storm 0.9.1

2014-04-06 Thread Neelesh
Hi Daria, I'm curious about why you decided to go back to zmq instead of Netty. Did you hit any snags? On Fri, Apr 4, 2014 at 6:34 AM, Daria Mayorova wrote: > Thanks for the info, Jon. > > You're right, I've found this repository - > https://github.com/ptgoetz/storm-0mq. I guess this is what yo

Re: Replace Netty with ZeroMQ in Storm 0.9.1

2014-04-06 Thread Svend Vanderveken
Daria, Here are the old installation instructions of Storm (pre Netty), that describe how to install a previous version, I think this was written with 0.8.2 in mind, together with its 0mq and jzmq dependencies, hopefully this should help you get 0mz working with the latest version of Storm : http

hiding storm parameters in Storm UI

2014-04-06 Thread Vladi Feigin
Hi All, Is there way to hide the exposing of some parameters in the storm UI? Thank you, Vlad