Re: [VOTE] Storm Logo Contest - Final Round

2014-06-10 Thread Jason Jackson
#10 - 5pts. On Tue, Jun 10, 2014 at 11:52 AM, Petter Abrahamsson pet...@jebus.nu wrote: #10 - 5pts On Mon, Jun 9, 2014 at 2:38 PM, P. Taylor Goetz ptgo...@gmail.com wrote: This is a call to vote on selecting the winning Storm logo from the 3 finalists. The three candidates are: *

Re: resource aware task scheduling in Storm cluster

2014-06-10 Thread Jason Jackson
Hi Alex, We're using a slightly modified version of https://github.com/nathanmarz/ storm-mesos in production. Currently there's no one supporting storm-mesos, nor have we been open sourcing our changes in this project at this point in time. // Jason On Mon, Mar 3, 2014 at 3:00 PM, Alexander S.

Re: [VOTE] Storm Logo Contest - Round 1

2014-05-16 Thread Jason Jackson
#10 - 5 points. On Fri, May 16, 2014 at 1:34 PM, Brian Enochson brian.enoch...@gmail.comwrote: #10 - 3 Points. #1 - 1 Point #2 - 1 Point Thanks, Brian On Thu, May 15, 2014 at 12:28 PM, P. Taylor Goetz ptgo...@gmail.comwrote: This is a call to vote on selecting the top 3 Storm logos

Re: Topology is stuck

2014-04-10 Thread Jason Jackson
consistency since all observers will see all updates in the same order. On Wed, Apr 9, 2014 at 2:50 PM, Jason Jackson jasonj...@gmail.com wrote: I have one theory that because reads in zookeeper are eventually consistent, this is a necessary condition for the bug to manifest. So one way to test

Re: Topology is stuck

2014-04-10 Thread Jason Jackson
more. On Thu, Apr 10, 2014 at 1:36 AM, Jason Jackson jasonj...@gmail.com wrote: Hi Ted, thanks for clearing up the language, I intended to express sequential consistency then. Yes you could do a forced sync too, that would be another way good test. Taylor, the bug that I witnessed only

Re: Problem in spouts emission

2014-04-10 Thread Jason Jackson
does the problem still occur if you disable acking? conf.setNumAckers(0). On Wed, Apr 9, 2014 at 9:48 AM, Shobha shobha.p...@gmail.com wrote: Hi All, We have storm cluster with 5 nodes, 25 spouts, 250 bolts, 25 workers. Storm is consuming the messages from kafka queue, and kafka queue has

Re: setting trident transaction window

2014-04-10 Thread Jason Jackson
trident doesn't expose tick tuples in it's API yet, even though it was added in storm a while ago. There's two problems I think you're talking about (1) windowed aggregations (2) reducing DB load. For (1) Trident can do aggregations at the batch level but this doesn't really help you for doing

Re: Topology is stuck

2014-04-10 Thread Jason Jackson
way to avoid such problems is to use a higher level library such as Curator. I forget if Storm already uses Curator, but I seem to remember not. On Thu, Apr 10, 2014 at 1:39 AM, Jason Jackson jasonj...@gmail.comwrote: My idea for the bug was that trident expects to read from zookeeper what

Re: Problem in spouts emission

2014-04-10 Thread Jason Jackson
? Regards, Shobha Rani Poli On Thu, Apr 10, 2014 at 3:41 AM, Jason Jackson jasonj...@gmail.comwrote: does the problem still occur if you disable acking? conf.setNumAckers(0). On Wed, Apr 9, 2014 at 9:48 AM, Shobha shobha.p...@gmail.com wrote: Hi All, We have storm cluster with 5 nodes

Re: Topology is stuck

2014-04-08 Thread Jason Jackson
mean by corrupted input? Do you mean that there's a bug in Trident itself that causes the tuples in a batch to somehow become corrupted? Thanks a lot! Danijel On Monday, April 7, 2014, Jason Jackson jasonj...@gmail.com wrote: This could happen if you have corrupted input that always

Re: JVM heap options for Supervisor

2014-04-08 Thread Jason Jackson
supervisor JVM does not load topology jar in it's classpath. We use 256mb heapsize, that ought to be fine. On Sun, Apr 6, 2014 at 11:11 PM, Srinath C srinat...@gmail.com wrote: Hi, I'm trying to figure out a reasonable amount of heap to grant to the supervisor process on the storm worker

Re: storm.yml conflict, which one to use?

2014-04-08 Thread Jason Jackson
The rule of thumb is to use the same storm.yaml file in both nimbus and supervisors, you won't have any problems with this. Some of the keys/vals are only read by supervisors e.g. (worker.childopts as that's the one that launches the worker JVM), and some are only read by nimbus

Re: JVM heap options for Supervisor

2014-04-08 Thread Jason Jackson
PM, Srinath C srinat...@gmail.com wrote: Thanks for the reply Jason. Supervisor doesn't need it in the classpath. But to provide the classpath to the worker processes it must be transferring the jar from nimbus. Hence the question. On Wed, Apr 9, 2014 at 6:32 AM, Jason Jackson jasonj

Re: Basic storm question

2014-04-06 Thread Jason Jackson
zookeeper doesn't retain a version history. On Fri, Apr 4, 2014 at 12:22 AM, M Tarkeshwar Rao m.tarkeshwar@ericsson.com wrote: Hi, Is it possible to get older version of the data of znode? Regards Tarkeshwar

Re: monitoring queuing delay and end-to-end latency?

2014-04-06 Thread Jason Jackson
end-to-end processing latency is the shown when acking is enabled and the complete latency for the spout. https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing On Thu, Apr 3, 2014 at 1:48 PM, Dong Mo monted...@gmail.com wrote: Dear list, Is there a way to monitor the

Re: Emitting/Acking in a Bolt

2014-04-06 Thread Jason Jackson
you only need to call ack once per input tuple, after all your emits are done. so at the end. On Mon, Mar 31, 2014 at 7:31 AM, Phil Burress philburress...@gmail.comwrote: Quick question aboutacking. If I had a bolt that had a loop; and inside that loop it emits a message to the stream for

Re: Replacing DRPC With Kafka

2014-04-03 Thread Jason Jackson
Never tried this. The DRPC Server is horizontally scalable, you can launch as many as you want, and spread your clients connections across all the servers. I think the idea with DRPC is that it's suppose to be very low latency between requestreply e.g. if it's used to power a website. Kafka will

Re: Profiling

2014-03-17 Thread Jason Jackson
have you tried yourkit profiler? On Mon, Mar 17, 2014 at 1:17 AM, Klausen Schaefersinho klaus.schaef...@gmail.com wrote: Hi, I am testing a typology that is frequently running out of memory. Has any body already made some experiences how to best profile storm? Kind Regards, Klaus