Re: JVM heap options for Supervisor

The supervisor calls downloadChunk on nimbus API https://github.com/apache/incubator-storm/blob/master/storm-core/src/storm.thrifteach chunk is persisted to local disk on the supervisor before the next chunk is grabbed. So the whole jar does need to be stored in RAM. On Tue, Apr 8, 2014 at 6:14 P

Re: JVM heap options for Supervisor

Thanks for the reply Jason. Supervisor doesn't need it in the classpath. But to provide the classpath to the worker processes it must be transferring the jar from nimbus. Hence the question. On Wed, Apr 9, 2014 at 6:32 AM, Jason Jackson wrote: > supervisor JVM does not load topology jar in it's

Re: storm.yml conflict, which one to use?

The rule of thumb is to use the same storm.yaml file in both nimbus and supervisors, you won't have any problems with this. Some of the keys/vals are only read by supervisors e.g. (worker.childopts as that's the one that launches the worker JVM), and some are only read by nimbus (nimbus.thrift.por

Re: JVM heap options for Supervisor

supervisor JVM does not load topology jar in it's classpath. We use 256mb heapsize, that ought to be fine. On Sun, Apr 6, 2014 at 11:11 PM, Srinath C wrote: > Hi, > I'm trying to figure out a reasonable amount of heap to grant to the > supervisor process on the storm worker machines. What a

Re: Topology is stuck

An example of "corrupted input" that causes a batch to fail would be for example if you expected a schema to your data that you read off kafka, or some queue, and for whatever reason the data didn't conform to your schema and the function that you implement that you pass to stream.each throws an ex

RE: Can Storm write an Aggregate Record to Postgres or SQL Server?

Many thanks! From: cody.a@gmail.com Date: Tue, 8 Apr 2014 15:05:48 -0500 Subject: Re: Can Storm write an Aggregate Record to Postgres or SQL Server? To: user@storm.incubator.apache.org Can you elaborate on how you want to "aggregate" data? If each log entry is essentially a timestamp, a tran

Re: Can Storm write an Aggregate Record to Postgres or SQL Server?

Can you elaborate on how you want to "aggregate" data? If each log entry is essentially a timestamp, a transaction type (since you mentioned this), and a numerical value (which you cant to sum during the 5-minute window), then you don't need tick tuples. The way we do aggregation is by mapping a t

RE: Can Storm write an Aggregate Record to Postgres or SQL Server?

Neil, Take a look at using "tick tuples" http://nathanmarz.github.io/storm/doc/backtype/storm/Config.html#TOPOLOGY_TICK_TUPLE_FREQ_SECS and the Storm RDBMS bolt https://github.com/nathanmarz/storm-contrib/tree/master/storm-rdbms -Roger From: Neil Carroll [mailto:carroll_n...@hotmail.com] Sent:

Can Storm write an Aggregate Record to Postgres or SQL Server?

I'm new to Storm and want to use it to aggregate log data over a 5 minute period and write aggregate records (for each transaction type) into a DCMS (SQL or Postgres). I believe Storm can do this - and is there sample code available? Thanks Neil

Re: storm.yml conflict, which one to use?

In that sense then fair enough, as you are only running UI in one location you probably need only configure that parameter on that node. You mentioned worker.childopts, I assume you have a worker on the same node as nimbus then? Even so I would expect this value to be consistent across all node

Re: storm.yml conflict, which one to use?

Hi Because it does not make sense to duplicate, for example, ui.childopts in supervisor'storm.yaml right? So I want to know how the storm.yaml is being used, so I can remove these duplicates if they are really dummy and not used. On Tue, Apr 8, 2014 at 9:38 PM, David Crossland wrote: > To my (r

Re: storm.yml conflict, which one to use?

To my (rather limited) knowledge storm.yaml should be consistent across all locations. It can be overridden in your topology jar. D From: howard chen Sent: ?Tuesday?, ?8? ?April? ?2014 ?14?:?28 To: user@storm.incubator.apache.org

storm.yml conflict, which one to use?

Hi, I deployed storm into different machines a. nimbus b. ui c. supervisor Let say I want to set the "worker.childopts", if my value of this setting in the nimbus' storm.yml and supervisor storm.yml are different, which value will storm cluster use? I ask this because I have seen that in the st

Re: ACK performance hit & Loggly abandoning Storm

There were a few points in the blog which either were interesting or i totally missed the point. - It was mentioned that they were using without ACK turned on. So in that case how would it be ensured that all the tuples were successfully processed by all the bolts. - ACKing along with