Re: Very high memory usage in one worker process

These are lot of logs and they are relatively large . Could you point me to the specific details to look for..? Vladi On Tue, Aug 5, 2014 at 8:28 AM, padma priya chitturi wrote: > Can you share with us the worker logs. > > --Padma Ch > > > On Tue, Aug 5, 2014 at 12:21 AM, Vladi Feigin wrote: >

Re: Very high memory usage in one worker process

Can you share with us the worker logs. --Padma Ch On Tue, Aug 5, 2014 at 12:21 AM, Vladi Feigin wrote: > Hi, > > Recently we observed very strange Storm behavior : > We had only one worker process, among dozens with very high memory usage > (a few gigs). We understand that this is not Java obj

Re: Good way to test when topology in local cluster is "fully active"

No worries. I haven't found the testing framework the easiest thing to use. If you see errors that say something like "cannot convert object to iterable" I use the code below to disable storm from being able to call System.exit() private static class ExitTrappedException extends SecurityExcep

Re: Good way to test when topology in local cluster is "fully active"

Nevermind, I wrote that before looking. This has been around since 0.8.1. Thanks again Vincent! On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet wrote: > Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much > into what's changed yet, other than Netty. > > > On Mon, Aug 4, 2014 at

Re: Good way to test when topology in local cluster is "fully active"

Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much into what's changed yet, other than Netty. On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell wrote: > Corey, > > Have you tried using the integration testing framework that comes with > storm? > > > Testing.withSimulatedTimeL

how can I make the Nimbus highly available ?

HI,all: how can I make the Nimbus highly available ? I am just confused about this. In the website, I just read below; "This means you can kill -9 Nimbus or the Supervisors and they’ll start back up like nothing happened. This design leads to Storm clusters being incredibly stable." Bu

Re: Good way to test when topology in local cluster is "fully active"

Corey, Have you tried using the integration testing framework that comes with storm? Testing.withSimulatedTimeLocalCluster(mkClusterParam, new TestJob() { @Override public void run(ILocalCluster cluster) throws Exception { CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam

Can you share your feeling about storm with mesos, if tried?

I'm considering of running storm on top of mesos, which is cluster management tool (apache opensource). and recently Nathan wrote storm-mesos for running storm on top of mesos. I want to ask one simple question if you have tried that. Is there any explicit advantage over running just storm

Good way to test when topology in local cluster is "fully active"

I'm testing some sliding window algorithms with tuples emitted from a mock spout based on a timer but the amount of time it takes the topology to fully start up and activate seems to vary from computer to computer. Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my tests are

Re: NoSuchMethorError

Thanks Derek. It turns out that testng 6.2 has repackaged snakeyaml and included it in the testng JAR file. After changing the order of some dependencies I was able to get my topology running locally in Eclipse. Thanks, Steve On 8/4/14 6:14 PM, "Derek Dagit" wrote: >I skimmed grepcode, and fo

Re: NoSuchMethorError

I skimmed grepcode, and found that Yaml(BaseConstructor) was available from snakeyaml version 1.7 onward. I would check if a version of snakeyaml <= 1.6 is in your classpath somehow. -- Derek On 8/4/14, 14:34, Ratay, Steve wrote: I am trying to run a local cluster using Storm 0.9.2, and getti

Re: Trouble reading resource files

This is not a Storm issue but a Java one, and you should definitively use 'getResourceAsStream' since you file is not a file: it's included in a JAR hence the ! in the URL. A solution is to extract this resource in a temporary file then use it. 2014-08-04 13:33 GMT+02:00 Yaron Rosenbaum : > Hi >

Re: Netty jar version in storm 0.9.2-incubating

Not required, looks like inclusion of 3.2.2 is a bug in the build script. I am running storm repackaged with netty 3.9, removed both netty 3.2.2 and 3.6.3, this let me use cassandra's driver that otherwise conflicted with storm. -- Andrey Yegorov On Mon, Aug 4, 2014 at 2:39 AM, Richards

storm with kafka and hadoop

Trying to know more about big data..I am trying to get my hands on to a simple/existing work that involves storm, kafka and hadoop. I came across  http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/ Along similar lines, can some one point me to a simple project/se

NoSuchMethorError

I am trying to run a local cluster using Storm 0.9.2, and getting a NoSuchMethodError. I am using Eclipse and have pulled all the Storm dependencies into my project. Most notably, I have the snakeyaml-1.11.jar file. Anyone else seeing this error or know where I've gone wrong? java.lang.NoSu

Very high memory usage in one worker process

Hi, Recently we observed very strange Storm behavior : We had only one worker process, among dozens with very high memory usage (a few gigs). We understand that this is not Java objects , since our heap size is only 1G. We don't use field grouping in this topology so this is not a hot spot. Does s

Storm workers not starting because of netty reconnect : [INFO] Reconnect started for Netty-Client

Hi, I have a topology that was deployed on a storm cluster and was running fine until I started facing the following issue. I can see that in supervisor logs, the supervisor is trying to launch the topology on a worker but it is not able to start it. 2014-08-04 18:27:33 b.s.d.supervisor [INFO

@ptgoetz storm-jms and ActiveMQ

Hi, Im currently using Storm 0.9.x, ActiveMQ and @ptgoetz storm-jms spout. I have multiple producers sending messages to the same queue. My topologies run multiple Worker/Executors/Taks. From what i understand: 1) One ActiveMQ consumer (storm-jms spout instance) takes messages from the queue pre

Re: Storm UI 0.9.2 bug (num workers displaying the num tasks and vice-versa)

Hi Spico, There is JIRA to track this [1][1]https://issues.apache.org/jira/browse/STORM-382. Thanks, Harsha On Mon, Aug 4, 2014, at 04:07 AM, Spico Florin wrote: Hello! In the Storm UI 0.9.2 I have observed that the num executors field is displaying the number of workers and the n

storm-0.9.2-incubating topology submission fails

Hi, I am trying to migrate to storm-0.9.2-incubating from storm-0.8.2. With storm-0.8.2 I submit topology as per the following post: https://groups.google.com/forum/?fromgroups#!searchin/storm-user/Can$20we$20submit$20the$20topology$20jar$20without$20using$20storm$20command$3F/storm-user/5JjZQyvi

Trouble reading resource files

Hi I'm trying to read resource files in one of my bolts. The resource files reside under '/profiles' within my JAR (verified). I'm using com.google.common.io.Resources like this: URL profilesUrl = Resources.getResource("profiles"); The returned URL (.toString) equals :jar:file:/Users/.../storm/t

Re: kafka-spout running error

hello, you can check your .jar application with command " jar tf " to see if class kafka/api/OffsetRequest.class is part of the jar. If not you can try to copy kafka-2.9.2-0.8.0.jar (or version you are using) in storm_lib directory Marcelo 2014-07-31 23:33 GMT+02:00 Sa Li : > Hi, all > > I am

Re: Simple and Elegant Way to Include Libraries / Jar Files

I would like to know how to deal with these libraries in general. Is it the fact that your project code must depend on the version and libraries Storm is adding to the classpath automatically? Do you always have to be aligned to the „old“ libs, storm is using by itself? There is and old discussi

Storm UI 0.9.2 bug (num workers displaying the num tasks and vice-versa)

Hello! In the Storm UI 0.9.2 I have observed that the num executors field is displaying the number of workers and the num workers is displaying the num executors. Is this a reported bug? Best regards, Florin

Re: Windowing queries on storm

Hi, You can either use trident topologies (see the manual) to process in batches or use storm's 'tick tuples' feature to aggregate using a defined time-window. Hope this helped. On Mon, Aug 4, 2014 at 1:37 PM, padma priya chitturi wrote: > Hi, > > I wanted to process streams in wondow of siz

Windowing queries on storm

Hi, I wanted to process streams in wondow of size say 'n' for every 't' sec. Does storm has inbuilt API's which support such scenarios of handling streams in a window ? Regards, Padma Ch

Re: Simple and Elegant Way to Include Libraries / Jar Files

> No dia 04/08/2014, às 10:17, Jafar Hafiz escreveu: > > Hi all, > > I am building a topology with a load of libraries, such as spring, geotools, > etc. Right now, I extract each jar files and pack it into one jar, (as > suggested by many forums), using maven. The problem here, each jar somet

Netty jar version in storm 0.9.2-incubating

Hi, I just download storm 0.9.2-incubating. I find two netty jar files in lib directory: netty-3.2.2.Final.jar netty-3.6.3.Final.jar Could you please confirm the right version? Are both these files required in the lib directory at the same time? Regards, Richards.

Simple and Elegant Way to Include Libraries / Jar Files

Hi all, I am building a topology with a load of libraries, such as spring, geotools, etc. Right now, I extract each jar files and pack it into one jar, (as suggested by many forums), using maven. The problem here, each jar sometime has conflicting files, so I have to merge manually. Another thing,