Understanding metrics.log

I added metrics to my storm implementation by implementing IMetric. It is working and can see the metrics log populated with all stats. I've a 3 node (3 worker) and 1 nimbus/zookeeper in Development. On WORKER1's Metrics Log I can see , some metrics with reference to WORKER2 and WORKER3. For exam

initializing storm-contrib submodules

storm-contrib masters: I still cannot pull down the contrib sub projects. After much research, I saw a post (https://github.com/nathanmarz/storm-contrib/issues/29) that seemed to advise non-commiters (like me) to convert the .gitmodules file into .gitmodules.ssh and .gitmodules.anon (and elimina

Metrics granularity needed?

Hi, Got a question about the level of performance metrics granularity people need/want for monitoring Storm. We just improved our Storm monitoring in SPM for Storm by aggregating and graphing metrics for each bolt and spout individually, so one can now select one or more

Re: PersistentAggregate across batches

the previous link didn't work, https://github.com/nathanmarz/storm/wiki/Trident-API-Overview#operations-on-grouped-streams On Tue, Apr 22, 2014 at 10:30 AM, Raphael Hsieh wrote: > Yes partially, > The part I was missing was getting old values and feeding it through the > aggregator again, which

Re: PersistentAggregate across batches

Yes partially, The part I was missing was getting old values and feeding it through the aggregator again, which still doesn't quite make sense to me. I am using an external datastore, so I am not able to use the vanilla MemcachedState, hence why I am implementing my own version of the IBackingMap.

Re: PersistentAggregate across batches

My understanding is that the process is 1. multiGet from the IBackingMap is called and returns a value for each key (or null if not present) 2. For each key, the old value from the get and new values in the batch are fed through the aggregator to produce one value per key 3. This value is then sto

Re: Topology submission exception caused by Class Not Found backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917

Well, it is good to hear that both of our sanity is intact; I'm not sure how I even thought to try changing around the way logging appeared on my classpath in the first place, but I'm glad I did. We have the same basic setup: trident topologies, custom logback XML on the classpath and an appender-

Re: Topology submission exception caused by Class Not Found backtype.storm.daemon.nimbus$normalize_conf$get_merged_conf_val__3916$fn__3917

I have seen this as well and thought I was going nuts. In my testing I could reliably reproduce it in local mode against 0.9.1-incubating. What I noticed in my testing: - It only occurred when a custom logback.xml was on the class path. - More specifically, it only happened when there was an “ap

Re: Where to put the topology and where to run it

You have to use maven for creating the jar of your topology. Then you can submit the topology using >>bin/storm jar path-to-jar namespace.namespace. your_main_file On Apr 22, 2014 8:43 PM, "Bilal Al Fartakh" wrote: > > thanks Andrew , so for a test can I use my supervisor machine as a storm > cl

Re: Can't start wordCountTopology in local mode

Assuming you are using the code from the GIT repo, including the storm-starter. The most likley reason IntelliJ can’t see it, is because storm-core is excluded in the POM.XML file from being included during build.It isn’t even finding the classes. Look in the POM.XML file, find the part

Re: PersistentAggregate across batches

The Reducer/Combiner Aggregators hold logic in order to aggregate across an entire batch, however it does not have the logic to aggregate between batches. In order for this to happen, it must read the previous TransactionId and value from the datastore, determine whether this incoming data is in th

Re: Where to put the topology and where to run it

thanks Andrew , so for a test can I use my supervisor machine as a storm client ? and for deploying , do I have to use maven ? 2014-04-22 13:45 GMT+00:00 Andrew Perepelytsya : > Typically you package your topology and dependencies into a jar, then > 'deploy' it with storm client (any machine ha

Re: Where to put the topology and where to run it

Typically you package your topology and dependencies into a jar, then 'deploy' it with storm client (any machine having storm client deps and having a config pointing to a nimbus server). Nimbus takes care of physically rolling out the topology after. Andrew On Tue, Apr 22, 2014 at 9:40 AM, Bilal

Where to put the topology and where to run it

I have two machines nimbus and supervisor where do I have to put my project (topology) and how to run it ? I'm sure that most of users here must know the answer . any comment that will be helpful is appreciable :) -- *Al Fartakh Bilal*

Re: Storm.yaml

finally it works and the supervisor is shown in the Storm UI : -I modified the /etc/hosts file in each server -and I stopped the iptables 2014-04-21 14:33 GMT+00:00 Bilal Al Fartakh : > logs dir (storm ) are empty in both machines , yes in my nimbus server I > have these > 14814 CassandraDaem