Re: Trident State and Static State

That would only work if it's an external state (e.g. state is kept in a database) as opposed to keeping state in memory within the topology. On Mon, Jun 16, 2014 at 6:38 PM, Romain Leroux wrote: > Can we define a StateFactory (sf) and do those 2 things with it ? (or > would that be a bad practi

Re: Trident State and Static State

Can we define a StateFactory (sf) and do those 2 things with it ? (or would that be a bad practice?) - update a state with something like partitionAggregate(sf) or partitionPersist(sf) - create a newStatic(sf) for querying this state (that we have updated) later in the topology On Jun 17, 2014

how to create maven project using eclipse rse

Hi, all I’ve been struggling to import the maven project: storm-starter into eclipse on mac. I installed a storm cluster on a remote ubuntu server, and I git cloned the storm starter into the ubuntu server already. I run the sample topologies locally on ubuntu server successfully. Now I want

Re: Extracting Performance Metrics

CodeHale doesn't handle extreme skew on measurements well last time I looked. For throughput, averages are great. For latency, you need very high percentiles to understand what is happening. On Mon, Jun 16, 2014 at 6:00 PM, Michael Rose wrote: > What kind of issues does Metrics have that lead

Re: Extracting Performance Metrics

If you can afford a bit more time for insertion, consider also t-digest. Differences relative to the high dynamic range histogram system include: - HDR histograms assume an exponential distribution. t-digest handles arbitrary distributions - t-digest is much more accurate near extreme values.

RE: Extracting Performance Metrics

What kind of issues does Metrics have that leads you to recommend HdrHistogram? On Jun 16, 2014 6:57 PM, "Dan" wrote: > Be careful when using Coda Hale's Metrics package when measuring latency. > Consider using Gil Tene's > High Dynamic Range Histogram instead: > > http://hdrhistogram.github.io/H

RE: Extracting Performance Metrics

Be careful when using Coda Hale's Metrics package when measuring latency. Consider using Gil Tene'sHigh Dynamic Range Histogram instead: http://hdrhistogram.github.io/HdrHistogram/ -Dan From: and...@parsely.com Date: Mon, 16 Jun 2014 18:20:11 -0400 Subject: Re: Extracting Performance Metrics To: u

Re: Extracting Performance Metrics

Also, I came across this presentation by Visible Measures which actually walks through a lot of great options covering most of what you want to know about: http://files.meetup.com/5809742/storm%20monitoring.pdf One other thing to be aware of is that in Storm 0.9.2 (forthcoming release), there is

Re: Extracting Performance Metrics

I haven't used it yet, but a lot of people get pointed to metrics_storm: https://github.com/ooyala/metrics_storm With this blog post that discusses it: http://engineering.ooyala.com/blog/open-sourcing-metrics-storm Michael Noll also has a nice blog post about streaming Storm 0.9 metrics to Grap

Extracting Performance Metrics

Dear all, I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1 zookeeper nodes. I am executing multiple topologies on the cluster and I want to extract different metrics that I am mentioning below. Can someone help me by recommending tools that I can use to extract this information.

Re: Custom metrics using IMetrics interface

IMetric and the metrics parts in storm are built on top of storm itself. If you need something that sends out metrics more regularly then Imetric does, just send them as regular tuples to a bolt you setup to collect them. - Bobby From: Xueming Li mailto:james.xueming...@gmail.com>> Reply-To:

Re: If I register a metrics object with a bolt/spout task, will it run in the same thread as the task?

Yes it will run in the same thread as the task. On 6/3/14, 10:14 PM, "Xueming Li" wrote: >Sorry, changed the title to make it more accurate. > > >On Tue, Jun 3, 2014 at 11:12 PM, Xueming Li >wrote: > >> Or do I need to worry about synchronization issue? Thanks in advance! >>

Re: Trident transactional topology stuck re-emitting batches with Netty, but running fine with ZMQ (was Re: Topology is stuck)

I have not seen this before, if you could file a JIRA on this that would be great. - Bobby From: Danijel Schiavuzzi mailto:dani...@schiavuzzi.com>> Reply-To: "user@storm.incubator.apache.org" mailto:user@storm.incubator.apache.org>> Date: Wednesday, June

Re: Trident State and Static State

Static state just refers to a state that is not maintained by your Trident topology but which you still want to be able to query, so something like a database that some other system is responsible for updating. On Mon, Jun 16, 2014 at 4:21 AM, Carlos Rodriguez wrote: > Hi guys, > > We are using

Re: HI,what is stormcode.ser?

Hi Jie, stormcode.ser contains a serialized json of uploaded topology. It contains all the components(spouts,bolts) ,component config , component parallelism. -Harsha On Mon, Jun 16, 2014, at 04:40 AM, jie liu wrote: thanks

Nimbus restart itself and detached from supervisord

Hello. We are using supervisord to manage storm/nimbus, over the time, we've found that there is another duplicated nimbus process which will be re-spawned and inherit from "init" (not supervisord) Any idea? Our supervisord.conf config: command=/opt/storm/bin/storm nimbus user=root autostart=t

Re: hot swap of topology

Hi Aaron, We do rolling deploys of our topologies by appending the build number to each topology. storm-topology-1 is active Jenkins submits storm-topology-2 Allow storm-topology-2 to become active & check health (or else halt) Deactivate storm-topology-1 & wait a few minutes (opportunity to halt

Re: Supervisor kills *all* workers for topology due to heartbeat :timed-out state

:timed-out means that the worker did not heartbeat to the supervisor in time. (This happens on local disk.) Check that your workers have enough jvm heap space. If not, garbage collection for the JVM will cause progressively slower heartbeats until the supervisor thinks they are dead and kills

hot swap of topology

Has anyone done any work with redeploying a topology with minimal downtime? I'm imagining a new storm command, or maybe a new function of the StormSubmitter class that: uploads the new code to the cluster, initializes bolts and spouts, turns off the old spouts, turns on the new spouts waits for ol

RuntimeException: unexpected tuple source

Hi all, Can someone tell me, what is the reason of this exception?? java.lang.RuntimeException: java.lang.RuntimeException: Received unexpected tuple source: $mastercoord-bg5:4, stream: $commit, id: {-325565218700865050=6198405391722663141}, [1:17] at backtype.storm.utils.DisruptorQueu

HI,what is stormcode.ser?

thanks

Trident State and Static State

Hi guys, We are using Riak as a State to write and read data from different riak buckets. (We are using this code https://github.com/hackreduce/storm-hackathon/blob/master/src/main/java/org/hackreduce/storm/example/riak/RiakBackingMap.java ) We first write to a bucket with partitionPersist, which

HI,why does the topology’s worker distribution not smooth

i hava 10 topology in my cluster.when i submit some topology later,the worker across the cluster in machine changed not smooth.there is printscreen in the accessory. i want some suggestion or train of thinking thanks a lot -

Re: Storm Logging

why don‘t try log collection system,as flume or mq 2014-06-16 15:02 GMT+08:00 Prabhath Suminda : > Hi, > > I am currently running 3 nodes storm cluster with 6 workers on each. > So there are 6 worker logs in each node. It is bit troublesome to > monitor logs in 3 different nodes. Is there any me

Storm Logging

Hi, I am currently running 3 nodes storm cluster with 6 workers on each. So there are 6 worker logs in each node. It is bit troublesome to monitor logs in 3 different nodes. Is there any mechanism to point all the logs to a single location. Regards, Prabhath