Re: How to let a topology know that it's time to stop?

*@Spico: *The code as promised: http://nrecursions.blogspot.in/2016/05/more-concepts-of-apache-storm-you-need.html#morecreativetopologystructures *@Matthias:* Still no luck. I tried this in the bolt code: Map conf = Utils.readStormConfig(); conf.put("nimbus.leader", "localhost"); Also tried

Re: Storm 1.0.0 upgrade Serialization issue

Filed: https://issues.apache.org/jira/browse/STORM-1773 KB, could you share sample topology which hits serialization issue? I would like to check whether patch helps resolving it or not. 2016년 5월 10일 (화) 오후 12:14, Jungtaek Lim 님이 작성: > Samuel and KB, > > I think Storm 1.x

Re: Storm 1.0.0 upgrade Serialization issue

Samuel and KB, I think Storm 1.x hits the bug on commons-io (IO-368 ). I'll file an issue. Thanks for reporting. Best Regards, Jungtaek Lim (HeartSaVioR) 2016년 5월 10일 (화) 오전 1:33, KB 님이 작성: > All, > > This problem persists

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

I guess both application performance and ZK are all possible to have performance issues since stats are recorded from critical path, and it incurs heartbeat message getting bigger which means more ZK write load. I thought about asynchronous metrics recording, but it should enqueue record task to

Re: Spout Thread Waiting

Hi Adrien Did you set a value to max spout pending? Could it be that you have reached the max number of tuples in process? Do you see acks or fails happen during that period? Great to hear that you are using StormCrawler BTW Julien On 9 May 2016 at 20:48, Adrien Carreira

Re: Spout Thread Waiting

I think the problem is when My topology is working the thread calling nextTuple seems to be busy... Why the method isn't called ? Someone can guid me to some documentation or the code calling nextTuple just to understand what is blocking Thank you guys 2016-05-09 9:57 GMT+02:00 Adrien

Re: How Does Nimbus Decide to Restart Topology?

Looking at the storm worker thread dumps, it looks like the bolt/spout components are all waiting for the topology to become active (including the __system and __acker components): "Thread-37-__acker" #70 prio=5 os_prio=0 tid=0x7f6feca57000 nid=0x3a8d waiting on condition [0x7f6f54ef2000]

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Adam, Performance issue raised by Lim in #2 is not about the application performance, but the zookeeper where these metrics are being written to. Zookeeper doesn't handle heavy frequent writes well. This problem will become more apparent in larger clusters. On Mon, May 9, 2016 at 7:22 PM, Adam

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Jungtaek, thanks for the followup response. For #1, having this in the Storm UI would be very nice and I think of general interest to anyone who is tasked with maintaining Storm deployments and certainly during development for capacity and stress testing. I'm not sure what it takes to get it

Re: How to let a topology know that it's time to stop?

Utils.readStormConfig() tries to read "./storm.yaml" from local disc (ie, supervisor machine that executes the bolt) -- as it is using "working-directory" a guess it does not find the file, and thus value "nimbus.host" is not set. Make sure that storm.yaml is found be the worker, or set

Re: How to let a topology know that it's time to stop?

@Spico: Will share. The streams implementation is working beautifully. Only the topology killing is failing. *Tried:* Map conf = Utils.readStormConfig(); NimbusClient cc = NimbusClient.getConfiguredClient(conf); Nimbus.Client client = cc.getClient(); client.killTopology("myStorm"); *I get these

Re: How to let a topology know that it's time to stop?

Hi! You welcome Navine. I'm also interested in the solution. Can you please share your remarks and (some code :)) after the implementation? Thanks. Regards,\ Florin On Mon, May 9, 2016 at 7:20 AM, Navin Ipe wrote: > @Matthias: That's genius! I didn't know

Custom scheduler For Storm 1.0.0

Hello all, I have site aware scheduler that used to work with the previous versions of storm (0.9.6).But with storm 1.0.0 all scheduling is done but messages are not passed from spout to downstream nodes . Are there any major changes except the package hierarchy in terms of implementing

Apache Storm, n bolts store data n pipelines

Hi all, I have an Apache Storm topology that have 1 spout (data from kafka) and 10 bolts that compute data and then try to storage. My problem is that a have 10 pipelines that try to store data from Storm to a single instance of Mongodb. What's the best solution for this problem? I have

Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Some more ideas from me, A) auxiliary optional metrics for Spout Some implementations of Spout could give more information. One of the things is the lag between a data source and Spout offset which is direct indication that whether Spout (or Topology if it's throttled by Spout) is catching up

Spout Thread Waiting

Hi there, I'm using Storm to build a web-crawler, using Storm Crawler SDK. I'm also using Redis to store new links discovered. I've a Spout to consume those url. After many debug , I've built the Spout like this : public class OutlinkSpoutRedis extends BaseRichSpout { private static final