Re: [DISCUSS] Would like to make collective intelligence about Metrics on Storm

Hi Adam, Thanks for the great input! Let me share my thought about two things. 1. There's metrics for disruptor queue so if you attach metrics consumer it will be provided to consumer. Sojourn time for queue is also provided (kudos to Li Wang) but it's based on queueing theory and has one

Defunct workers still hold ports

hi, There're some defunct workers in my storm cluster(version:0.9.5): deploy1634 1 0 2015 ?07:11:45 [java] deploy5607 1 2 Mar25 ?23:59:26 [java] deploy9154 1 2 Jan13 ?3-05:31:28 [java] deploy 14292 1 4 Mar11 ?2-20:59:31 [java]

Re: Fail then ack

The tuple still fail. The fail will remove the item from the bolt's low level cache, the next ack action does nothing. RegardsLongda --From:Matthias J. Sax Send Time:2016年5月8日(星期日) 00:06To:user

Re: How to let a topology know that it's time to stop?

Alternative is to use a control message on a separate stream that goes to all bolt tasks using all grouping. On May 8, 2016 3:20 PM, "Matthias J. Sax" wrote: > To synchronize this, use an additional "shut down bolt" that used > parallelism of one. "shut down bolt" must be

Re: How to let a topology know that it's time to stop?

To synchronize this, use an additional "shut down bolt" that used parallelism of one. "shut down bolt" must be notified by all parallel DbBolts after they performed the flush. If all notifications are received, there are not in-flight message and thus "shut down bolt" can kill the topology safely.

Re: Pull data from Redis every minute

hi! for me it looks like proceesing the data in a specific window. you coul achive this by using the new feature in storm 1.0 namely window bolt. via spout you get the data thta you need and in the window bolt do the sum. be careful with the time thta you are using processing time versus event

Re: How to let a topology know that it's time to stop?

hi! there is this solution of sending a poison pill message from the spout. on bolt wil receiv your poison pill and will kill topology via storm storm nimbus API. one potentential issue whith this approach is that due to your topology structure regarding the parralelism of your bolts nd the time

Re: How do you run a task in a separate thread in a Bolt?

Oh and finally, another route you can take is to simply bump up the concurrency using Storm's setting (assuming that you need more concurrency) and just tolerate blocking Storm's thread while doing IO. While it may not be the most elegant or efficient solution it's easy and will probably work in

Re: How do you run a task in a separate thread in a Bolt?

Just remembered; I'm not 100% sure if it's still the case but the output collector (where you emit the tuples generated) is not thread safe, so you must synchronize on it when you use your own threads. I.e. like this: synchronized(collector){ collector.emit(newTuple) } On Sun, May 8, 2016

Re: How do you run a task in a separate thread in a Bolt?

> Yes, I considered a threadpool, but the confusion was about where to declare them, initiate a thread run and when to join the threads. Any code samples or pseudocode that could help? You would mark the thread pool transient and allocate/shutdown using the lifecycle callback methods provided by

Re: How to let a topology know that it's time to stop?

You can get the number of bolt instances from TopologyContext that is provided in Bolt.prepare() Furthermore, you could put a loop into your topology, ie, a bolt reads it's own output; if you broadcast (ie, allGrouping) this feedback-loop-stream you can let bolt instances talk to each other.

Re: How do you run a task in a separate thread in a Bolt?

Yes, I considered a threadpool, but the confusion was about where to declare them, initiate a thread run and when to join the threads. Any code samples or pseudocode that could help? Besides, there's this thread where a person advises

Re: How do you run a task in a separate thread in a Bolt?

There's nothing that keeps you from simply having a thread pool in your bolts. Or you could go for an async DB client. You will have to be careful about providing back pressure (e.g. by using a bounded queue). On Sun, May 8, 2016 at 12:12 PM, Navin Ipe wrote: >

How do you run a task in a separate thread in a Bolt?

Hi, I've wanted to do this and this post confirms the idea: http://stackoverflow.com/a/36106683/453673 But when I have a spout that constantly has nextTuple() being called by Storm and I have a bolt that constantly has execute() being called whenever it receives a tuple, how do I program the