RE: Re: How will storm replay the tuple tree?

2016-09-14 Thread Hart, James W.
replays (especially in heavy throughput topologies). If you can elaborate a little more on the performance cost of tracking tuples or point to a document reflecting the same, that will be of great help. Best, T.I. On Tue, Sep 13, 2016 at 12:26 PM, Hart, James W. mailto:jwh...@seic.com>> w

RE: How will storm replay the tuple tree?

2016-09-13 Thread Hart, James W.
Failures should be very infrequent, if they are not then rethink the code and architecture. The performance cost of tracking tuples in the way that would be required to replay at the failure is large, basically that method would slow everything way down for very infrequent failures. From: S G

RE: Running a long task in bolt prepare() method

2016-08-25 Thread Hart, James W.
bolt can’t process events without deserializing and indexing the data in those files, which could take anything up to several minutes. This can’t easily be farmed out to an external service, due to various processing and infrastructure limitations SimonC From: Hart, James W. [mailto:jwh

RE: Running a long task in bolt prepare() method

2016-08-23 Thread Hart, James W.
Can you elaborate on what kind work is being done at startup? If you are building some kind of cacheable lookup data, I would build that elsewhere in a persistent cache, like redis, and then fetch and access it through redis. From: Simon Cooper [mailto:simon.coo...@featurespace.co.uk] Sent: Tue

General question about how many topologies can run in a smallish cluster.

2016-07-15 Thread Hart, James W.
I have a 5 VM cluster with 16 gig 8 core machines, and 3 of the machines are worker nodes. Can anyone give input on how many topologies should/can be run on the cluster? We are currently running 40 topologies in this dev cluster and having tons of stability and topology startup issues. These

RE: thread safe output collector

2016-04-29 Thread Hart, James W.
I’m working on a topology that will be similar to this application so I was thinking about this yesterday. I’m thinking that if there is any significant work to do on messages in making them into tuples, shouldn’t the message be emitted and the work be in a bolt? I don’t think that bolt execut