Re: postgresql -> spout

I know that this is very late to jump in but if you are integrating into a system that is already using postgres as a message broker/queue, why not poll your postgres queue directly from nexttuple instead of pushing everything onto another queue? Just use jdbc to connect to postgres from nexttuple

Re: Is there any way for my application code to get notified after it gets deserialized on a worker node and before spouts/bolts are opened/prepared ?

The bolt base classes have a prepare method: https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseBasicBolt.html and the spout base classes have a similar activate method: https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseRichSpout.html Is that s

Re: Interesting Comparison

To play devil's advocate, if you believe the stream performance gains, then the 40k will likely pay for itself in needing to deploy a fraction of the resources for the same throughput. On Mon, May 12, 2014 at 09:02:53AM -0400, John Welcher wrote: > Hi > > Streams also cost 40,000 US while Storm

Re: Doubts on Apache Storm

On Tue, May 06, 2014 at 03:21:13PM +0530, milind.pa...@polarisft.com wrote: > > Hi, > >Is Nimbus mandatory for storm? (Our development env is neither using >Nimbus nor any other cloud environment) I think you might have misunderstood nimbus. It is a daemon that is part of storm, *not* n

Re: PDF processing use case in storm!!

sers may be using the service at the same time.It may be for > different file or the same file. > > Thanks > Deepak > > > On Mon, Apr 28, 2014 at 7:41 PM, Marc Vaillant > wrote: > > I think it's important to know whether or not some form of paralle

Re: PDF processing use case in storm!!

I think it's important to know whether or not some form of parallelism (other than throughput) is required, otherwise a standard webservice seems sufficient for this use case. On Mon, Apr 28, 2014 at 07:46:35AM -0400, Andrew Perepelytsya wrote: > You can build request response type topologies via

Re: Setting up Storm behind HTTP in Clojure

Have you looked at Trident + DRPC? https://github.com/nathanmarz/storm/wiki/Trident-tutorial Also, I came across the following once but I've never tried it and I'm not sure how mature it is: https://github.com/chriskchew/restexpress-storm Marc On Fri, Apr 18, 2014 at 04:40:06PM +0200, Joël Ku

Re: Can be storm-deploy script used for deploying storm 0.9.x version on Amazon?

Hi Florin, I just wanted to suggest that you also look at Wirbelsturm by Michael Noll as an alternative to storm-deploy https://github.com/miguno/wirbelsturm. I think that you will find it more complete, better documented, and more mainstream because it uses vagrant and puppet instead of pallet

One solution to the stdio redirect issue

I put together a more complete solution to the insidious STDOUT/STDERR buffer filling issue. Basically, if STDOUT/STDERR is not redirected/consumed in cluster mode it will fill the buffer and eventually take down your topology. The original thread on this issue was not migrated to JIRA but relate

Re: Wirbelsturm released, 1-click deployments of Storm clusters

only one of the two is not enough. See the link above for what > breaks in each case if you still try. > > The problem is fixed in puppet-zookeeper 1.0.4 and in the latest > (master/trunk) version of Wirbelsturm. > > Hope this helps! > Michael > > > > On 03/19/201

Re: Wirbelsturm released, 1-click deployments of Storm clusters

Hi Michael, Thanks very much for your hard work on this, your puppet scripts have been very helpful. We are having a specific issue with supervision of zookeeper and I wonder if you have encountered something similar or if we are doing something wrong. Even with the stopasgroup=true supervisord

which heartbeat(s) to modify so that debug sessions don't timeout?

I'm trying to debug some native code that runs in a task using gdb. When I attach to the process, storm holds me to one or more of its 30s heartbeat timeouts while stepping through code, at which point it kills the process and therefore prematurely ends my debugging session. I'm having trouble fig

Re: Storm 0.9.0.1 - topology halts processing after a while

lution. For those who > have been bitten by this it’s painful — zero indication of what the issue is. I did find the following for the JNI case which I have not tried yet: http://tabbott.com/2010/09/capturing-native-code-output/ Marc > > At the very least, it should be well documen

Are the old github issues archived somewhere?

Including closed issues? E.g. I'm specifically looking for the following issue, which is gone: https://github.com/nathanmarz/storm/issues/31#issuecomment-17607249 Thanks, Marc

Re: Storm 0.9.0.1 - topology halts processing after a while

I was going to mention something similar. The stdout/stderr issue is insidious. Only happens in cluster mode and while it was fixed for some stdout/stderr pathways, others have not been fixed AFAIK. Taylor has mentioned one, another is stdout/stderr coming from native code via JNI (i.e. printf,

Re: Can a topology be configured to force a maximum of 1 executor per worker?

Marc, > > I believe keeping the total number of executors(i.e. parallelism) across > all the components(bolts,spouts) to be less than or equal to the total > number of workers can be one way to achieve this. > > Thanks > Bijoy > > >

Re: Can a topology be configured to force a maximum of 1 executor per worker?

can be one way to achieve this. > > Thanks > Bijoy > > > On Wed, Feb 5, 2014 at 9:36 PM, Marc Vaillant wrote: > > Suppose that you have a bolt whose tasks are not thread safe but you > still want parallelism. It seems that this could be achieved via >

Can a topology be configured to force a maximum of 1 executor per worker?

Suppose that you have a bolt whose tasks are not thread safe but you still want parallelism. It seems that this could be achieved via multiprocessing by forcing a maximium of 1 executor per worker. With this constraint, if you chose a parallelism hint of 4 (with default executors) you would get 4