Re: Apache Storm Graduation to a TLP

2014-09-22 Thread Corey Nolet
Congrats! On Mon, Sep 22, 2014 at 5:16 PM, P. Taylor Goetz ptgo...@gmail.com wrote: I’m pleased to announce that Apache Storm has graduated to a Top-Level Project (TLP), and I’d like to thank everyone in the Storm community for your contributions and help in achieving this important

Re: Decouple topology configuration from code

2014-09-16 Thread Corey Nolet
Awhile ago I had written a camel adapter for storm so that spout inputs could come from camel. Not sure how useful it would be for you but its located here: https://github.com/calrissian/storm-recipes/blob/master/camel/src/main/java/org/calrissian/recipes/camel/spout/CamelConsumerSpout.java Hi

Re: Decouple topology configuration from code

2014-09-16 Thread Corey Nolet
Also, Trident is a DSL for rapidly producing useful analytics in Storm and I've been working on a DSL that makes streams processing for complex event processing possible. That one is located here: https://github.com/calrissian/flowmix On Sep 16, 2014 4:29 AM, dominique.vill...@orange.com wrote:

Re: Kafka + Storm

2014-08-14 Thread Corey Nolet
Kafka is also distributed in nature, which is not something easily achieved by queuing brokers like ActiveMQ or JMS (1.0) in general. Kafka allows data to be partitioned across many machines which can grow as necessary as your data grows. On Thu, Aug 14, 2014 at 11:20 PM, Justin Workman

Re: Good way to test when topology in local cluster is fully active

2014-08-05 Thread Corey Nolet
it handles IPv4/6. Try adding the following JVM parameter when running your tests: -Djava.net.preferIPv4Stack=true -Taylor On Aug 4, 2014, at 8:49 PM, Corey Nolet cjno...@gmail.com wrote: I'm testing some sliding window algorithms with tuples emitted from a mock spout based on a timer

Re: Good way to test when topology in local cluster is fully active

2014-08-05 Thread Corey Nolet
Sorry- the ipv4 fix worked. On Tue, Aug 5, 2014 at 9:13 PM, Corey Nolet cjno...@gmail.com wrote: This did work. Thanks! On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz ptgo...@gmail.com wrote: My guess is that the slowdown you are seeing is a result of the new version of ZooKeeper

Re: Good way to test when topology in local cluster is fully active

2014-08-05 Thread Corey Nolet
: -Djava.net.preferIPv4Stack=true -Taylor On Aug 4, 2014, at 8:49 PM, Corey Nolet cjno...@gmail.com wrote: I'm testing some sliding window algorithms with tuples emitted from a mock spout based on a timer but the amount of time it takes the topology to fully start up and activate seems to vary from

Re: Good way to test when topology in local cluster is fully active

2014-08-04 Thread Corey Nolet
); completeTopologyParam.setStormConf(daemonConf); completeTopologyParam.setTopologyName(getTopologyName()); Map result = Testing.completeTopology(cluster, topology, completeTopologyParam); }); -Vincent On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet cjno...@gmail.com wrote: I'm testing some sliding window algorithms

Re: Good way to test when topology in local cluster is fully active

2014-08-04 Thread Corey Nolet
Nevermind, I wrote that before looking. This has been around since 0.8.1. Thanks again Vincent! On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet cjno...@gmail.com wrote: Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much into what's changed yet, other than Netty. On Mon

Re: unsubscribe

2014-05-17 Thread Corey Nolet
Eric, if your intention is to unsubscribe from the mailing list, you should send a note to user-unsubscr...@storm.incubator.apache.org[1]. [1] http://mail-archives.apache.org/mod_mbox/incubator-storm-user/ On Fri, May 16, 2014 at 8:58 AM, eric perler ericper...@hotmail.com wrote: unsubscribe

Re: Interesting Comparison

2014-05-12 Thread Corey Nolet
Interesting that the paper was written by IBM people defending an IBM product. Not saying that it's biased or anything... Nathan, I agree that the windowing is better served as a layer on top. Personally, I appreciate that Storm deals with clustering, distributed state, fault-tolerance, and

Re: Flush aggregated data every X seconds

2014-04-24 Thread Corey Nolet
Raphael, in your case it sounds like a TickSpout could be useful where you emit a tuple every n time slices and then sleep until needing to emit another. I'm not sure how that'd work in a Trident aggregator, however. I'm not sure if this is something Nathan or the community would approve of, but

Timing / orchestration in Trident flows

2014-04-18 Thread Corey Nolet
I'm trying to do some timed/count-based orchestrations of streams in Storm/Trident. Some of my timing problems include timed or count-based emissions of tuples from aggregators and tumbling windows (whereby I'm batching up data and I emit every so often or I emit on every 500th tuple). I am