Re: Decouple topology configuration from code

2014-09-17 Thread Andrew Montalenti
Storm bundles a comprehensive Clojure DSL for topology definition. In the streamparse project, we extended this DSL to make Python bolts and spouts easier to define, and the bundled streamparse CLI tool can submit Clojure DSL topologies to remote or local clusters. Some docs on this are here.

Re: Storm 0.9.2 Performance on EC2 Larges

2014-07-11 Thread Andrew Montalenti
Something is likely wrong with the topology's layout and I'm sure it's fixable. HTH, --- Andrew Montalenti Co-Founder CTO http://parse.ly On Fri, Jul 11, 2014 at 6:38 PM, Gary Malouf malouf.g...@gmail.com wrote: Hi everyone, We've been banging our heads against the wall trying to get

Re: b.s.m.n.Client [INFO] Reconnect

2014-07-10 Thread Andrew Montalenti
fyi, you won't be able to change to zeromq in 0.9.2 from Netty unless you use this updated storm-0mq module that I forked: https://github.com/Parsely/storm-0mq/blob/master/README.md I also added a detailed README with instructions. We'll get these changes merged into ptgoetz's repo eventually so

Re: v0.9.2-incubating and .ser files

2014-06-20 Thread Andrew Montalenti
you're seeing. As Nathan mentioned, any additional information (thread dumps, etc.) you could provide would help. Thanks (and sorry for the inconvenience), Taylor On Jun 19, 2014, at 6:09 PM, Andrew Montalenti and...@parsely.com wrote: Another interesting 0.9.2 issue I came across

Re: v0.9.2-incubating and .ser files

2014-06-19 Thread Andrew Montalenti
mentioned, any additional information (thread dumps, etc.) you could provide would help. Thanks (and sorry for the inconvenience), Taylor On Jun 19, 2014, at 6:09 PM, Andrew Montalenti and...@parsely.com wrote: Another interesting 0.9.2 issue I came across: the IConnection interface has changed

v0.9.2-incubating and .ser files

2014-06-18 Thread Andrew Montalenti
I built the v0.9.2-incubating rc-3 locally and once verifying that it worked for our topology, pushed it into our cluster. So far, so good. One thing for the community to be aware of. If you try to upgrade an existing v0.9.1-incubating or 0.8 cluster to v0.9.2-incubating, you may hit exceptions

Re: Extracting Performance Metrics

2014-06-16 Thread Andrew Montalenti
I haven't used it yet, but a lot of people get pointed to metrics_storm: https://github.com/ooyala/metrics_storm With this blog post that discusses it: http://engineering.ooyala.com/blog/open-sourcing-metrics-storm Michael Noll also has a nice blog post about streaming Storm 0.9 metrics to

Supervisor kills *all* workers for topology due to heartbeat :timed-out state

2014-06-14 Thread Andrew Montalenti
I am trying to understand why for a topology I am trying to run on 0.9.1-incubating, the supervisor on the machine is killing *all* of the topology's Storm workers periodically. Whether I use topology.workers=1,2,4, or 8, I always get logs like this:

Re: [VOTE] Storm Logo Contest - Final Round

2014-06-13 Thread Andrew Montalenti
#9 - 5 pts. On Mon, Jun 9, 2014 at 2:38 PM, P. Taylor Goetz ptgo...@gmail.com wrote: This is a call to vote on selecting the winning Storm logo from the 3 finalists. The three candidates are: * [No. 6 - Alec Bartos]( http://storm.incubator.apache.org/2014/04/23/logo-abartos.html) *

Re: MultiLag (Python) bolt gives error

2014-06-05 Thread Andrew Montalenti
On the worker machine, do you have Python installed? You can check by running python -V. You need to ensure you're using the same $PATH as whatever environment is running your Storm supervisor/worker. From the exception stack trace, it looks like your ShellBolt does not see a python interpreter on

Re: Multilang shell bolts exceptions

2014-06-05 Thread Andrew Montalenti
What multi-lang driver for PHP are you using? There is a way for multi-lang drivers to report errors up to Storm (see ShellBolt's handling of the error command here https://github.com/apache/incubator-storm/blob/master/storm-core/src/jvm/backtype/storm/task/ShellBolt.java#L114-L115). Perhaps your

Re: python via java+scriptengine+jython

2014-06-05 Thread Andrew Montalenti
Don't see why this wouldn't be possible. I assume a ScriptEngine implementation would be using jython, so the usual caveats about jython apply (e.g. you don't have access to all 3rd-party libraries and you tend to be behind in Python versions). What kind of comparison are you looking for, a

Re: MultiLag (Python) bolt gives error

2014-06-05 Thread Andrew Montalenti
but its not there. Why? How n where should i give its path ? Assigning which variable? *** This message has been sent using QMobile A500 *** Andrew Montalenti and...@parsely.com wrote: On the worker machine, do you have Python installed? You can check by running python -V. You need to ensure

Re: Writing Bolts in Python

2014-06-02 Thread Andrew Montalenti
development. --- Andrew Montalenti Co-Founder CTO http://parse.ly On Mon, Jun 2, 2014 at 12:37 PM, Ashu Goel a...@shopkick.com wrote: Hi all, I am experimenting with writing bolts in Python and was wondering how the relationship between the Java and Python code works. For example, I have

Re: Storm with Python

2014-05-30 Thread Andrew Montalenti
... not entirely sure how this is different from what storm already provides. I was looking more for a DSL that we could use w/ Python 2.6 and be 100% Python, but it looks like that is not available. -Ashu On May 30, 2014, at 2:14 PM, Andrew Montalenti and...@parsely.com wrote: For one thing

Re: Storm with Python

2014-05-29 Thread Andrew Montalenti
and if you join our mailing list, feel free to make requests. We continue to develop it actively and in an open manner. -Andrew Montalenti CTO, Parse.ly On May 29, 2014 6:35 PM, Ashu Goel a...@shopkick.com wrote: (the reason being is that we are still running Python 2.6 but Petrel is only