Hi Bobby, I seem to get everything I need working except this part. It seems like I'm making a simple mistake. I checked the JIRA post you gave and tried the following.
1. When I tried, c:\python>python c:\storm\bin\storm list, I got this error ******************************************** *The storm client can only be run from within a release. You appear to be trying to run the client from a checkout of Storm's source code.* *You can download a Storm release at http://storm-project.net/downloads.html <http://storm-project.net/downloads.html>* ******************************************** *c:\python>* 2. When I tried running from inside storm or storm-starter directory, I get an error too c:\python>python c:\storm\storm list *python: can't open file 'c:\storm\storm': [Errno 2] No such file or directory* c:\python>python c:\storm\examples\storm-starter\storm list *python: can't open file 'c:\storm\examples\storm-starter\storm': [Errno 2] No such file or directory* The comments in the post seems to suggest the same methods that I tried 1. "In Windows, users should run command <PYTHON_PATH>\python bin\storm blabla...." 2. "I use storm-starter example. ``` C:\apache-storm-0.9.2-incubating\examples\storm-starter>python ..\..\bin\storm j ar storm-starter-topologies-0.9.2-incubating.jar storm.starter.WordCountTopology production-topology remote" Where and how should I run the storm command? FYI: I also ran the command below whoch does not give errors but also has no effect (atleast in the UI console). The number of workers and executors remain the same. c:\python>python c:\storm\examples\storm-starter\multilang\resources\storm.py rebalance word-count-java -w 5 -n 6 -e TestSpout1=4 -e Count=2 Thanks, Harsha *Contact Details: * Harsha Balasubramanian Graduate Student at University of Toronto Masters in Computer Science Mobile : 647-771-3596 Email 1: harsha...@cs.toronto.edu Email 2: harsha...@gmail.com LinkedIn : ca.linkedin.com/in/harshabala/ On Wed, Dec 24, 2014 at 4:31 PM, Harsha Balasubramanian <harsha...@gmail.com > wrote: > Thanks for the prompt reply. Will check that. > > Thanks, > Harsha > > *Contact Details: * > Harsha Balasubramanian > Graduate Student at University of Toronto > Masters in Computer Science > > Mobile : 647-771-3596 > Email 1: harsha...@cs.toronto.edu > Email 2: harsha...@gmail.com > LinkedIn : ca.linkedin.com/in/harshabala/ > > On Wed, Dec 24, 2014 at 4:13 PM, Harsha <st...@harsha.io> wrote: > >> looks like there is a bug storm.cmd in make_command_arguments method. >> Its converting -e count=10 to -e count 10 (removing =) which is causing >> this issue. >> If you using trunk you can look STORM-487 which makes python storm >> command work in windows. >> >> >> On Wed, Dec 24, 2014, at 12:09 PM, Harsha Balasubramanian wrote: >> > Thanks Bobby. >> > >> > I setup a local server and CLI, UI and my topology run fine. However >> > there >> > is a feature that doesn't work and I need some help to figure out why. >> > >> > 1. I followed the instructions here to setup the storm system, >> > >> http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/ >> > >> > 2. My feature requires the usage of the CLI tool to re-balance the >> > topology >> > when it needs more resources. For that I thought I could use the >> > following >> > CLI command, >> > >> > storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10 >> > >> > I found this in the storm documentation at the bottom of this page: >> > >> http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html >> > >> > 3. When I run this command in my setup, I get the error pasted below. Is >> > re-balancing supported at the spouts and bolts level? Am I using the >> > wrong >> > version of the storm jar. >> > >> > This works: >> > *C:\storm>storm rebalance WordCount -n 15* >> > *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at >> > localhost:6627* >> > *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount >> > is >> > rebalancing* >> > >> > This does not work: >> > *C:\storm>storm rebalance WordCount -n 5 -e Count=10* >> > *Exception in thread "main" java.lang.StringIndexOutOfBoundsException: >> > String index out of range: -1* >> > * at java.lang.String.substring(Unknown Source)* >> > * at >> > >> backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)* >> > * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)* >> > * at clojure.tools.cli$cli.doInvoke(cli.clj:130)* >> > * at clojure.lang.RestFn.invoke(RestFn.java:460)* >> > * at >> > backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)* >> > * at clojure.lang.RestFn.applyTo(RestFn.java:137)* >> > * at backtype.storm.command.rebalance.main(Unknown Source)* >> > >> > >> > >> > Thanks, >> > Harsha >> > >> > *Contact Details: * >> > Harsha Balasubramanian >> > Graduate Student at University of Toronto >> > Masters in Computer Science >> > >> > Mobile : 647-771-3596 >> > Email 1: harsha...@cs.toronto.edu >> > Email 2: harsha...@gmail.com >> > LinkedIn : ca.linkedin.com/in/harshabala/ >> > >> > On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans >> > <ev...@yahoo-inc.com.invalid> >> > wrote: >> > >> > > Harsha, >> > > Sorry about that, I have been out at a conference. >> > > >> > > for 1 you can get the number of acks so far, but you are going to >> have to >> > > calculate the rate yourself, it keeps track of the number of acks over >> > > several time windows, but reports them as whole numbers. >> > > What do you mean by a local cluster? Do you mean a single node >> cluster >> > > you launched yourself? then yes. If you mean a local-mode cluster, >> then no. >> > > - Bobby >> > > >> > > >> > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian < >> > > harsha...@gmail.com> wrote: >> > > >> > > >> > > Hi, >> > > >> > > Can someone pls answer my questions from the above thread. >> > > >> > > >> > > *I have 2 questions for you;* >> > > *1. In one of your earlier emails, you mentioned that there is >> > > a limited Metrics implementation which is available in the current >> Storm >> > > version. * >> > > >> > > *Will this provide 'acks threshold', i.e, the number of acks per >> second (or >> > > minute)? If not, can you please suggest a way in which I can >> implement this >> > > myself. * >> > > >> > > *2. Is it possible to use the CLI and UI tools from a local cluster?* >> > > >> > > Thanks, >> > > Harsha >> > > >> > > *Contact Details: * >> > > Harsha Balasubramanian >> > > Graduate Student at University of Toronto >> > > Masters in Computer Science >> > > >> > > Mobile : 647-771-3596 >> > > Email 1: harsha...@cs.toronto.edu >> > > Email 2: harsha...@gmail.com >> > > LinkedIn : ca.linkedin.com/in/harshabala/ >> > > >> > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian < >> > > harsha...@gmail.com> wrote: >> > > > >> > > > Hi Bobby, >> > > > >> > > > Hope this email finds you well. I.m about to start designing the >> > > > auto-scaling system. Created a JIRA for it, >> > > > https://issues.apache.org/jira/browse/STORM-594 >> > > > >> > > > Please take a look and let me know if there are any concerns. >> > > > >> > > > I have 2 questions for you; >> > > > 1. In one of your earlier emails, you mentioned that there is a >> *limited >> > > *Metrics >> > > > implementation which is available in the current Storm version. >> > > > Will this provide 'acks threshold'? >> > > > >> > > > 2. Is it possible to use the CLI tool from a local cluster? >> > > > >> > > > Thanks, >> > > > Harsha >> > > > >> > > > *Contact Details: * >> > > > Harsha Balasubramanian >> > > > Graduate Student at University of Toronto >> > > > Masters in Computer Science >> > > > >> > > > Mobile : 647-771-3596 >> > > > Email 1: harsha...@cs.toronto.edu >> > > > Email 2: harsha...@gmail.com >> > > > LinkedIn : ca.linkedin.com/in/harshabala/ >> > > > >> > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian < >> > > > harsha...@gmail.com> wrote: >> > > >> >> > > >> Thanks for the detailed explanation Bobby. I will keep this in >> mind when >> > > >> I create my design document. >> > > >> >> > > >> I will mostly not deal with adding/removing bolts to the topology >> and >> > > >> restrict myself to threads and tasks. This is because of the time I >> > > have to >> > > >> complete the project. Once I submit my report (early January), I >> can >> > > extend >> > > >> what I have implemented to more use cases. >> > > >> >> > > >> Thanks, >> > > >> Harsha >> > > >> >> > > >> Thanks, >> > > >> Harsha >> > > >> >> > > >> *Contact Details: * >> > > >> Harsha Balasubramanian >> > > >> Graduate Student at University of Toronto >> > > >> Professional Masters in Computer Science >> > > >> >> > > >> Email 1: harsha...@cs.toronto.edu >> > > >> Email 2: harsha...@gmail.com >> > > >> LinkedIn : ca.linkedin.com/in/harshabala/ >> > > >> >> > > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans >> > > <ev...@yahoo-inc.com.invalid >> > > >> > wrote: >> > > >> >> > > >>> Sure, >> > > >>> The main thing that storm is missing right now is an metrics >> feedback >> > > >>> system to the scheduler (or possibly a separate piece of code that >> > > decides >> > > >>> when to grow/shrink a topology). We have some basic metrics, but >> they >> > > are >> > > >>> not exposed to the scheduler. The other question is how do we >> deal >> > > with >> > > >>> creating/destroying new bolts, especially around dealing with >> different >> > > >>> groupings. Some groupings make it difficult. There a number of >> ways >> > > to >> > > >>> get around this, but I think the simplest way is to not >> create/destroy >> > > >>> instances, but to treat it as a resources problem, and give them >> more >> > > or >> > > >>> less resources as needed. But that still needs to be discussed. - >> > > Bobby >> > > >>> >> > > >>> >> > > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha >> Balasubramanian < >> > > >>> harsha...@gmail.com> wrote: >> > > >>> >> > > >>> >> > > >>> Hi Bobby, >> > > >>> >> > > >>> Thanks for getting back to me. It is encouraging to hear that >> this >> > > will >> > > >>> be a good addition to Storm if done well. >> > > >>> >> > > >>> I have not implemented the changes yet. My project just started. >> It >> > > >>> spans >> > > >>> over the next 2 months. In a few days, I will create a JIRA task >> and >> > > >>> submit >> > > >>> my proposal. It would be great to brainstorm with the Storm >> community >> > > and >> > > >>> iron out my design. >> > > >>> >> > > >>> >> > > >>> Thanks, >> > > >>> Harsha >> > > >>> >> > > >>> Thanks, >> > > >>> Harsha >> > > >>> >> > > >>> *Contact Details: * >> > > >>> Harsha Balasubramanian >> > > >>> Graduate Student at University of Toronto >> > > >>> Professional Masters in Computer Science >> > > >>> >> > > >>> Email 1: harsha...@cs.toronto.edu >> > > >>> Email 2: harsha...@gmail.com >> > > >>> LinkedIn : ca.linkedin.com/in/harshabala/ >> > > >>> >> > > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans >> > > <ev...@yahoo-inc.com.invalid >> > > >>> > >> > > >>> wrote: >> > > >>> >> > > >>> > Yes, this type of a feature would be great to have. I am rally >> > > curious >> > > >>> > how you have done the changes, as there are a lot of potential >> > > pitfalls >> > > >>> > here. At a minimum it would just be great to have the feedback >> > > >>> framework >> > > >>> > in place so we can iterate on these changes. >> > > >>> > - Bobby >> > > >>> > >> > > >>> > >> > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha < >> > > st...@harsha.io> >> > > >>> > wrote: >> > > >>> > >> > > >>> > >> > > >>> > Hi, >> > > >>> > It will definitely interesting to the storm community. It >> will be >> > > >>> > great if you can create a JIRA and post your code as PR for >> others >> > > >>> > to try out and review the code. >> > > >>> > Thanks, >> > > >>> > Harsha >> > > >>> > >> > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote: >> > > >>> > > Please let me know if my project (outlined below) will be >> useful to >> > > >>> > > Storm. >> > > >>> > > >> > > >>> > > Thanks, >> > > >>> > > Harsha >> > > >>> > > >> > > >>> > > *Contact Details: * >> > > >>> > > Harsha Balasubramanian >> > > >>> > > Graduate Student at University of Toronto >> > > >>> > > Professional Masters in Computer Science >> > > >>> > > >> > > >>> > > Email 1: harsha...@cs.toronto.edu >> > > >>> > > Email 2: harsha...@gmail.com >> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/ >> > > >>> > > >> > > >>> > > ---------- Forwarded message ---------- >> > > >>> > > From: Harsha Balasubramanian <harsha...@gmail.com> >> > > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM >> > > >>> > > Subject: Auto-Scaling >> > > >>> > > To: dev@storm.apache.org >> > > >>> > > >> > > >>> > > >> > > >>> > > Hi, >> > > >>> > > >> > > >>> > > I am a Graduate student at the University of Toronto. As >> part of >> > > my >> > > >>> > > Advanced Database Systems course, I have proposed to >> implement an >> > > >>> Auto >> > > >>> > > Scaling mechanism for Storm topologies (using a Feedback >> System). >> > > >>> > > >> > > >>> > > I've gone through the pages on how to contribute to the Storm >> > > >>> project and >> > > >>> > > have some questions. Please let me know if auto-scaling is >> being >> > > >>> worked >> > > >>> > > on >> > > >>> > > currently. Also, should this be a project in StormCore or >> > > >>> StormProcessor >> > > >>> > > ? >> > > >>> > > >> > > >>> > > If this project will be a good addition to Storm, I will >> create a >> > > new >> > > >>> > > JIRA >> > > >>> > > task for it and add the details of my proposed implementation. >> > > >>> Please do >> > > >>> > > let me know. >> > > >>> > > >> > > >>> > > Thanks, >> > > >>> > > Harsha >> > > >>> > > >> > > >>> > > *Contact Details: * >> > > >>> > > Harsha Balasubramanian >> > > >>> > > Graduate Student at University of Toronto >> > > >>> > > Professional Masters in Computer Science >> > > >>> > > >> > > >>> > > Email 1: harsha...@cs.toronto.edu >> > > >>> > > Email 2: harsha...@gmail.com >> > > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/ >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > >>> >> > > >>> >> > > >>> >> > > >>> >> > > >> >> > > >> >> > > >> > > >> > > >> > > >> > >
