Thanks Bobby. I setup a local server and CLI, UI and my topology run fine. However there is a feature that doesn't work and I need some help to figure out why.
1. I followed the instructions here to setup the storm system, http://ptgoetz.github.io/blog/2013/12/18/running-apache-storm-on-windows/ 2. My feature requires the usage of the CLI tool to re-balance the topology when it needs more resources. For that I thought I could use the following CLI command, storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10 I found this in the storm documentation at the bottom of this page: http://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html 3. When I run this command in my setup, I get the error pasted below. Is re-balancing supported at the spouts and bolts level? Am I using the wrong version of the storm jar. This works: *C:\storm>storm rebalance WordCount -n 15* *2493 [main] INFO backtype.storm.thrift - Connecting to Nimbus at localhost:6627* *3573 [main] INFO backtype.storm.command.rebalance - Topology WordCount is rebalancing* This does not work: *C:\storm>storm rebalance WordCount -n 5 -e Count=10* *Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1* * at java.lang.String.substring(Unknown Source)* * at backtype.storm.command.rebalance$parse_executor.invoke(rebalance.clj:24)* * at clojure.tools.cli$apply_specs.invoke(cli.clj:80)* * at clojure.tools.cli$cli.doInvoke(cli.clj:130)* * at clojure.lang.RestFn.invoke(RestFn.java:460)* * at backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:31)* * at clojure.lang.RestFn.applyTo(RestFn.java:137)* * at backtype.storm.command.rebalance.main(Unknown Source)* Thanks, Harsha *Contact Details: * Harsha Balasubramanian Graduate Student at University of Toronto Masters in Computer Science Mobile : 647-771-3596 Email 1: harsha...@cs.toronto.edu Email 2: harsha...@gmail.com LinkedIn : ca.linkedin.com/in/harshabala/ On Mon, Dec 22, 2014 at 4:42 PM, Bobby Evans <ev...@yahoo-inc.com.invalid> wrote: > Harsha, > Sorry about that, I have been out at a conference. > > for 1 you can get the number of acks so far, but you are going to have to > calculate the rate yourself, it keeps track of the number of acks over > several time windows, but reports them as whole numbers. > What do you mean by a local cluster? Do you mean a single node cluster > you launched yourself? then yes. If you mean a local-mode cluster, then no. > - Bobby > > > On Tuesday, December 16, 2014 2:35 PM, Harsha Balasubramanian < > harsha...@gmail.com> wrote: > > > Hi, > > Can someone pls answer my questions from the above thread. > > > *I have 2 questions for you;* > *1. In one of your earlier emails, you mentioned that there is > a limited Metrics implementation which is available in the current Storm > version. * > > *Will this provide 'acks threshold', i.e, the number of acks per second (or > minute)? If not, can you please suggest a way in which I can implement this > myself. * > > *2. Is it possible to use the CLI and UI tools from a local cluster?* > > Thanks, > Harsha > > *Contact Details: * > Harsha Balasubramanian > Graduate Student at University of Toronto > Masters in Computer Science > > Mobile : 647-771-3596 > Email 1: harsha...@cs.toronto.edu > Email 2: harsha...@gmail.com > LinkedIn : ca.linkedin.com/in/harshabala/ > > On Mon, Dec 15, 2014 at 12:23 PM, Harsha Balasubramanian < > harsha...@gmail.com> wrote: > > > > Hi Bobby, > > > > Hope this email finds you well. I.m about to start designing the > > auto-scaling system. Created a JIRA for it, > > https://issues.apache.org/jira/browse/STORM-594 > > > > Please take a look and let me know if there are any concerns. > > > > I have 2 questions for you; > > 1. In one of your earlier emails, you mentioned that there is a *limited > *Metrics > > implementation which is available in the current Storm version. > > Will this provide 'acks threshold'? > > > > 2. Is it possible to use the CLI tool from a local cluster? > > > > Thanks, > > Harsha > > > > *Contact Details: * > > Harsha Balasubramanian > > Graduate Student at University of Toronto > > Masters in Computer Science > > > > Mobile : 647-771-3596 > > Email 1: harsha...@cs.toronto.edu > > Email 2: harsha...@gmail.com > > LinkedIn : ca.linkedin.com/in/harshabala/ > > > > On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian < > > harsha...@gmail.com> wrote: > >> > >> Thanks for the detailed explanation Bobby. I will keep this in mind when > >> I create my design document. > >> > >> I will mostly not deal with adding/removing bolts to the topology and > >> restrict myself to threads and tasks. This is because of the time I > have to > >> complete the project. Once I submit my report (early January), I can > extend > >> what I have implemented to more use cases. > >> > >> Thanks, > >> Harsha > >> > >> Thanks, > >> Harsha > >> > >> *Contact Details: * > >> Harsha Balasubramanian > >> Graduate Student at University of Toronto > >> Professional Masters in Computer Science > >> > >> Email 1: harsha...@cs.toronto.edu > >> Email 2: harsha...@gmail.com > >> LinkedIn : ca.linkedin.com/in/harshabala/ > >> > >> On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans > <ev...@yahoo-inc.com.invalid > >> > wrote: > >> > >>> Sure, > >>> The main thing that storm is missing right now is an metrics feedback > >>> system to the scheduler (or possibly a separate piece of code that > decides > >>> when to grow/shrink a topology). We have some basic metrics, but they > are > >>> not exposed to the scheduler. The other question is how do we deal > with > >>> creating/destroying new bolts, especially around dealing with different > >>> groupings. Some groupings make it difficult. There a number of ways > to > >>> get around this, but I think the simplest way is to not create/destroy > >>> instances, but to treat it as a resources problem, and give them more > or > >>> less resources as needed. But that still needs to be discussed. - > Bobby > >>> > >>> > >>> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian < > >>> harsha...@gmail.com> wrote: > >>> > >>> > >>> Hi Bobby, > >>> > >>> Thanks for getting back to me. It is encouraging to hear that this > will > >>> be a good addition to Storm if done well. > >>> > >>> I have not implemented the changes yet. My project just started. It > >>> spans > >>> over the next 2 months. In a few days, I will create a JIRA task and > >>> submit > >>> my proposal. It would be great to brainstorm with the Storm community > and > >>> iron out my design. > >>> > >>> > >>> Thanks, > >>> Harsha > >>> > >>> Thanks, > >>> Harsha > >>> > >>> *Contact Details: * > >>> Harsha Balasubramanian > >>> Graduate Student at University of Toronto > >>> Professional Masters in Computer Science > >>> > >>> Email 1: harsha...@cs.toronto.edu > >>> Email 2: harsha...@gmail.com > >>> LinkedIn : ca.linkedin.com/in/harshabala/ > >>> > >>> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans > <ev...@yahoo-inc.com.invalid > >>> > > >>> wrote: > >>> > >>> > Yes, this type of a feature would be great to have. I am rally > curious > >>> > how you have done the changes, as there are a lot of potential > pitfalls > >>> > here. At a minimum it would just be great to have the feedback > >>> framework > >>> > in place so we can iterate on these changes. > >>> > - Bobby > >>> > > >>> > > >>> > On Wednesday, November 12, 2014 2:05 PM, Harsha < > st...@harsha.io> > >>> > wrote: > >>> > > >>> > > >>> > Hi, > >>> > It will definitely interesting to the storm community. It will be > >>> > great if you can create a JIRA and post your code as PR for others > >>> > to try out and review the code. > >>> > Thanks, > >>> > Harsha > >>> > > >>> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote: > >>> > > Please let me know if my project (outlined below) will be useful to > >>> > > Storm. > >>> > > > >>> > > Thanks, > >>> > > Harsha > >>> > > > >>> > > *Contact Details: * > >>> > > Harsha Balasubramanian > >>> > > Graduate Student at University of Toronto > >>> > > Professional Masters in Computer Science > >>> > > > >>> > > Email 1: harsha...@cs.toronto.edu > >>> > > Email 2: harsha...@gmail.com > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/ > >>> > > > >>> > > ---------- Forwarded message ---------- > >>> > > From: Harsha Balasubramanian <harsha...@gmail.com> > >>> > > Date: Tue, Nov 11, 2014 at 8:15 PM > >>> > > Subject: Auto-Scaling > >>> > > To: dev@storm.apache.org > >>> > > > >>> > > > >>> > > Hi, > >>> > > > >>> > > I am a Graduate student at the University of Toronto. As part of > my > >>> > > Advanced Database Systems course, I have proposed to implement an > >>> Auto > >>> > > Scaling mechanism for Storm topologies (using a Feedback System). > >>> > > > >>> > > I've gone through the pages on how to contribute to the Storm > >>> project and > >>> > > have some questions. Please let me know if auto-scaling is being > >>> worked > >>> > > on > >>> > > currently. Also, should this be a project in StormCore or > >>> StormProcessor > >>> > > ? > >>> > > > >>> > > If this project will be a good addition to Storm, I will create a > new > >>> > > JIRA > >>> > > task for it and add the details of my proposed implementation. > >>> Please do > >>> > > let me know. > >>> > > > >>> > > Thanks, > >>> > > Harsha > >>> > > > >>> > > *Contact Details: * > >>> > > Harsha Balasubramanian > >>> > > Graduate Student at University of Toronto > >>> > > Professional Masters in Computer Science > >>> > > > >>> > > Email 1: harsha...@cs.toronto.edu > >>> > > Email 2: harsha...@gmail.com > >>> > > LinkedIn : ca.linkedin.com/in/harshabala/ > >>> > > >>> > > >>> > > >>> > > >>> > >>> > >>> > >>> > >> > >> > > > >
