Hi Bobby, Hope this email finds you well. I.m about to start designing the auto-scaling system. Created a JIRA for it, https://issues.apache.org/jira/browse/STORM-594
Please take a look and let me know if there are any concerns. I have 2 questions for you; 1. In one of your earlier emails, you mentioned that there is a *limited *Metrics implementation which is available in the current Storm version. Will this provide 'acks threshold'? 2. Is it possible to use the CLI tool from a local cluster? Thanks, Harsha *Contact Details: * Harsha Balasubramanian Graduate Student at University of Toronto Masters in Computer Science Mobile : 647-771-3596 Email 1: harsha...@cs.toronto.edu Email 2: harsha...@gmail.com LinkedIn : ca.linkedin.com/in/harshabala/ On Wed, Nov 12, 2014 at 6:28 PM, Harsha Balasubramanian <harsha...@gmail.com > wrote: > > Thanks for the detailed explanation Bobby. I will keep this in mind when I > create my design document. > > I will mostly not deal with adding/removing bolts to the topology and > restrict myself to threads and tasks. This is because of the time I have to > complete the project. Once I submit my report (early January), I can extend > what I have implemented to more use cases. > > Thanks, > Harsha > > Thanks, > Harsha > > *Contact Details: * > Harsha Balasubramanian > Graduate Student at University of Toronto > Professional Masters in Computer Science > > Email 1: harsha...@cs.toronto.edu > Email 2: harsha...@gmail.com > LinkedIn : ca.linkedin.com/in/harshabala/ > > On Wed, Nov 12, 2014 at 6:05 PM, Bobby Evans <ev...@yahoo-inc.com.invalid> > wrote: > >> Sure, >> The main thing that storm is missing right now is an metrics feedback >> system to the scheduler (or possibly a separate piece of code that decides >> when to grow/shrink a topology). We have some basic metrics, but they are >> not exposed to the scheduler. The other question is how do we deal with >> creating/destroying new bolts, especially around dealing with different >> groupings. Some groupings make it difficult. There a number of ways to >> get around this, but I think the simplest way is to not create/destroy >> instances, but to treat it as a resources problem, and give them more or >> less resources as needed. But that still needs to be discussed. - Bobby >> >> >> On Wednesday, November 12, 2014 3:10 PM, Harsha Balasubramanian < >> harsha...@gmail.com> wrote: >> >> >> Hi Bobby, >> >> Thanks for getting back to me. It is encouraging to hear that this will >> be a good addition to Storm if done well. >> >> I have not implemented the changes yet. My project just started. It >> spans >> over the next 2 months. In a few days, I will create a JIRA task and >> submit >> my proposal. It would be great to brainstorm with the Storm community and >> iron out my design. >> >> >> Thanks, >> Harsha >> >> Thanks, >> Harsha >> >> *Contact Details: * >> Harsha Balasubramanian >> Graduate Student at University of Toronto >> Professional Masters in Computer Science >> >> Email 1: harsha...@cs.toronto.edu >> Email 2: harsha...@gmail.com >> LinkedIn : ca.linkedin.com/in/harshabala/ >> >> On Wed, Nov 12, 2014 at 3:47 PM, Bobby Evans <ev...@yahoo-inc.com.invalid >> > >> wrote: >> >> > Yes, this type of a feature would be great to have. I am rally curious >> > how you have done the changes, as there are a lot of potential pitfalls >> > here. At a minimum it would just be great to have the feedback >> framework >> > in place so we can iterate on these changes. >> > - Bobby >> > >> > >> > On Wednesday, November 12, 2014 2:05 PM, Harsha <st...@harsha.io> >> > wrote: >> > >> > >> > Hi, >> > It will definitely interesting to the storm community. It will be >> > great if you can create a JIRA and post your code as PR for others >> > to try out and review the code. >> > Thanks, >> > Harsha >> > >> > On Wed, Nov 12, 2014, at 11:16 AM, Harsha Balasubramanian wrote: >> > > Please let me know if my project (outlined below) will be useful to >> > > Storm. >> > > >> > > Thanks, >> > > Harsha >> > > >> > > *Contact Details: * >> > > Harsha Balasubramanian >> > > Graduate Student at University of Toronto >> > > Professional Masters in Computer Science >> > > >> > > Email 1: harsha...@cs.toronto.edu >> > > Email 2: harsha...@gmail.com >> > > LinkedIn : ca.linkedin.com/in/harshabala/ >> > > >> > > ---------- Forwarded message ---------- >> > > From: Harsha Balasubramanian <harsha...@gmail.com> >> > > Date: Tue, Nov 11, 2014 at 8:15 PM >> > > Subject: Auto-Scaling >> > > To: dev@storm.apache.org >> > > >> > > >> > > Hi, >> > > >> > > I am a Graduate student at the University of Toronto. As part of my >> > > Advanced Database Systems course, I have proposed to implement an Auto >> > > Scaling mechanism for Storm topologies (using a Feedback System). >> > > >> > > I've gone through the pages on how to contribute to the Storm project >> and >> > > have some questions. Please let me know if auto-scaling is being >> worked >> > > on >> > > currently. Also, should this be a project in StormCore or >> StormProcessor >> > > ? >> > > >> > > If this project will be a good addition to Storm, I will create a new >> > > JIRA >> > > task for it and add the details of my proposed implementation. Please >> do >> > > let me know. >> > > >> > > Thanks, >> > > Harsha >> > > >> > > *Contact Details: * >> > > Harsha Balasubramanian >> > > Graduate Student at University of Toronto >> > > Professional Masters in Computer Science >> > > >> > > Email 1: harsha...@cs.toronto.edu >> > > Email 2: harsha...@gmail.com >> > > LinkedIn : ca.linkedin.com/in/harshabala/ >> > >> > >> > >> > >> >> >> >> > >
