Hi Doug, Thanks for the nice writeup -- at this point we would like to start carving out what the API would like to allow people to extend with their own intelligent load balancing algorithms. Load balancing of this type is recognized as a hard research problem(*) and I would like to see Hypertable be used as central to that research.
We'll need a set of abstractions and interfaces that allow a variety of approaches that people might like to apply -- reinforcement learning, constraint programming, or other AI style planning approaches. I'd like to collect some feedback from a few different sources to see if we can solve for the minimal interface that gives everyone the inputs and the controls they need to devise a balancer. Then, we can produce an API recommendation. Also, we should add a bit more detail to talk about policy for calling the balancer. Intelligence in the balancing strategy might probably better be served by allowing the balancer policy itself to be learned or optimized. For example, the system might be better left in a slightly unbalanced condition rather than pay the balancing cost if it's learned that the unbalanced condition is temporary. I'll point out the problem and this mailing list to folks doing research in ML (particularly SysML or ML applied to systems management) but also from the constraint computing field. (*) see for example: http://www2.cs.uni-paderborn.de/cs/ag-monien/RESEARCH/LOADBAL/ On Sun, Dec 19, 2010 at 9:15 PM, Doug Judd <[email protected]> wrote: > We're now 100% focused on load balancing. This will handle the addition of > new nodes as well as normal load imbalances. I've come up with a design and > have described it in the following document: > > http://code.google.com/p/hypertable/wiki/LoadBalancing > > Please read it and send your feedback. Thanks! > > - Doug > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<hypertable-dev%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- Gordon Rios -- Cork Constraint Computation Centre http://www.4c.ucc.ie/web/people.jsp?id=144 http://www.linkedin.com/in/gordonrios Ireland: +353 86 089 2416 USA: +1 650 906 3473 -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
