Hi Charles, Thanks for the feedback. Comments inline ...
On Thu, Dec 23, 2010 at 8:12 AM, Charles <[email protected]> wrote: > > [...] 1. Although the design document talks about passing training data to > the LoadBalancer object, glancing briefly at the pseudocode for the > class definition it's not clear to me what the API is for passing the > training data is or what the format would be. > We were thinking more along the lines of unsupervised learning. We certainly could explore a supervised learning approach. Any ideas for what the training data should look like? Also, how could we go about generating it? > 2. I'm assuming that the LoadBalancer has full access to the items in > the data table itself during the balance operation, in case it wants > to collect data about them for use in the prediction. Not clear if > this would be useful b/c of the cost involved in gathering this data, > but interesting to explore. > When you say "data table" I assume you're referring to the sys/RS_STATS table. The LoadBalancer will have full access to the items in this table. In fact this table exists solely to feed the LoadBalancer performance statistics. This table is populated by the RangeServers directly to minimize the impact of statistics gathering on the system. This means that the load balancer and the RangeServers will need to coordinate on what information gets collected, how often, and how much historical information will be kept around. 3. I would expect that practical implementations of LoadBalancer would > want a way to serialise their nontrivial state (presumably using HT > itself), but not sure if there's any special API support required for > that. (Maybe a reserved table for LB data?) > The LoadBalancer runs in the Master process (at least for now). The Master has a meta log (MML) that is written to the underlying DFS. Currently the plan is to have the basic balancer serialize state about in-progress balance operations in the MML. We can use the MML to persist other LoadBalancer state as well. However, the MML is designed to hold a very small amount of data. If you think there might be need for persisting a very large amount of state, we can consider another system table (sys/balancer). 4. It may be worth providing a convenience implementation of > LoadBalancer that works in the batch setting like the basic algorithm, > i.e., a superclass for load balancers that want to operate once a day > based on data that has been collected in the last 24 hours. > Sounds good. > 5. A LoadBalancer might want to use different strategies for the cases > of adding a new range server versus high variance among servers. Is > there a way for the master to signal which of these situations is the > case? > The monitoring data that gets fed into the LoadBalancer on a regular interval (e.g. 30 seconds) will contain range server and range count information for all of the range servers. If a new range server suddenly appears with a range count of zero, that would imply that a new server was added. 6. Of course an effective challenge problem would also require a test > workload that is challenging enough to be representative of real > usages of the load balancer. As close to real usage as possible would > be best, to try to forestall the danger of designing ML algorithms > that are strong enough to learn the features of the synthetic problem > generator but not that of real data. > We'll work on pulling some real-world workload together. The realtime Twitter stream sample <http://dev.twitter.com/pages/streaming_api> might be a good place to start. > 7. It is unclear what the optimal granularity for aggregating the > range counts would be (could be less than 30 sec, or more). Might > want to have this settable parameter of the master. Note that this is > orthogonal to how often the master decides to send data to the load > balancer, e.g., the master could send data every thirty seconds that > are 6 bins of counts recorded every 5 sec. The LoadBalancer API has a method for publishing the stats gathering interval, so balancers would have a way to change it. There's not much cost associated with sending data to the LoadBalancer, so I don't think the vector approach would be necessary. We empirically chose 30 seconds as the default because there is some overhead involved in gathering statistics, including network communication with all of the range servers and mutex locking/unlocking for each range managed by each server. 8. Wrt the objective functions, different objective performance > metrics > that are of interest to the user, and the user might want to have > knobs to say (e.g.) exactly what SLA they would like satisfied. But > it's not clear to me whether this is part of load balancing (i.e., > deciding which ranges are served by which server) or auto-scaling > (deciding how many servers to have). It may be too early to lock down > an API on this without having more experience with practical > SML/Optimization load balancers. > We were thinking that "load average" would be a good overall objective performance metric. But now that you mention it, I suppose optimizing for query latency or overall throughput, might yield a different balance. Adding this sort of user input is trivial to do. On a related note, there are a number of other places in the system that could benefit from machine learning. Query cache size is one that comes to mind. Currently the query cache size is statically configured, with a default size of 50MB. It would be great to learn the optimal size for this cache to improve query latency. - Doug -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
