On Fri, Apr 10, 2009 at 12:02 AM, Josh Adams <[email protected]> wrote: > > [Feel free to remove hypertable-dev if you feel it's more of a -user > thread...] > > On Thu, Apr 9, 2009 at 9:25 AM, Doug Judd <[email protected]> wrote: > > I think the variance you observed here must have just been a coincidence. > > This property is no longer used by the KFS broker. > > Hehe, fair enough. :-) It was most likely was a coincidence, but I > thought I'd ask for kicks: is there any possibility that the value of > RangeServer.AccessGroup.CellCache.PageSize could have anything to do > with how quickly the DFS broker's memory would grow?
No, it shouldn't. One thing that might help is to install tcmalloc (google-perftools) and then re-build. You'll need to have tcmalloc installed in all your runtime environments. > Sort of still on-topic is the general question of which config values > in particular should we try to tweak in order to tune various types of > performance behavior? The ConfigProperties page and --help-config > give some description of what there is, but I haven't seen much in the > way of a practical performance example mapping to config suggestion. Generally speaking, you shouldn't have to tweak config much to get optimal performance. We've tried to set things up so that the system will auto-tune. A lot of those config options are for testing. We use them to force certain events (e.g. compactions) to happen during a regression test. Right now we're interested in smoothing out occasional drops in raw > select performance. For example, I can do select * from our "events" > table and most of the time I reliably get around 16MB/s of throughput. > However, sometimes the throughput goes down to something like 200KB/s > for 20-30 seconds, then goes back up to normal. Watching the > RangeServers during the slowdowns I can usually find one of them being > really busy with something like major log compaction. I'm trying to > figure out both which config options to try and what is the best > general path to identify bottlenecks throughout the stack. I'm > wondering if there is an easy way of asking the RangeServers if they > are seeing poor performance from the DFS/network or if they are being > slow for some other reason, beyond what I can get from parsing their > logs (which can certainly be quite useful.) Compactions can definitely cause a drag on the system. However, that's a pretty significant plunge in performance. Scanner locks have been pushed down as low as possible. This may be causing the problem. I filed issue 157 <http://code.google.com/p/hypertable/issues/detail?id=157> on it a while back. It would be interesting to know if the disk subsystems on any of your machines are getting saturated during this low throughput condition. If so, then there probably is not much we can do about it. However, if it's a CPU bottleneck (which I suspect it is), then issue 157 should help mitigate the problem. - Doug --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
