On Fri, May 22, 2009 at 7:21 AM, Kirill Shabunov <[email protected]> wrote:
> Thanks a lot! > > I've tried, it seems to be much more stable overall. No component have > crashed so far. I'm glad. If the RC is working for you, please vote for it over in hbase-dev mailing list. > > > I see some new messages, like this: > WARN org.apache.hadoop.hbase.regionserver.HLog: > regionserver/0:0:0:0:0:0:0:0:60020.logFlusher took 1661ms optional sync to > HLog; editcount=0 > This is a new log I added. It triggers every time an append or a "sync" on the commit log takes > 1 second. I added the log to highlight hdfs slow periods taking on writes. Its looking like we are going to have to put up a pool of commit log writers if we want to keep up write rates. The logging is just so we begin to gather stats. In the above, it looks like it happened just after file open (editcount=0). I also changed some settings: > > 1. upped hbase.master.lease.period to allow regions to be busy longer. > Thats probably fine. > > 2. decreased some thresholds, where I hope it should reduce RAM usage: > - hbase.hregion.memcache.flush.size -- smaller buffers (?). > - hbase.hregion.max.filesize -- smaller the region, less RAM for the > index (?). > Regards memory use, I would have left the above settings at default. In 0.19.x era hbase, its the size of your storefile indices that takes up bulk of heap. See http://wiki.apache.org/hadoop/Hbase/FAQ#4. This should all be much better in 0.20.0 hbase. St.Ack
