Kevin, Too many hlogs means that the inserts are hitting a lot of regions, that those regions aren't filled enough to flush so that we have to force flush them to give some room. When you added region servers, it spread the regions load so that hlogs were getting filled at a slower rate.
Could you tell us more about the rate of insertion, size of data, and number of regions per region server? Thx, J-D On Tue, Dec 15, 2009 at 10:34 AM, Kevin Peterson <kevin...@gmail.com> wrote: > We're running a 13 node HBase cluster. We had some problems a week ago with > it being overloaded and errors related to not being able to find a block on > HDFS, but adding four more nodes and increasing max heap from 3GB to 4.5GB > on all nodes fixed any problems. > > Looking at the logs now, though, we see that HLogs are not getting removed: > > 2009-12-15 01:45:48,426 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll > /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260867136036, > entries=210524, calcsize=63757422, filesize=41073798. New hlog > /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421 > 2009-12-15 01:45:48,427 INFO org.apache.hadoop.hbase.regionserver.HLog: Too > many hlogs: logs=130, maxlogs=96; forcing flush of region with oldest edits: > articles-article-id,f15489ea-38a4-4127-9179-1b2dc5f3b5d4,1260083783909 > 2009-12-15 01:57:14,188 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region > articles,\x00\x00\x01\x25\x8C\x0F\xCB\x18\xB5U\xF7\xC6\x5DoH\xB8\x98\xEBH,E\x7C\x07\x14,1260830133341 > 2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll > /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260870348421, > entries=92795, calcsize=63908073, filesize=54042783. New hlog > /hbase/.logs/mi-prod-app33,60020,1260495617070/hlog.dat.1260871037510 > 2009-12-15 01:57:17,519 INFO org.apache.hadoop.hbase.regionserver.HLog: Too > many hlogs: logs=131, maxlogs=96; forcing flush of region with oldest edits: > articles-article-id,f1cd1b02-3d1b-453c-b44f-94ec5a1e3a46,1260007536878 > > From reading the log message, I interpret this as saying that every time it > rolls an hlog, if there are more than maxlogs logs, it will flush one > region. I'm assuming that a log could have edits for multiple regions, so > this seems to mean that if we have 100 regions and maxlogs set to 96, if it > flushes one region each time it rolls a log, it will create 100 logs before > it flushes all regions and is able to delete the log, so it will reach > steady state at 196 hlogs. Is this correct? > > We're concerned because when we had problems last week, we saw lots of log > messages related to "Too many hlogs" and had assumed they were related to > the problems. Is this anything to worry about? >
