Many things going on here. 1- The logs are receiving edits from all the regions, so a log can't be cleared until all the regions it contains are flushed.
2- When you close regions (drop a tables implies doing that), it doesn't force roll logs (it would be very bad on performance) so they won't get cleared right away even if they can be. 3- Those HLogSplitter lines seem to indicate that you didn't shut down HBase properly since it had to replay the logs. Hope this helps, J-D On Wed, Feb 22, 2012 at 3:03 PM, Alok Singh <[email protected]> wrote: > Environment > Hbase: 0.92 > Hadoop: hadoop-0.20.2-cdh3u3 > > I am testing hbase 0.92 for a new storage system we are building. In > the tests, I insert around 2-3 billion rows and then run some > scans/queries against it to test the performance. Once the tests are > complete, I drop all of the tables and recreate them to clear out the > test data. But, it doesn't look like the /hbase/root/.logs is getting > cleared. It seems to just keep growing, and the only way to clear it > out is to restart the hbase cluster. After the restarts, the > regionserver logs are filled with this: > --------- > 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory > doesn't exist: > hdfs://10.130.1.15:7080/foo/hbase/root/device/9605bdd29d02212f23499296542341d8. > It is very likely that it was already split so it's safe to discard > those edits. > 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory > doesn't exist: > hdfs://10.130.1.15:7080/foo/hbase/root/device.device/b3b96cd17e683e52f8c72e14229f5567. > It is very likely that it was already split so it's safe to discard > those edits. > 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory > doesn't exist: > hdfs://10.130.1.15:7080/foo/hbase/root/device/83832bcdc8a1bde1094d9039594cb067. > It is very likely that it was already split so it's safe to discard > those edits. > 12/02/22 22:57:58 INFO wal.HLogSplitter: This region's directory > doesn't exist: > hdfs://10.130.1.15:7080/foo/hbase/root/device/140ded7a67c8cf2a0e7cbbf037033889. > It is very likely that it was already split so it's safe to discard > those edits. > ..... > ---------- > > Size of /hbase/root/device: 110GB > Size of /hbase/root/.logs: 770GB > > Alok
