Hi All, We have a cluster with 4 nodes, one for the Master server, and others for Region Server.
In our case, there are 3 Downloaders crawling some specific web pages which will be saved in HBase(0.19.3) then. After running the Downloaders for a while, we found that there are tons of Exceptions in the hbase log files like this: 2009-07-27 12:12:25,833 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush of ~1.8m for region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 in 316ms, sequence id=4774267, compaction requested=true 2009-07-27 12:12:25,833 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1734514356/CF_INFORMATION 2009-07-27 12:12:25,834 DEBUG org.apache.hadoop.hbase.regionserver.HStore: closed 1734514356/CF_CONTENT 2009-07-27 12:12:25,834 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 2009-07-27 12:12:25,837 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: org.apache.hadoop.hbase.NotServingRegionException: Region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 closed 2009-07-27 12:12:25,837 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: org.apache.hadoop.hbase.NotServingRegionException: Region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 closed 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020, call getRow([...@187885c2, [...@1095901b, null, 9223372036854775807, 1, -1) from 192.168.33.9:59836: error: org.apache.hadoop.hbase.NotServingRegionException: Region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 closed org.apache.hadoop.hbase.NotServingRegionException: Region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 closed at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857) at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921) at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020, call getRow([...@28333b1e, [...@3b6e4330, null, 9223372036854775807, 1, -1) from 192.168.33.5:58297: error: org.apache.hadoop.hbase.NotServingRegionException: Region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 closed org.apache.hadoop.hbase.NotServingRegionException: Region webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 closed at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857) at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921) at org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) And from then , the Downloaders got a big drop in the speed of page crawling. Any ideas? -- Regards Angus
