I think there is a swap happening on regionservers due to less memory... increase the allotted memory in hadoop-site.xml for all the slaves.
On Mon, Jul 27, 2009 at 3:19 PM, Angus He <[email protected]> wrote: > Hi All, > > We have a cluster with 4 nodes, one for the Master server, and others > for Region Server. > > In our case, there are 3 Downloaders crawling some specific web pages > which will be saved in HBase(0.19.3) then. > After running the Downloaders for a while, we found that there are > tons of Exceptions in the hbase log files like this: > > 2009-07-27 12:12:25,833 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush > of ~1.8m for region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > in 316ms, sequence id=4774267, compaction requested=true > 2009-07-27 12:12:25,833 DEBUG > org.apache.hadoop.hbase.regionserver.HStore: closed > 1734514356/CF_INFORMATION > 2009-07-27 12:12:25,834 DEBUG > org.apache.hadoop.hbase.regionserver.HStore: closed > 1734514356/CF_CONTENT > 2009-07-27 12:12:25,834 INFO > org.apache.hadoop.hbase.regionserver.HRegion: Closed > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > 2009-07-27 12:12:25,837 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.NotServingRegionException: Region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > closed > 2009-07-27 12:12:25,837 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.NotServingRegionException: Region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > closed > 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC > Server handler 2 on 60020, call getRow([...@187885c2, [...@1095901b, null, > 9223372036854775807, 1, -1) from 192.168.33.9:59836: error: > org.apache.hadoop.hbase.NotServingRegionException: Region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > closed > org.apache.hadoop.hbase.NotServingRegionException: Region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > closed > at > org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857) > at > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921) > at > org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543) > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) > 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC > Server handler 5 on 60020, call getRow([...@28333b1e, [...@3b6e4330, null, > 9223372036854775807, 1, -1) from 192.168.33.5:58297: error: > org.apache.hadoop.hbase.NotServingRegionException: Region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > closed > org.apache.hadoop.hbase.NotServingRegionException: Region > webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 > closed > at > org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857) > at > org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921) > at > org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543) > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) > > > And from then , the Downloaders got a big drop in the speed of page > crawling. > > Any ideas? > > > -- > Regards > Angus >
