Yup , I found this in the Master log file. 2009-07-27 12:13:10,880 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT: webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868: webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 split; daughters: webpage,http://www.tianya.cn/techforum/content/362/20957.shtml,1248667945476, webpage,http://www.tianya.cn/techforum/content/50/565730.shtml,1248667945476: safeMode=false from 192.168.33.5:60020
BTW, There is about a 50-second time lag between Region server and Master server. On Tue, Jul 28, 2009 at 11:45 AM, stack<[email protected]> wrote: > I was looking at this today. > > When you start to get the NotServingRegionException, has the region just > split? You can tell by looking for "webpage, > http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868"; in the > master log. If it has a message about splitting just before you get the > below NSRE exception, then it seems like its a case of HBASE-1671. > Hopefully will have a fix this evening. > > St.Ack > > On Mon, Jul 27, 2009 at 2:49 AM, Angus He <[email protected]> wrote: > >> Hi All, >> >> We have a cluster with 4 nodes, one for the Master server, and others >> for Region Server. >> >> In our case, there are 3 Downloaders crawling some specific web pages >> which will be saved in HBase(0.19.3) then. >> After running the Downloaders for a while, we found that there are >> tons of Exceptions in the hbase log files like this: >> >> 2009-07-27 12:12:25,833 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache flush >> of ~1.8m for region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> in 316ms, sequence id=4774267, compaction requested=true >> 2009-07-27 12:12:25,833 DEBUG >> org.apache.hadoop.hbase.regionserver.HStore: closed >> 1734514356/CF_INFORMATION >> 2009-07-27 12:12:25,834 DEBUG >> org.apache.hadoop.hbase.regionserver.HStore: closed >> 1734514356/CF_CONTENT >> 2009-07-27 12:12:25,834 INFO >> org.apache.hadoop.hbase.regionserver.HRegion: Closed >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> 2009-07-27<http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868%0A2009-07-27>12:12:25,837 >> ERROR >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> org.apache.hadoop.hbase.NotServingRegionException: Region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> closed >> 2009-07-27 12:12:25,837 ERROR >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> org.apache.hadoop.hbase.NotServingRegionException: Region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> closed >> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC >> Server handler 2 on 60020, call getRow([...@187885c2, [...@1095901b, null, >> 9223372036854775807, 1, -1) from 192.168.33.9:59836: error: >> org.apache.hadoop.hbase.NotServingRegionException: Region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> closed >> org.apache.hadoop.hbase.NotServingRegionException: Region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> closed >> at >> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543) >> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) >> 2009-07-27 12:12:25,841 INFO org.apache.hadoop.ipc.HBaseServer: IPC >> Server handler 5 on 60020, call getRow([...@28333b1e, [...@3b6e4330, null, >> 9223372036854775807, 1, -1) from 192.168.33.5:58297: error: >> org.apache.hadoop.hbase.NotServingRegionException: Region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> closed >> org.apache.hadoop.hbase.NotServingRegionException: Region >> webpage, >> http://www.tianya.cn/techforum/content/362/20957.shtml,1248277178868 >> closed >> at >> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1857) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1921) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.getFull(HRegion.java:1020) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1543) >> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912) >> >> >> And from then , the Downloaders got a big drop in the speed of page >> crawling. >> >> Any ideas? >> >> >> -- >> Regards >> Angus >> > -- Regards Angus
