I notice in the above that you have file descriptors set to 1024 still. I'd suggest reading the 'Getting Started' section: http://hadoop.apache.org/hbase/docs/r0.19.0/api/overview-summary.html#overview_description .
How many regions do you have loaded? On such a small machine, I'd be surprised if you got it all to run together. St.Ack On Thu, Feb 26, 2009 at 7:14 AM, Andrew McCall <[email protected]>wrote: > Hi J-D, > > I'm running the HBase 0.19 release downloaded to the website on a pretty > lacklustre linux machine; 1.6Ghz single core processor, 512MB ram with 1.5GB > swap space enabled and 80GB hard disk. It's a testing machine, I've been > using for development. It's running Hadoop core 0.19 in pseudo distributed > mode and running a map/reduce reduce crawl using nutch and HBase. I've been > planning to move to an EC3 cluster in the near future but I've been putting > it off since everything seemed to come up and work on the testing box. > > It's entirely possible that it is a result of a swap, how do I tell? I've > looked in all the logs for hadoop, hbase and my job and can't see that > anything stopped logging around then, the logs all have entries for 9:34. > I've also increased my lease timeout from the default to 600,000 ms in my > hbase-site, which is below as a result of another issue. I did search of the > mailing list for the other issue and answers suggested that the lease > timeout should be the same as my map/reduce job timeout. > > <configuration> > <property> > <name>hbase.rootdir</name> > <value>hdfs://localhost:9000/hbase</value> > <description>The directory shared by region servers. > </description> > </property> > <property> > <name>hbase.regionserver.lease.period</name> > <value>600000</value> > </property> > </configuration> > > Thanks for the help, > Andrew > > > > On 26 Feb 2009, at 13:41, Jean-Daniel Cryans wrote: > > Andrew, >> >> Oh yes sorry, your setup is in local mode so no region server in a >> different >> thread. My bad! >> >> That lease timeout probably means that you were swapping and a full jvm >> garbage collecting occurred which blocked the main thread. Can you make >> sure >> that it did swap at that moment? >> >> It would also be helpful to get the usual info like hardware, OS, hbase >> version, etc. >> >> Thx! >> >> J-D >> >> On Thu, Feb 26, 2009 at 8:35 AM, Andrew McCall <[email protected] >> >wrote: >> >> Hi J-D, >>> >>> I did check the region server logs and they didn't seem to be very useful >>> at all: >>> >>> Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on betty.goroam.net >>> ulimit -n 1024 >>> 2009-02-23 22:11:58,817 WARN >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a >>> distinct >>> region server because hbase.master is set to 'local' mode >>> >>> Is all that is in them, once for each time I started HBase. >>> >>> I saw the shutdown message, which was preceded by this: >>> >>> 2009-02-24 09:34:41,579 INFO >>> org.apache.hadoop.hbase.master.ServerManager: >>> 127.0.0.1:40076 lease expired >>> >>> I can't find any evidence it's related to my map/reduce task, it's the >>> only >>> one in the log files and it's suspiciously just before the process >>> shutdown >>> message - but, I'm new to this and honestly don't know what to make of >>> it. >>> >>> Andrew >>> >>> >>> On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote: >>> >>> Andrew, >>> >>>> >>>> For some reason, your region server shut down around 9:36: >>>> >>>> 2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase. >>>> >>>> >>>>> master.RegionServerOperation: process shutdown of server >>>>> 127.0.0.1:40076 >>>>> : >>>>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, >>>>> >>>>> >>>> >>>> But just before that the master received a startup message from it. Did >>>> you >>>> take a look at your region server log? What does it say around that >>>> time? >>>> >>>> Thx >>>> >>>> J-D >>>> >>>> On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall < >>>> [email protected] >>>> >>>>> wrote: >>>>> >>>> >>>> I've been testing a nuch on HBase and I'm running into an issue with it >>>> >>>>> when it's been running for a while. >>>>> >>>>> I'm running hbase 0.19 on a single local host pseudo distributed hadoop >>>>> 0.19 cluster. >>>>> >>>>> After some time I get the following in my logs: >>>>> >>>>> 2009-02-24 09:31:18,664 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:31:18,664 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:31:24,293 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:31:24,293 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:31:27,217 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:31:27,217 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:31:55,533 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:32:18,665 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:32:35,749 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:32:38,957 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:32:38,957 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:33:18,771 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:33:54,889 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:33:57,387 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:34:41,579 INFO >>>>> org.apache.hadoop.hbase.master.ServerManager: >>>>> 127.0.0.1:40076 lease expired >>>>> 2009-02-24 09:34:53,440 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:34:53,441 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:34:53,441 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:35:38,720 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:35:38,720 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:35:56,951 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:36:05,667 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>> MSG_CALL_SERVER_STARTUP >>>>> 2009-02-24 09:36:05,668 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:36:05,884 INFO >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown >>>>> of >>>>> server 127.0.0.1:40076: logSplit: false, rootRescanned: false, >>>>> numberOfMetaRegions: 1, >>>>> onlineMetaRegions.size(): 1 >>>>> 2009-02-24 09:36:17,689 INFO org.apache.hadoop.hbase.regionserver.HLog: >>>>> Splitting 1 log(s) in >>>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076 >>>>> 2009-02-24 09:36:25,034 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 >>>>> 076} complete >>>>> 2009-02-24 09:36:25,034 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:36:36,598 INFO org.apache.hadoop.hbase.regionserver.HLog: >>>>> log >>>>> file splitting completed for >>>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076 >>>>> 2009-02-24 09:36:56,953 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:37:05,279 ERROR >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >>>>> at >>>>> >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) >>>>> 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC >>>>> Server >>>>> handler 4 on 40076, call openScanner([...@1ac5a78, [...@6799dc, >>>>> [...@1a66cfc, >>>>> 9223372036854775807, null) from 127.0.0.1 >>>>> :50139: error: org.apache.hadoop.hbase.NotServingRegionException: >>>>> .META.,,1 >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >>>>> at >>>>> >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) >>>>> 2009-02-24 09:37:07,051 WARN >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server: >>>>> 127.0.0.1:40076} >>>>> org.apache.hadoop.hbase.NotServingRegionException: >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >>>>> at >>>>> >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) >>>>> >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> >>>>> >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) >>>>> at >>>>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137) >>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:65) >>>>> 2009-02-24 09:37:07,230 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> All 1 .META. region(s) scanned >>>>> 2009-02-24 09:37:50,642 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>> Closed .META.,,1 >>>>> 2009-02-24 09:37:56,955 INFO >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1, >>>>> startKey: <>, server: 127.0.0.1:40076} >>>>> 2009-02-24 09:37:57,126 ERROR >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >>>>> at >>>>> >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) >>>>> 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC >>>>> Server >>>>> handler 2 on 40076, call openScanner([...@155bf4e, [...@82b436, >>>>> [...@1056757, >>>>> 9223372036854775807, null) from 127.0.0.1 >>>>> :35804: error: org.apache.hadoop.hbase.NotServingRegionException: >>>>> .META.,,1 >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >>>>> at >>>>> >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) >>>>> 2009-02-24 09:38:00,412 WARN >>>>> org.apache.hadoop.hbase.master.BaseScanner: >>>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server: >>>>> 127.0.0.1:40076} >>>>> org.apache.hadoop.hbase.NotServingRegionException: >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) >>>>> at >>>>> >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) >>>>> >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> >>>>> >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >>>>> at >>>>> >>>>> >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) >>>>> at >>>>> >>>>> >>>>> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) >>>>> at >>>>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137) >>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:65) >>>>> >>>>> This just repeats forever, until I restart HBase. >>>>> >>>>> Any help would be greatly appreciated. >>>>> Andrew >>>>> >>>>> >>>>> >>> >
