Andrew, For some reason, your region server shut down around 9:36:
2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase. > > master.RegionServerOperation: process shutdown of server 127.0.0.1:40076: > logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, But just before that the master received a startup message from it. Did you take a look at your region server log? What does it say around that time? Thx J-D On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <[email protected]>wrote: > I've been testing a nuch on HBase and I'm running into an issue with it > when it's been running for a while. > > I'm running hbase 0.19 on a single local host pseudo distributed hadoop > 0.19 cluster. > > After some time I get the following in my logs: > > 2009-02-24 09:31:18,664 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:31:18,664 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:31:24,293 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: > -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:31:24,293 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:31:27,217 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: > .META.,,1, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:31:27,217 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:31:55,533 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: > -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:32:18,665 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:32:35,749 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:32:38,957 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: > .META.,,1, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:32:38,957 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:33:18,771 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:33:54,889 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: > -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:33:57,387 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:34:41,579 INFO org.apache.hadoop.hbase.master.ServerManager: > 127.0.0.1:40076 lease expired > 2009-02-24 09:34:53,440 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: > .META.,,1, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:34:53,441 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:34:53,441 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:35:38,720 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: > .META.,,1, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:35:38,720 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:35:56,951 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:36:05,667 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP > 2009-02-24 09:36:05,668 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scan of 1 row(s) of meta region {regionname: > -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:36:05,884 INFO > org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of > server 127.0.0.1:40076: logSplit: false, rootRescanned: false, > numberOfMetaRegions: 1, > onlineMetaRegions.size(): 1 > 2009-02-24 09:36:17,689 INFO org.apache.hadoop.hbase.regionserver.HLog: > Splitting 1 log(s) in > hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076 > 2009-02-24 09:36:25,034 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scan of 2 row(s) of meta region {regionname: > .META.,,1, startKey: <>, server: 127.0.0.1:40 > 076} complete > 2009-02-24 09:36:25,034 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:36:36,598 INFO org.apache.hadoop.hbase.regionserver.HLog: log > file splitting completed for > hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076 > 2009-02-24 09:36:56,953 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:37:05,279 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 40076, call openScanner([...@1ac5a78, [...@6799dc, [...@1a66cfc, > 9223372036854775807, null) from 127.0.0.1 > :50139: error: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > 2009-02-24 09:37:07,051 WARN org.apache.hadoop.hbase.master.BaseScanner: > Scan one META region: {regionname: .META.,,1, startKey: <>, server: > 127.0.0.1:40076} > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95) > at > org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185) > at > org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) > at > org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) > at > org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137) > at org.apache.hadoop.hbase.Chore.run(Chore.java:65) > 2009-02-24 09:37:07,230 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 1 .META. region(s) scanned > 2009-02-24 09:37:50,642 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed .META.,,1 > 2009-02-24 09:37:56,955 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.metaScanner scanning meta region {regionname: .META.,,1, > startKey: <>, server: 127.0.0.1:40076} > 2009-02-24 09:37:57,126 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 40076, call openScanner([...@155bf4e, [...@82b436, [...@1056757, > 9223372036854775807, null) from 127.0.0.1 > :35804: error: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > 2009-02-24 09:38:00,412 WARN org.apache.hadoop.hbase.master.BaseScanner: > Scan one META region: {regionname: .META.,,1, startKey: <>, server: > 127.0.0.1:40076} > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699) > at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95) > at > org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185) > at > org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73) > at > org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129) > at > org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137) > at org.apache.hadoop.hbase.Chore.run(Chore.java:65) > > This just repeats forever, until I restart HBase. > > Any help would be greatly appreciated. > Andrew >
