Re: Hbase stops working

Thu, 26 Feb 2009 05:35:47 -0800 

Hi J-D,
I did check the region server logs and they didn't seem to be very useful at all: 

Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on betty.goroam.net
ulimit -n 1024

2009-02-23 22:11:58,817 WARN  
org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a  
distinct region server because hbase.master is set to 'local' mode


Is all that is in them, once for each time I started HBase.

I saw the shutdown message, which was preceded by this:


2009-02-24 09:34:41,579 INFO  
org.apache.hadoop.hbase.master.ServerManager: 127.0.0.1:40076 lease  
expired



I can't find any evidence it's related to my map/reduce task, it's the  
only one in the log files and it's suspiciously just before the  
process shutdown message - but, I'm new to this and honestly don't  
know what to make of it.


Andrew

On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote:


Andrew,

For some reason, your region server shut down around 9:36:

2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase.



master.RegionServerOperation: process shutdown of server  
127.0.0.1:40076:

logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,




But just before that the master received a startup message from it.  
Did you
take a look at your region server log? What does it say around that  
time?


Thx

J-D


On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <[email protected] 
>wrote:



I've been testing a nuch on HBase and I'm running into an issue  
with it

when it's been running for a while.


I'm running hbase 0.19 on a single local host pseudo distributed  
hadoop

0.19 cluster.

After some time I get the following in my logs:


2009-02-24 09:31:18,664 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:31:18,664 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:31:24,293 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region  
{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:31:24,293 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,, 
0,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:31:27,217 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region  
{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:31:27,217 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:31:55,533 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region  
{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:32:18,665 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:32:35,749 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,, 
0,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:32:38,957 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region  
{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:32:38,957 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:33:18,771 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:33:54,889 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region  
{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:33:57,387 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,, 
0,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:34:41,579 INFO  
org.apache.hadoop.hbase.master.ServerManager:

127.0.0.1:40076 lease expired

2009-02-24 09:34:53,440 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region  
{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:34:53,441 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:34:53,441 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}

2009-02-24 09:35:38,720 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region  
{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:35:38,720 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:35:56,951 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:36:05,667 INFO

org.apache.hadoop.hbase.regionserver.HRegionServer:  
MSG_CALL_SERVER_STARTUP
2009-02-24 09:36:05,668 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region  
{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:36:05,884 INFO

org.apache.hadoop.hbase.master.RegionServerOperation: process  
shutdown of

server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1

2009-02-24 09:36:17,689 INFO  
org.apache.hadoop.hbase.regionserver.HLog:

Splitting 1 log(s) in
hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076

2009-02-24 09:36:25,034 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region  
{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete

2009-02-24 09:36:25,034 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:36:36,598 INFO  
org.apache.hadoop.hbase.regionserver.HLog: log

file splitting completed for
hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076

2009-02-24 09:36:56,953 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:37:05,279 ERROR

org.apache.hadoop.hbase.regionserver.HRegionServer: Failed  
openScanner

org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)

      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
1699)

      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
      at

sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)
      at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
      at

org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)
2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC  
Server
handler 4 on 40076, call openScanner([...@1ac5a78, [...@6799dc,  
[...@1a66cfc,

9223372036854775807, null) from 127.0.0.1

:50139: error:  
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1

org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)

      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
1699)

      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
      at

sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)
      at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
      at

org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)
2009-02-24 09:37:07,051 WARN  
org.apache.hadoop.hbase.master.BaseScanner:

Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)

      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
1699)

      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
      at

sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)
      at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
      at

org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)



      at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native

Method)
      at

sun 
.reflect 
.NativeConstructorAccessorImpl 
.newInstance(NativeConstructorAccessorImpl.java:39)

      at

sun 
.reflect 
.DelegatingConstructorAccessorImpl 
.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java: 
513)

      at

org 
.apache 
.hadoop 
.hbase 
.RemoteExceptionHandler 
.decodeRemoteException(RemoteExceptionHandler.java:95)

      at

org 
.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java: 
185)

      at

org 
.apache 
.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java: 
73)

      at

org 
.apache 
.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java: 
129)

      at

org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java: 
137)

      at org.apache.hadoop.hbase.Chore.run(Chore.java:65)

2009-02-24 09:37:07,230 INFO  
org.apache.hadoop.hbase.master.BaseScanner:

All 1 .META. region(s) scanned

2009-02-24 09:37:50,642 INFO  
org.apache.hadoop.hbase.regionserver.HRegion:

Closed .META.,,1

2009-02-24 09:37:56,955 INFO  
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,, 
1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:37:57,126 ERROR

org.apache.hadoop.hbase.regionserver.HRegionServer: Failed  
openScanner

org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)

      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
1699)

      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
      at

sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)
      at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
      at

org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)
2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC  
Server
handler 2 on 40076, call openScanner([...@155bf4e, [...@82b436,  
[...@1056757,

9223372036854775807, null) from 127.0.0.1

:35804: error:  
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1

org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)

      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
1699)

      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
      at

sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)
      at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
      at

org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)
2009-02-24 09:38:00,412 WARN  
org.apache.hadoop.hbase.master.BaseScanner:

Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:40076}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)

      at

org 
.apache 
.hadoop 
.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
1699)

      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
      at

sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)
      at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
      at

org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)



      at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native

Method)
      at

sun 
.reflect 
.NativeConstructorAccessorImpl 
.newInstance(NativeConstructorAccessorImpl.java:39)

      at

sun 
.reflect 
.DelegatingConstructorAccessorImpl 
.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java: 
513)

      at

org 
.apache 
.hadoop 
.hbase 
.RemoteExceptionHandler 
.decodeRemoteException(RemoteExceptionHandler.java:95)

      at

org 
.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java: 
185)

      at

org 
.apache 
.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java: 
73)

      at

org 
.apache 
.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java: 
129)

      at

org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java: 
137)

      at org.apache.hadoop.hbase.Chore.run(Chore.java:65)

This just repeats forever, until I restart HBase.

Any help would be greatly appreciated.
Andrew

























					Reply via email to