Re: Hbase stops working

Thu, 26 Feb 2009 10:23:24 -0800

What is a rough sort of minimum of cores or machines I should be looking at for a deployment for development purposes? Also where would I be best putting everything?
Would a two or three single coremachine cluster with be enough to run everything? Should I split the jobtracker, namenode and hbase master on the three machines in the cluster? Or would I be better off getting one bigger multicore machine as a jobtracker, namenode and master to use for development then add others later? 


Given what I've learned, it looks like I'll be spending the next few  
days moving to EC2, and continuing my development there because I  
think trying to continue on this machine is probably a lost cause and  
waste of time. :)


Thanks for all the help and feedback everyone.

Andrew

On 26 Feb 2009, at 17:20, Jim Kellerman (POWERSET) wrote:


With a single core machine, I'm amazed it works at all. Between the
namenode, data node(s), job tracker, task tracker, hbase master and
regionserver(s), that's a whole lot of threads (on the order of
several hundred). Thread starvation is a real possibility as the
jvm makes no guarantees about which runnable thread gets the cpu next.

---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)


-----Original Message-----

From: [email protected] [mailto:[email protected]] On Behalf Of  
Jean-

Daniel Cryans
Sent: Thursday, February 26, 2009 8:01 AM
To: [email protected]
Subject: Re: Hbase stops working

Andrew,


At 512MB I'm sure it swapped, tho it would be hard to tell now  
unless you
installed some monitoring software. Next time, you should at least  
monitor

your job using top to make sure it's the root cause of the problem.

Something that may help you is in

http://wiki.apache.org/hadoop/Hbase/Troubleshooting#7 mainly the  
third
bullet point in Resolution (adding -XX:+UseConcMarkSweepGC in hbase- 
env.sh

in the HBASE_OPTS and uncommenting it).

J-D

On Thu, Feb 26, 2009 at 10:14 AM, Andrew McCall
<[email protected]>wrote:


Hi J-D,


I'm running the HBase 0.19 release downloaded to the website on a  
pretty
lacklustre linux machine; 1.6Ghz single core processor, 512MB ram  
with

1.5GB

swap space enabled and 80GB hard disk. It's a testing machine,  
I've been

using for development. It's running Hadoop core 0.19 in pseudo

distributed

mode and running a map/reduce reduce crawl using nutch and HBase.  
I've

been

planning to move to an EC3 cluster in the near future but I've been

putting

it off since everything seemed to come up and work on the testing  
box.


It's entirely possible that it is a result of a swap, how do I tell?

I've

looked in all the logs for hadoop, hbase and my job and can't see  
that

anything stopped logging around then, the logs all have entries for

9:34.

I've also increased my lease timeout from the default to 600,000  
ms in

my

hbase-site, which is below as a result of another issue. I did  
search of

the

mailing list for the other issue and answers suggested that the  
lease

timeout should be the same as my map/reduce job timeout.

<configuration>
      <property>
          <name>hbase.rootdir</name>
          <value>hdfs://localhost:9000/hbase</value>
          <description>The directory shared by region servers.
          </description>
        </property>
        <property>
          <name>hbase.regionserver.lease.period</name>
          <value>600000</value>
        </property>
</configuration>

Thanks for the help,
Andrew



On 26 Feb 2009, at 13:41, Jean-Daniel Cryans wrote:

Andrew,


Oh yes sorry, your setup is in local mode so no region server in a
different
thread. My bad!


That lease timeout probably means that you were swapping and a  
full jvm
garbage collecting occurred which blocked the main thread. Can  
you make

sure
that it did swap at that moment?


It would also be helpful to get the usual info like hardware, OS,  
hbase

version, etc.

Thx!

J-D

On Thu, Feb 26, 2009 at 8:35 AM, Andrew McCall

<[email protected]

wrote:


Hi J-D,


I did check the region server logs and they didn't seem to be very

useful

at all:


Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on  
betty.goroam.net

ulimit -n 1024
2009-02-23 22:11:58,817 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a
distinct
region server because hbase.master is set to 'local' mode

Is all that is in them, once for each time I started HBase.

I saw the shutdown message, which was preceded by this:

2009-02-24 09:34:41,579 INFO
org.apache.hadoop.hbase.master.ServerManager:
127.0.0.1:40076 lease expired


I can't find any evidence it's related to my map/reduce task,  
it's the

only
one in the log files and it's suspiciously just before the process
shutdown

message - but, I'm new to this and honestly don't know what to  
make of

it.

Andrew


On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote:

Andrew,



For some reason, your region server shut down around 9:36:

2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase.



master.RegionServerOperation: process shutdown of server
127.0.0.1:40076
:
logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,





But just before that the master received a startup message from  
it.

Did

you

take a look at your region server log? What does it say around  
that

time?

Thx

J-D

On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <
[email protected]


wrote:




I've been testing a nuch on HBase and I'm running into an issue  
with

it



when it's been running for a while.

I'm running hbase 0.19 on a single local host pseudo distributed

hadoop

0.19 cluster.

After some time I get the following in my logs:

2009-02-24 09:31:18,664 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:31:18,664 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:31:24,293 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region

{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:31:24,293 INFO
org.apache.hadoop.hbase.master.BaseScanner:

RegionManager.rootScanner scanning meta region {regionname: - 
ROOT-

,,0,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:31:27,217 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region

{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:31:27,217 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:31:55,533 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region

{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:32:18,665 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:32:35,749 INFO
org.apache.hadoop.hbase.master.BaseScanner:

RegionManager.rootScanner scanning meta region {regionname: - 
ROOT-

,,0,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:32:38,957 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region

{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:32:38,957 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:33:18,771 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:33:54,889 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region

{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:33:57,387 INFO
org.apache.hadoop.hbase.master.BaseScanner:

RegionManager.rootScanner scanning meta region {regionname: - 
ROOT-

,,0,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:34:41,579 INFO
org.apache.hadoop.hbase.master.ServerManager:
127.0.0.1:40076 lease expired
2009-02-24 09:34:53,440 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region

{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:34:53,441 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:34:53,441 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:35:38,720 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region

{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:35:38,720 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:35:56,951 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:36:05,667 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
MSG_CALL_SERVER_STARTUP
2009-02-24 09:36:05,668 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region

{regionname:

-ROOT-,,0, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:36:05,884 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process

shutdown

of
server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1
2009-02-24 09:36:17,689 INFO

org.apache.hadoop.hbase.regionserver.HLog:

Splitting 1 log(s) in
hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
2009-02-24 09:36:25,034 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2 row(s) of meta region

{regionname:

.META.,,1, startKey: <>, server: 127.0.0.1:40
076} complete
2009-02-24 09:36:25,034 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:36:36,598 INFO

org.apache.hadoop.hbase.regionserver.HLog:

log
file splitting completed for
hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
2009-02-24 09:36:56,953 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:37:05,279 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed

openScanner

org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer

.java:2065)

  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ

er.java:1699)

  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown  
Source)

  at




sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm

pl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)
  at

org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 
632)

  at



org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)

2009-02-24 09:37:06,102 INFO  
org.apache.hadoop.ipc.HBaseServer: IPC

Server
handler 4 on 40076, call openScanner([...@1ac5a78, [...@6799dc,
[...@1a66cfc,
9223372036854775807, null) from 127.0.0.1

:50139: error:  
org.apache.hadoop.hbase.NotServingRegionException:

.META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer

.java:2065)

  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ

er.java:1699)

  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown  
Source)

  at




sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm

pl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)
  at

org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 
632)

  at



org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)

2009-02-24 09:37:07,051 WARN
org.apache.hadoop.hbase.master.BaseScanner:

Scan one META region: {regionname: .META.,,1, startKey: <>,  
server:

127.0.0.1:40076}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer

.java:2065)

  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ

er.java:1699)

  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown  
Source)

  at




sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm

pl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)
  at

org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 
632)

  at



org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)



  at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native

Method)
  at




sun 
.reflect 
.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc

essorImpl.java:39)

  at




sun 
.reflect 
.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr

uctorAccessorImpl.java:27)

  at

java.lang.reflect.Constructor.newInstance(Constructor.java:513)

  at




org 
.apache 
.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remot

eExceptionHandler.java:95)

  at




org 
.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java: 
185

)

  at




org 
.apache 
.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.j

ava:73)

  at




org 
.apache 
.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.jav

a:129)

  at


org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java: 
137)

  at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
2009-02-24 09:37:07,230 INFO
org.apache.hadoop.hbase.master.BaseScanner:
All 1 .META. region(s) scanned
2009-02-24 09:37:50,642 INFO
org.apache.hadoop.hbase.regionserver.HRegion:
Closed .META.,,1
2009-02-24 09:37:56,955 INFO
org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname:

.META.,,1,

startKey: <>, server: 127.0.0.1:40076}
2009-02-24 09:37:57,126 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Failed

openScanner

org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer

.java:2065)

  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ

er.java:1699)

  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown  
Source)

  at




sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm

pl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)
  at

org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 
632)

  at



org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)

2009-02-24 09:37:59,129 INFO  
org.apache.hadoop.ipc.HBaseServer: IPC

Server
handler 2 on 40076, call openScanner([...@155bf4e, [...@82b436,
[...@1056757,
9223372036854775807, null) from 127.0.0.1

:35804: error:  
org.apache.hadoop.hbase.NotServingRegionException:

.META.,,1
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer

.java:2065)

  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ

er.java:1699)

  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown  
Source)

  at




sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm

pl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)
  at

org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 
632)

  at



org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)

2009-02-24 09:38:00,412 WARN
org.apache.hadoop.hbase.master.BaseScanner:

Scan one META region: {regionname: .META.,,1, startKey: <>,  
server:

127.0.0.1:40076}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer

.java:2065)

  at




org 
.apache 
.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServ

er.java:1699)

  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown  
Source)

  at




sun 
.reflect 
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm

pl.java:25)

  at java.lang.reflect.Method.invoke(Method.java:597)
  at

org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java: 
632)

  at



org.apache.hadoop.hbase.ipc.HBaseServer 
$Handler.run(HBaseServer.java:895)



  at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native

Method)
  at




sun 
.reflect 
.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc

essorImpl.java:39)

  at




sun 
.reflect 
.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr

uctorAccessorImpl.java:27)

  at

java.lang.reflect.Constructor.newInstance(Constructor.java:513)

  at




org 
.apache 
.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remot

eExceptionHandler.java:95)

  at




org 
.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java: 
185

)

  at




org 
.apache 
.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.j

ava:73)

  at




org 
.apache 
.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.jav

a:129)

  at


org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java: 
137)

  at org.apache.hadoop.hbase.Chore.run(Chore.java:65)

This just repeats forever, until I restart HBase.

Any help would be greatly appreciated.
Andrew































					Reply via email to