Don't feel bad, I think we all messed up our first HBase setup. Did you delete /hbase while HBase was running? If so, first shut it down/kill -9, clear out the folder and the the Master will take care of recreating the ROOT and META on restart.
J-D On Tue, Jan 12, 2010 at 6:03 PM, steven zhuang <[email protected]> wrote: > hi, Jean. > Thanks a lot. > I am really an idiot of Hbase. > I removed the /hbase root directory from HDFS once, hoping it > will rebuild the whole META-regions thing. Then I found the exception is > still there every time I use the shell command. > Before all that I am gonna ask, I have one question :"Is it OK if > we run hbase shell command on any slave/region server? > I have checked the log, seems the master will request the wrong > regionserver for a region it's not servicing: > > 2010-01-12 20:25:11,996 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60020, call getRegionInfo([...@dc9766) from 192.168.1.98:55351: > error: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0 > org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2309) > > I am still analyzing the master log, for the most recent start, > there seems no exception records in the log. > > > > > On Wed, Jan 13, 2010 at 9:39 AM, Jean-Daniel Cryans > <[email protected]>wrote: > >> It seems it found the ROOT region but META wasn't assigned. Either you >> didn't wait enough after starting hbase or you should look at the >> master's log for the reason why that region wasn't assigned. >> >> J-D >> >> On Tue, Jan 12, 2010 at 5:36 PM, steven zhuang <[email protected]> >> wrote: >> > That's done, thanks, Jean. >> > >> > But now there is another problem. Now I can start the cluster >> > without any exception(good!), but at any node, when I run list/create, I >> > always get this exception, although afterwards I checked the table is >> > created. >> > >> > 10/01/12 20:25:16 DEBUG client.HConnectionManager$TableServers: Found >> ROOT >> > at 192.168.1.104:60020 >> > 10/01/12 20:25:16 DEBUG client.HConnectionManager$TableServers: >> > locateRegionInMeta attempt 0 of 5 failed; retrying after sleep of 2000 >> > org.apache.hadoop.hbase.client.NoServerForRegionException: No server >> address >> > listed in -ROOT- for region .META.,,1 >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:668) >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:590) >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:563) >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:407) >> > >> > >> > >> > On Wed, Jan 13, 2010 at 8:57 AM, Jean-Daniel Cryans <[email protected] >> >wrote: >> > >> >> Just make sure your OS doesn't resolve itself as 127.0.0.1, usual >> >> suspect if you are using ubuntu is to look at /etc/hosts and make sure >> >> your hostname resolves to your IP. >> >> >> >> J-D >> >> >> >> On Tue, Jan 12, 2010 at 4:52 PM, steven zhuang <[email protected] >> > >> >> wrote: >> >> > thanks, Jean, >> >> > I figured out that, in the netstat's output I can see >> >> > 127.0.0.1:60000, I don't know if this means it only listen on >> connection >> >> > request from the same machine. >> >> > About the hbase.master configuration, is there >> anything >> >> I >> >> > can use to replace it? >> >> > >> >> > >> >> > On Wed, Jan 13, 2010 at 1:36 AM, Jean-Daniel Cryans < >> [email protected] >> >> >wrote: >> >> > >> >> >> > 10/01/11 21:16:46 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode >> >> >> /hbase/master >> >> >> > got 127.0.1.1:60000 >> >> >> >> >> >> This means that your master registered itself in Zookeeper as >> >> >> 127.0.0.1, you seem to have a network configuration problem. >> >> >> >> >> >> Also the hbase.master configuration is deprecated and unused. >> >> >> >> >> >> J-D >> >> >> >> >> >> On Tue, Jan 12, 2010 at 6:16 AM, steven zhuang < >> [email protected] >> >> > >> >> >> wrote: >> >> >> > hello, list, >> >> >> > >> >> >> > I am now setting up a HBase cluster using HBase >> version >> >> >> > 0.20.2. But I have met some problems which I googled a lot and got >> no >> >> >> > answer. >> >> >> > Please help me. >> >> >> > >> >> >> > I modified the Hbase-site.xml and copy the whole >> >> directory >> >> >> to >> >> >> > another machine. >> >> >> > Using one as the master, after I started the hbase >> >> server, I >> >> >> > CAN see Hmaster / HQuorumPeer / HRegionServer running on Master >> >> >> > and HQuorumPeer / HRegionServer running on the slave node. >> >> >> > Here is what's weird: >> >> >> > I can enter the hbase shell on master node, but on the >> >> other >> >> >> > region server I cannot execute any command, a "list" command would >> >> cause >> >> >> a >> >> >> > list of exception. >> >> >> > >> >> >> > 10/01/11 21:16:46 DEBUG client.HConnectionManager$ClientZKWatcher: >> Got >> >> >> > ZooKeeper event, state: SyncConnected, type: None, path: null >> >> >> > 10/01/11 21:16:46 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode >> >> >> /hbase/master >> >> >> > got 127.0.1.1:60000 >> >> >> > 10/01/11 21:16:46 INFO client.HConnectionManager$TableServers: >> >> getMaster >> >> >> > attempt 0 of 5 failed; retrying after sleep of 2000 >> >> >> > java.net.ConnectException: Connection refused >> >> >> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> >> >> > at >> >> >> > >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >> >> >> > at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) >> >> >> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) >> >> >> > >> >> >> > I can create table in the master node's Hbase shell, but >> >> there >> >> >> > sometime is some exception like: >> >> >> > 10/01/12 06:08:15 DEBUG client.HConnectionManager$TableServers: >> >> >> > locateRegionInMeta attempt 2 of 5 failed; retrying after sleep of >> 2000 >> >> >> > org.apache.hadoop.hbase.client.NoServerForRegionException: No >> server >> >> >> address >> >> >> > listed in .META. for region t3,,1263305290760 >> >> >> > at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:668) >> >> >> > at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:594) >> >> >> > at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:557) >> >> >> > >> >> >> > But after this I can use list to see that the table HAS BEEN >> >> BUILT >> >> >> > inside the hdfs. >> >> >> > >> >> >> > the Hbase-site.xml I used: >> >> >> > >> >> >> >> >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> > <configuration> >> >> >> > <property> >> >> >> > <name>hbase.rootdir</name> >> >> >> > <value>hdfs://sz:8998/hbase</value> >> >> >> > </property> >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.cluster.distributed</name> >> >> >> > <value>true</value> >> >> >> > </property> >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.master</name> >> >> >> > <value>sz:60000</value> >> >> >> > </property> >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.tmp.dir</name> >> >> >> > <value>/home/steven/data/hbase-${user.name}</value> >> >> >> > </property> >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.zookeeper.property.dataDir</name> >> >> >> > <value>${hbase.tmp.dir}/zookeeper</value> >> >> >> > </property> >> >> >> > >> >> >> > <!-- >> >> >> > The following three properties are used together to create the >> list >> >> of >> >> >> > host:peer_port:leader_port quorum servers for ZooKeeper. >> >> >> > --> >> >> >> > <property> >> >> >> > <name>hbase.zookeeper.quorum</name> >> >> >> > <value>sz,hadoop3</value> >> >> >> > </property> >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.zookeeper.peerport</name> >> >> >> > <value>2888</value> >> >> >> > </property> >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.zookeeper.leaderport</name> >> >> >> > <value>3888</value> >> >> >> > </property> >> >> >> > >> >> >> >> >> >> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> >> > >> >> >> > >> >> >> > -- >> >> >> > best wishes. >> >> >> > steven >> >> >> > >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > best wishes. >> >> > steven >> >> > >> >> >> > >> > >> > >> > -- >> > best wishes. >> > steven >> > >> > > > > -- > best wishes. > steven >
