Still having the same problem: Here is the master log
2012-04-16 15:26:32,717 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: ZooKeeper available but no active master location found 2012-04-16 15:26:32,718 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: getMaster attempt 9 of 10 failed; no more retrying. org.apache.hadoop.hbase.MasterNotRunningException at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:564) at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:95) at org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:55) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) and here is what it says from zookeeper shell [zk: localhost:2181(CONNECTED) 0] ls /hbase [splitlog, unassigned, root-region-server, rs, table, master, shutdown] [zk: localhost:2181(CONNECTED) 1] get /hbase/master ip-10-251-27-130.ec2.internal:60000 cZxid = 0xd00000025 ctime = Mon Apr 16 15:25:18 EDT 2012 mZxid = 0xd00000025 mtime = Mon Apr 16 15:25:18 EDT 2012 pZxid = 0xd00000025 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x136bc97498d0003 dataLength = 35 numChildren = 0 and here is my /etc/hosts [root@ip-10-251-27-130 bin]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain 10.250.9.220 ip-10-250-9-220 zoo1 10.251.110.50 ip-10-251-110-50 zoo2 10.250.54.148 ip-10-250-54-148 datanode 10.251.27.130 ip-10-251-27-130 namenode ip-10-251-27-130.ec2.internal I run zookeepers on namenode, zoo1 and zoo2 Thanks Henri Pipe On Fri, Apr 13, 2012 at 1:01 PM, Stack <st...@duboce.net> wrote: > What do you see in the master log? > St.Ack > > On Fri, Apr 13, 2012 at 11:00 AM, Henri Pipe <henri.p...@gmail.com> wrote: > > I had tried zkCli (ls /hbase and get /hbase/master) , but it returns the > > correct value. > > > > [zk: localhost:2181(CONNECTED) 2] get /hbase/master > > ip-10-251-27-130:60000 > > cZxid = 0xa00000032 > > ctime = Thu Apr 12 20:03:23 EDT 2012 > > mZxid = 0xa00000032 > > mtime = Thu Apr 12 20:03:23 EDT 2012 > > pZxid = 0xa00000032 > > > > Also, I do have the namenode listed in my config > > > > Here is my hbase-site.xml file: > > > > <configuration> > > <property> > > <name>hbase.rootdir</name> > > <value>hdfs://namenode:9000/hbase</value> > > </property> > > > > Henri Pipe > > > > > > On Fri, Apr 13, 2012 at 1:58 AM, N Keywal <nkey...@gmail.com> wrote: > > > >> Hi, > >> > >> Literally, it means that ZooKeeper is there but the hbase client can't > find > >> the hbase master address in it. > >> By default, the node used is /hbase/master, and it contains the hostname > >> and port of the master. > >> > >> You can check its content in ZK by doing a "get /hbase/master" in > >> bin/zkCli.sh (see > >> > >> > http://zookeeper.apache.org/doc/r3.4.3/zookeeperStarted.html#sc_ConnectingToZooKeeper > >> ). > >> > >> There should be a root cause for this, so it worths looking for other > error > >> messages in the logs (master especially). > >> > >> N. > >> > >> On Fri, Apr 13, 2012 at 1:23 AM, Henri Pipe <henri.p...@gmail.com> > wrote: > >> > >> > "client.HConnectionManager$HConnectionImplementation: ZooKeeper > available > >> > but no active master location found" > >> > > >> > Having a problem with master startup that I have not seen before. > >> > > >> > running the following packages: > >> > > >> > hadoop-hbase-0.90.4+49.137-1 > >> > hadoop-0.20-secondarynamenode-0.20.2+923.197-1 > >> > hadoop-hbase-thrift-0.90.4+49.137-1 > >> > hadoop-zookeeper-3.3.4+19.3-1 > >> > hadoop-0.20-datanode-0.20.2+923.197-1 > >> > hadoop-0.20-namenode-0.20.2+923.197-1 > >> > hadoop-0.20-tasktracker-0.20.2+923.197-1 > >> > hadoop-hbase-regionserver-0.90.4+49.137-1 > >> > hadoop-zookeeper-server-3.3.4+19.3-1 > >> > hadoop-0.20-0.20.2+923.197-1 > >> > hadoop-0.20-jobtracker-0.20.2+923.197-1 > >> > hadoop-hbase-master-0.90.4+49.137-1 > >> > [root@ip-10-251-27-130 logs]# java -version > >> > java version "1.6.0_31" > >> > Java(TM) SE Runtime Environment (build 1.6.0_31-b04) > >> > Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) > >> > > >> > I start master and region server on another node. > >> > > >> > Master is initialized, but as soon as I try to check the > master_status or > >> > do a zkdump via web interface, it blows up with: > >> > > >> > 2012-04-12 19:16:10,453 INFO > >> > > >> > > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> > ZooKeeper available but no active master location found > >> > 2012-04-12 19:16:10,453 INFO > >> > > >> > > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> > getMaster attempt 10 of 10 failed; retrying after sleep of 16000 > >> > > >> > I am running three zookeepers: > >> > > >> > # The number of milliseconds of each tick > >> > tickTime=2000 > >> > # The number of ticks that the initial > >> > # synchronization phase can take > >> > initLimit=10 > >> > # The number of ticks that can pass between > >> > # sending a request and getting an acknowledgement > >> > syncLimit=5 > >> > # the directory where the snapshot is stored. > >> > dataDir=/mnt/zookeeper > >> > # The maximum number of zookeeper client connections > >> > maxClientCnxns=2000 > >> > # the port at which the clients will connect > >> > clientPort=2181 > >> > server.1=10.251.27.130:2888:3888 > >> > server.2=10.250.9.220:2888:3888 > >> > server.3=10.251.110.50:2888:3888 > >> > > >> > I can telnet to the zookeepers just fine. > >> > > >> > Here is my hbase-site.xml file: > >> > > >> > <configuration> > >> > <property> > >> > <name>hbase.rootdir</name> > >> > <value>hdfs://namenode:9000/hbase</value> > >> > </property> > >> > <property> > >> > <name>hbase.cluster.distributed</name> > >> > <value>true</value> > >> > </property> > >> > <property> > >> > <name>hbase.zookeeper.quorum</name> > >> > <value>10.251.27.130,10.250.9.220,10.251.110.50</value> > >> > </property> > >> > <property> > >> > <name>hbase.zookeeper.property.dataDir</name> > >> > <value>/hadoop/zookeeper/data</value> > >> > </property> > >> > <property> > >> > <name>hbase.zookeeper.property.maxClientCnxns</name> > >> > <value>2000</value> > >> > <final>true</final> > >> > </property> > >> > </configuration> > >> > > >> > Any thoughts? Any help is greatly appreciated. > >> > > >> > Thanks > >> > > >> > Henri Pipe > >> > > >> >