Re: Zookeeper available but no active master location found

2012-04-16 Thread Henri Pipe
Still having the same problem:

Here is the master log

2012-04-16 15:26:32,717 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
ZooKeeper available but no active master location found
2012-04-16 15:26:32,718 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
getMaster attempt 9 of 10 failed; no more retrying.
org.apache.hadoop.hbase.MasterNotRunningException
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:564)
at org.apache.hadoop.hbase.client.HBaseAdmin.init(HBaseAdmin.java:95)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:55)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

and here is what it says from zookeeper shell

[zk: localhost:2181(CONNECTED) 0] ls /hbase
[splitlog, unassigned, root-region-server, rs, table, master, shutdown]
[zk: localhost:2181(CONNECTED) 1] get /hbase/master
ip-10-251-27-130.ec2.internal:6
cZxid = 0xd0025
ctime = Mon Apr 16 15:25:18 EDT 2012
mZxid = 0xd0025
mtime = Mon Apr 16 15:25:18 EDT 2012
pZxid = 0xd0025
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x136bc97498d0003
dataLength = 35
numChildren = 0

and here is my /etc/hosts

[root@ip-10-251-27-130 bin]# cat /etc/hosts
127.0.0.1   localhost   localhost.localdomain
10.250.9.220ip-10-250-9-220 zoo1
10.251.110.50ip-10-251-110-50 zoo2
10.250.54.148ip-10-250-54-148 datanode
10.251.27.130   ip-10-251-27-130 namenode ip-10-251-27-130.ec2.internal

I run zookeepers on namenode, zoo1 and zoo2

Thanks

Henri Pipe


On Fri, Apr 13, 2012 at 1:01 PM, Stack st...@duboce.net wrote:

 What do you see in the master log?
 St.Ack

 On Fri, Apr 13, 2012 at 11:00 AM, Henri Pipe henri.p...@gmail.com wrote:
  I had tried zkCli (ls /hbase and get /hbase/master) , but it returns the
  correct value.
 
  [zk: localhost:2181(CONNECTED) 2] get /hbase/master
  ip-10-251-27-130:6
  cZxid = 0xa0032
  ctime = Thu Apr 12 20:03:23 EDT 2012
  mZxid = 0xa0032
  mtime = Thu Apr 12 20:03:23 EDT 2012
  pZxid = 0xa0032
 
  Also, I do have the namenode listed in my config
 
  Here is my hbase-site.xml file:
 
  configuration
   property
 namehbase.rootdir/name
 valuehdfs://namenode:9000/hbase/value
   /property
 
  Henri Pipe
 
 
  On Fri, Apr 13, 2012 at 1:58 AM, N Keywal nkey...@gmail.com wrote:
 
  Hi,
 
  Literally, it means that ZooKeeper is there but the hbase client can't
 find
  the hbase master address in it.
  By default, the node used is /hbase/master, and it contains the hostname
  and port of the master.
 
  You can check its content in ZK by doing a get /hbase/master in
  bin/zkCli.sh (see
 
 
 http://zookeeper.apache.org/doc/r3.4.3/zookeeperStarted.html#sc_ConnectingToZooKeeper
  ).
 
  There should be a root cause for this, so it worths looking for other
 error
  messages in the logs (master especially).
 
  N.
 
  On Fri, Apr 13, 2012 at 1:23 AM, Henri Pipe henri.p...@gmail.com
 wrote:
 
   client.HConnectionManager$HConnectionImplementation: ZooKeeper
 available
   but no active master location found
  
   Having a problem with master startup that I have not seen before.
  
   running the following packages:
  
   hadoop-hbase-0.90.4+49.137-1
   hadoop-0.20-secondarynamenode-0.20.2+923.197-1

Zookeeper available but no active master location found

2012-04-12 Thread Henri Pipe
client.HConnectionManager$HConnectionImplementation: ZooKeeper available
but no active master location found

Having a problem with master startup that I have not seen before.

running the following packages:

hadoop-hbase-0.90.4+49.137-1
hadoop-0.20-secondarynamenode-0.20.2+923.197-1
hadoop-hbase-thrift-0.90.4+49.137-1
hadoop-zookeeper-3.3.4+19.3-1
hadoop-0.20-datanode-0.20.2+923.197-1
hadoop-0.20-namenode-0.20.2+923.197-1
hadoop-0.20-tasktracker-0.20.2+923.197-1
hadoop-hbase-regionserver-0.90.4+49.137-1
hadoop-zookeeper-server-3.3.4+19.3-1
hadoop-0.20-0.20.2+923.197-1
hadoop-0.20-jobtracker-0.20.2+923.197-1
hadoop-hbase-master-0.90.4+49.137-1
[root@ip-10-251-27-130 logs]# java -version
java version 1.6.0_31
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)

I start master and region server on another node.

Master is initialized, but as soon as I try to check the master_status or
do a zkdump via web interface, it blows up with:

2012-04-12 19:16:10,453 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
ZooKeeper available but no active master location found
2012-04-12 19:16:10,453 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
getMaster attempt 10 of 10 failed; retrying after sleep of 16000

I am running three zookeepers:

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/mnt/zookeeper
# The maximum number of zookeeper client connections
maxClientCnxns=2000
# the port at which the clients will connect
clientPort=2181
server.1=10.251.27.130:2888:3888
server.2=10.250.9.220:2888:3888
server.3=10.251.110.50:2888:3888

I can telnet to the zookeepers just fine.

Here is my hbase-site.xml file:

configuration
  property
namehbase.rootdir/name
valuehdfs://namenode:9000/hbase/value
  /property
  property
namehbase.cluster.distributed/name
valuetrue/value
  /property
property
namehbase.zookeeper.quorum/name
value10.251.27.130,10.250.9.220,10.251.110.50/value
/property
property
namehbase.zookeeper.property.dataDir/name
value/hadoop/zookeeper/data/value
/property
property
namehbase.zookeeper.property.maxClientCnxns/name
value2000/value
finaltrue/final
/property
/configuration

Any thoughts? Any help is greatly appreciated.

Thanks

Henri Pipe