Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing
Hi Sridhar! Please be sure to copy your responses to user@hbase and not just an individual. The attached log shows that your master service does not see any Region Servers as available. You should check the logs for the region servers to see why they aren't starting. On Sat, Apr 4, 2015 at 11:37 AM, sridhararao mutluri drm...@hotmail.com wrote: Sean, I used mycluster/hbase in hbase-site.xml and no luck to me. I tried to use 0.98.4 with distributed mode and hadoop ha.Now new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again later at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. I tried to use only hostname instead of ip. I am attaching log for your personal observation and good suggestions. Thanks, Sridhar From: bus...@cloudera.com Date: Fri, 3 Apr 2015 11:34:24 -0500 Subject: Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing To: user@hbase.apache.org It should be property namehbase.rootdir/name valuehdfs://mycluster/hbase/value /property On Fri, Apr 3, 2015 at 11:19 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Sean, Thanks for your speedy reply. We are suing hadoop HA with namespace mycluster. Please suggest below property value like this: namehbase.rootdir/name valuehdfs://mycluster:8020/hbase/value /propertyThanks,Sridhar From: bus...@cloudera.com Date: Fri, 3 Apr 2015 11:11:42 -0500 Subject: Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing To: user@hbase.apache.org you should check the master's log for what is causing it to be delayed in starting. A few configuration errors: 1) zk dataDir in hbase configs property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property This property is not needed, because you have specified a ZK quorum and are in distributed mode. 2) HDFS address property namefs.defaultFS/name valuehdfs://mycluster/value /property Did you redact this, or is your HDFS set up in HA with the nameservice mycluster? If it's the latter, then this config is wrong: property namehbase.rootdir/name valuehdfs:// 10.1.252.86:8020/hbase/value /property This is presumably the address of one of your NameNodes. If you are using HA, then this should use the nameservice instead of a particular host. Otherwise, HBase will get errors whenever the named host is not active. 3) IP addresses In general, you should be using FQDN for hosts in your cluster and not IP addresses. If you are using IP addresses because DNS is problematic, please be aware that several parts of the cluster are going to attempt to use reverse DNS to turn that back into a host name. You can save yourself trouble by doing the work upfront to make sure forward and reverse DNS behaves properly (even if just with coordinated /etc/hosts files) On Fri, Apr 3, 2015 at 11:04 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, I am facing below error.Please suggest solution: hbase(main):001:0 create 'cars', 'vi' ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1869) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1874) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2067) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1262) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398
RE: Hbase 1.0 Distributed Mode:Issue:Master is initializing
Hi Sean, Thanks for your speedy reply. We are suing hadoop HA with namespace mycluster. Please suggest below property value like this: namehbase.rootdir/name valuehdfs://mycluster:8020/hbase/value /propertyThanks,Sridhar From: bus...@cloudera.com Date: Fri, 3 Apr 2015 11:11:42 -0500 Subject: Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing To: user@hbase.apache.org you should check the master's log for what is causing it to be delayed in starting. A few configuration errors: 1) zk dataDir in hbase configs property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property This property is not needed, because you have specified a ZK quorum and are in distributed mode. 2) HDFS address propertynamefs.defaultFS/name valuehdfs://mycluster/value /property Did you redact this, or is your HDFS set up in HA with the nameservice mycluster? If it's the latter, then this config is wrong: propertynamehbase.rootdir/namevaluehdfs:// 10.1.252.86:8020/hbase/value /property This is presumably the address of one of your NameNodes. If you are using HA, then this should use the nameservice instead of a particular host. Otherwise, HBase will get errors whenever the named host is not active. 3) IP addresses In general, you should be using FQDN for hosts in your cluster and not IP addresses. If you are using IP addresses because DNS is problematic, please be aware that several parts of the cluster are going to attempt to use reverse DNS to turn that back into a host name. You can save yourself trouble by doing the work upfront to make sure forward and reverse DNS behaves properly (even if just with coordinated /etc/hosts files) On Fri, Apr 3, 2015 at 11:04 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, I am facing below error.Please suggest solution: hbase(main):001:0 create 'cars', 'vi' ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1869) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1874) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2067) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1262) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) hbase-site.xml shows: configuration propertynamehbase.master/name value10.1.252.86:6/value /property property namehbase.rootdir/namevaluehdfs:// 10.1.252.86:8020/hbase/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name value10.1.252.86,10.1.252.87,10.1.252.88/value /property Hadoop core-site.xml: configurationpropertynamefs.defaultFS/name valuehdfs://mycluster/value/property property Zookeeper status:[hadoop@.. bin]$ ./zkServer.sh statusJMX enabled by defaultUsing config: /home/hadoop/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: follower Thanks,Sridhar -- Sean
Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing
you should check the master's log for what is causing it to be delayed in starting. A few configuration errors: 1) zk dataDir in hbase configs property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property This property is not needed, because you have specified a ZK quorum and are in distributed mode. 2) HDFS address propertynamefs.defaultFS/name valuehdfs://mycluster/value /property Did you redact this, or is your HDFS set up in HA with the nameservice mycluster? If it's the latter, then this config is wrong: propertynamehbase.rootdir/namevaluehdfs:// 10.1.252.86:8020/hbase/value /property This is presumably the address of one of your NameNodes. If you are using HA, then this should use the nameservice instead of a particular host. Otherwise, HBase will get errors whenever the named host is not active. 3) IP addresses In general, you should be using FQDN for hosts in your cluster and not IP addresses. If you are using IP addresses because DNS is problematic, please be aware that several parts of the cluster are going to attempt to use reverse DNS to turn that back into a host name. You can save yourself trouble by doing the work upfront to make sure forward and reverse DNS behaves properly (even if just with coordinated /etc/hosts files) On Fri, Apr 3, 2015 at 11:04 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, I am facing below error.Please suggest solution: hbase(main):001:0 create 'cars', 'vi' ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1869) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1874) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2067) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1262) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) hbase-site.xml shows: configuration propertynamehbase.master/name value10.1.252.86:6/value /property property namehbase.rootdir/namevaluehdfs:// 10.1.252.86:8020/hbase/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name value10.1.252.86,10.1.252.87,10.1.252.88/value /property Hadoop core-site.xml: configurationpropertynamefs.defaultFS/name valuehdfs://mycluster/value/property property Zookeeper status:[hadoop@.. bin]$ ./zkServer.sh statusJMX enabled by defaultUsing config: /home/hadoop/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: follower Thanks,Sridhar -- Sean
Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing
It should be property namehbase.rootdir/name valuehdfs://mycluster/hbase/value /property On Fri, Apr 3, 2015 at 11:19 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Sean, Thanks for your speedy reply. We are suing hadoop HA with namespace mycluster. Please suggest below property value like this: namehbase.rootdir/name valuehdfs://mycluster:8020/hbase/value /propertyThanks,Sridhar From: bus...@cloudera.com Date: Fri, 3 Apr 2015 11:11:42 -0500 Subject: Re: Hbase 1.0 Distributed Mode:Issue:Master is initializing To: user@hbase.apache.org you should check the master's log for what is causing it to be delayed in starting. A few configuration errors: 1) zk dataDir in hbase configs property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property This property is not needed, because you have specified a ZK quorum and are in distributed mode. 2) HDFS address propertynamefs.defaultFS/name valuehdfs://mycluster/value /property Did you redact this, or is your HDFS set up in HA with the nameservice mycluster? If it's the latter, then this config is wrong: propertynamehbase.rootdir/name valuehdfs:// 10.1.252.86:8020/hbase/value /property This is presumably the address of one of your NameNodes. If you are using HA, then this should use the nameservice instead of a particular host. Otherwise, HBase will get errors whenever the named host is not active. 3) IP addresses In general, you should be using FQDN for hosts in your cluster and not IP addresses. If you are using IP addresses because DNS is problematic, please be aware that several parts of the cluster are going to attempt to use reverse DNS to turn that back into a host name. You can save yourself trouble by doing the work upfront to make sure forward and reverse DNS behaves properly (even if just with coordinated /etc/hosts files) On Fri, Apr 3, 2015 at 11:04 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, I am facing below error.Please suggest solution: hbase(main):001:0 create 'cars', 'vi' ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:1869) at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1874) at org.apache.hadoop.hbase.master.HMaster.ensureNamespaceExists(HMaster.java:2067) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1262) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) hbase-site.xml shows: configuration propertynamehbase.master/name value10.1.252.86:6/value /property property namehbase.rootdir/namevaluehdfs:// 10.1.252.86:8020/hbase/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name value10.1.252.86,10.1.252.87,10.1.252.88/value /property Hadoop core-site.xml: configurationpropertynamefs.defaultFS/name valuehdfs://mycluster/value/property property Zookeeper status:[hadoop@.. bin]$ ./zkServer.sh statusJMX enabled by defaultUsing config: /home/hadoop/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: follower Thanks,Sridhar -- Sean -- Sean
