Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Sridhar, The logs from solix4 are very interesting: solix 4: 2015-04-08 22:48:21,308 INFO [regionserver60020-SendThread( solix1.abcsupply.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server solix1.abcsupply. com/10.1.252.86:2181, sessionid = 0x14c9c32c6fd0004, negotiated timeout = 4 ... 2015-04-08 22:48:21,530 INFO [regionserver60020] regionserver.HRegionServer: reportForDuty to master=solix1.abcsupply.com,6,1428551050776 with port=60020, startcode=1428551291142 2015-04-08 22:48:21,777 WARN [regionserver60020] regionserver.HRegionServer: error telling master we are up com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused Which mean there is a live master since we can get a znode from ZooKeeper (coincidentally zk connected to solix1 ZK server) and that znode is telling us that the master in solix1 but we cannot connect to the master. Have you verified if iptables or any other firewall rule is blocking the connectivity between the RSs and the master? thanks, esteban. -- Cloudera, Inc. On Fri, Apr 10, 2015 at 12:29 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I uploaded al lregion logs in below address.The Hregions are up on three nodes and i saw erros connection refused https://gist.github.com/drmsr6/7e40958db1ea364e3541 Thanks, Sridhar From: bus...@cloudera.com Date: Fri, 10 Apr 2015 02:02:07 -0500 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: drm...@hotmail.com CC: user@hbase.apache.org; este...@cloudera.com the master log shows that no region servers ever come online. can you upload the region server logs? On Fri, Apr 10, 2015 at 1:15 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, Please find master log url and suggest any solution: https://gist.github.com/drmsr6/a25de9e60bb6229f796e Thanks, Sridhar -- Date: Thu, 9 Apr 2015 07:46:40 -0500 Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: bus...@cloudera.com To: drm...@hotmail.com CC: user@hbase.apache.org; este...@cloudera.com Please post a link to updated master logs. -- Sean On Apr 9, 2015 5:50 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, We increased maximum files and with reboot ulimit -n shows across all clusters [hadoop@solix1 bin]$ ulimit -n 65535 But unluckily we are getting same error now also: hbase(main):001:0 create 'buses', 'vi' *ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again later* at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Do we downgrade JDK from 1.8 to 1.7 and validate.Please suggest. Thanks, Sridhar -- From: drm...@hotmail.com To: este...@cloudera.com Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase Date: Thu, 9 Apr 2015 06:44:10 +0530 Hi Esteban, Thanks much for your speedy suggestion.I will implement and will come back to you. Regards, Dr.Sridhar -- From: este...@cloudera.com Date: Wed, 8 Apr 2015 18:06:21 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: drm...@hotmail.com what i meant to say is that you should try to change the ulimit to 64k -- Cloudera, Inc. On Wed, Apr 8, 2015 at 6:06 PM, Esteban Gutierrez este...@cloudera.com wrote: cool... so the HBase master started, but you might want to bump your ulimits -n by 60 times, 1024 of max open files is too low of hbase. -- Cloudera, Inc. On Wed, Apr 8, 2015 at 5:55 PM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, Please find the link: https://gist.github.com/drmsr6/bcf439f09cac95b59eca I am waiting for final inputs.:) Thanks, Dr.Sridhar, 919573453136
RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Hi Esteban, We increased maximum files and with reboot ulimit -n shows across all clusters [hadoop@solix1 bin]$ ulimit -n65535 But unluckily we are getting same error now also: hbase(main):001:0 create 'buses', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735)at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774)at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Do we downgrade JDK from 1.8 to 1.7 and validate.Please suggest. Thanks,Sridhar From: drm...@hotmail.com To: este...@cloudera.com Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase Date: Thu, 9 Apr 2015 06:44:10 +0530 Hi Esteban, Thanks much for your speedy suggestion.I will implement and will come back to you. Regards, Dr.Sridhar From: este...@cloudera.com Date: Wed, 8 Apr 2015 18:06:21 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: drm...@hotmail.com what i meant to say is that you should try to change the ulimit to 64k--Cloudera, Inc. On Wed, Apr 8, 2015 at 6:06 PM, Esteban Gutierrez este...@cloudera.com wrote: cool... so the HBase master started, but you might want to bump your ulimits -n by 60 times, 1024 of max open files is too low of hbase. --Cloudera, Inc. On Wed, Apr 8, 2015 at 5:55 PM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, Please find the link: https://gist.github.com/drmsr6/bcf439f09cac95b59eca I am waiting for final inputs.:) Thanks,Dr.Sridhar,919573453136 From: este...@cloudera.com Date: Wed, 8 Apr 2015 11:54:04 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: drm...@hotmail.com Sridhar, what is the URL to the log file? thanks --Cloudera, Inc. On Tue, Apr 7, 2015 at 6:35 PM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, I pasted logs in github.com: Thanks,Sridhar From: este...@cloudera.com Date: Tue, 7 Apr 2015 08:44:55 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org Sridhar, What do you see in the HBase Master logs? The exception you are getting from the HBase Master is just a side effect and not the real cause? Is it possible for you to upload the HBase Master logs to a site like pastebin.com or gist.github.com so we can look at? cheers, esteban. -- Cloudera, Inc. On Tue, Apr 7, 2015 at 8:03 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, The port the HBase Master should bind to 6 Thanks,Sridhar From: serega.shey...@gmail.com Date: Tue, 7 Apr 2015 16:40:54 +0200 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org CC: bus...@cloudera.com property namehbase.master/name valuehdfs://cluster1:6/value /property what is it? 2015-04-07 16:34 GMT+02:00 sridhararao mutluri drm...@hotmail.com: Hi, This is my hbase-site.xml: configuration propertynamehbase.master/name valuehdfs://cluster1:6/value /property propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user@hbase.apache.org CC: bus...@cloudera.com bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet
RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Please post a link to updated master logs. -- Sean On Apr 9, 2015 5:50 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, We increased maximum files and with reboot ulimit -n shows across all clusters [hadoop@solix1 bin]$ ulimit -n 65535 But unluckily we are getting same error now also: hbase(main):001:0 create 'buses', 'vi' *ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again later* at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Do we downgrade JDK from 1.8 to 1.7 and validate.Please suggest. Thanks, Sridhar -- From: drm...@hotmail.com To: este...@cloudera.com Subject: RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase Date: Thu, 9 Apr 2015 06:44:10 +0530 Hi Esteban, Thanks much for your speedy suggestion.I will implement and will come back to you. Regards, Dr.Sridhar -- From: este...@cloudera.com Date: Wed, 8 Apr 2015 18:06:21 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: drm...@hotmail.com what i meant to say is that you should try to change the ulimit to 64k -- Cloudera, Inc. On Wed, Apr 8, 2015 at 6:06 PM, Esteban Gutierrez este...@cloudera.com wrote: cool... so the HBase master started, but you might want to bump your ulimits -n by 60 times, 1024 of max open files is too low of hbase. -- Cloudera, Inc. On Wed, Apr 8, 2015 at 5:55 PM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, Please find the link: https://gist.github.com/drmsr6/bcf439f09cac95b59eca I am waiting for final inputs.:) Thanks, Dr.Sridhar, 919573453136 -- From: este...@cloudera.com Date: Wed, 8 Apr 2015 11:54:04 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: drm...@hotmail.com Sridhar, what is the URL to the log file? thanks -- Cloudera, Inc. On Tue, Apr 7, 2015 at 6:35 PM, sridhararao mutluri drm...@hotmail.com wrote: Hi Esteban, I pasted logs in github.com: Thanks, Sridhar From: este...@cloudera.com Date: Tue, 7 Apr 2015 08:44:55 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org Sridhar, What do you see in the HBase Master logs? The exception you are getting from the HBase Master is just a side effect and not the real cause? Is it possible for you to upload the HBase Master logs to a site like pastebin.com or gist.github.com so we can look at? cheers, esteban. -- Cloudera, Inc. On Tue, Apr 7, 2015 at 8:03 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, The port the HBase Master should bind to 6 Thanks,Sridhar From: serega.shey...@gmail.com Date: Tue, 7 Apr 2015 16:40:54 +0200 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org CC: bus...@cloudera.com property namehbase.master/name valuehdfs://cluster1:6/value /property what is it? 2015-04-07 16:34 GMT+02:00 sridhararao mutluri drm...@hotmail.com: Hi, This is my hbase-site.xml: configuration property namehbase.master/name valuehdfs://cluster1:6/value /property property namehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user
RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Hi, This is my hbase-site.xml: configuration propertynamehbase.master/name valuehdfs://cluster1:6/value /property property namehbase.rootdir/namevaluehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user@hbase.apache.org CC: bus...@cloudera.com bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet from shell output. Have you checked master log ? Cheers On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I am trying to use hbase 0.98 distributed mode with zk 3.4.6 hadoop ha 2.6.(JDK 1.8) I am having following issue and little help in google pages also I tried to start zk first after clearing zk data dir and tried to start master first and rs later and no luck I used mycluster/hbase in hbase-site.xml and no luck to me.tried to put hdfs-site.xml/core-site.xml in $Hbase_home/conf also. I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of hadoop where as we are using 2.6 and tried to copy those hadoop jars ..but no luck. A New new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. My bash_profile on classpath:export CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. my hbase_site.xml propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property and hadoop core-site is same. Any incompatibility between JDK1.8 with hbase or hadoop 2.6? Please suggest any solution. Thanks,Sridhar
RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Team, The port the HBase Master should bind to 6 Thanks,Sridhar From: serega.shey...@gmail.com Date: Tue, 7 Apr 2015 16:40:54 +0200 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org CC: bus...@cloudera.com property namehbase.master/name valuehdfs://cluster1:6/value /property what is it? 2015-04-07 16:34 GMT+02:00 sridhararao mutluri drm...@hotmail.com: Hi, This is my hbase-site.xml: configuration propertynamehbase.master/name valuehdfs://cluster1:6/value /property propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user@hbase.apache.org CC: bus...@cloudera.com bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet from shell output. Have you checked master log ? Cheers On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I am trying to use hbase 0.98 distributed mode with zk 3.4.6 hadoop ha 2.6.(JDK 1.8) I am having following issue and little help in google pages also I tried to start zk first after clearing zk data dir and tried to start master first and rs later and no luck I used mycluster/hbase in hbase-site.xml and no luck to me.tried to put hdfs-site.xml/core-site.xml in $Hbase_home/conf also. I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of hadoop where as we are using 2.6 and tried to copy those hadoop jars ..but no luck. A New new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. My bash_profile on classpath:export CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. my hbase_site.xml propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property and hadoop core-site is same. Any incompatibility between JDK1.8 with hbase or hadoop 2.6? Please suggest any solution. Thanks,Sridhar
Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
property namehbase.master/name valuehdfs://cluster1:6/value /property what is it? 2015-04-07 16:34 GMT+02:00 sridhararao mutluri drm...@hotmail.com: Hi, This is my hbase-site.xml: configuration propertynamehbase.master/name valuehdfs://cluster1:6/value /property propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user@hbase.apache.org CC: bus...@cloudera.com bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet from shell output. Have you checked master log ? Cheers On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I am trying to use hbase 0.98 distributed mode with zk 3.4.6 hadoop ha 2.6.(JDK 1.8) I am having following issue and little help in google pages also I tried to start zk first after clearing zk data dir and tried to start master first and rs later and no luck I used mycluster/hbase in hbase-site.xml and no luck to me.tried to put hdfs-site.xml/core-site.xml in $Hbase_home/conf also. I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of hadoop where as we are using 2.6 and tried to copy those hadoop jars ..but no luck. A New new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. My bash_profile on classpath:export CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. my hbase_site.xml propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property and hadoop core-site is same. Any incompatibility between JDK1.8 with hbase or hadoop 2.6? Please suggest any solution. Thanks,Sridhar
Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet from shell output. Have you checked master log ? Cheers On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I am trying to use hbase 0.98 distributed mode with zk 3.4.6 hadoop ha 2.6.(JDK 1.8) I am having following issue and little help in google pages also I tried to start zk first after clearing zk data dir and tried to start master first and rs later and no luck I used mycluster/hbase in hbase-site.xml and no luck to me.tried to put hdfs-site.xml/core-site.xml in $Hbase_home/conf also. I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of hadoop where as we are using 2.6 and tried to copy those hadoop jars ..but no luck. A New new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. My bash_profile on classpath:export CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. my hbase_site.xml propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property and hadoop core-site is same. Any incompatibility between JDK1.8 with hbase or hadoop 2.6? Please suggest any solution. Thanks,Sridhar
Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Sridhar, What do you see in the HBase Master logs? The exception you are getting from the HBase Master is just a side effect and not the real cause? Is it possible for you to upload the HBase Master logs to a site like pastebin.com or gist.github.com so we can look at? cheers, esteban. -- Cloudera, Inc. On Tue, Apr 7, 2015 at 8:03 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, The port the HBase Master should bind to 6 Thanks,Sridhar From: serega.shey...@gmail.com Date: Tue, 7 Apr 2015 16:40:54 +0200 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org CC: bus...@cloudera.com property namehbase.master/name valuehdfs://cluster1:6/value /property what is it? 2015-04-07 16:34 GMT+02:00 sridhararao mutluri drm...@hotmail.com: Hi, This is my hbase-site.xml: configuration propertynamehbase.master/name valuehdfs://cluster1:6/value /property propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user@hbase.apache.org CC: bus...@cloudera.com bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet from shell output. Have you checked master log ? Cheers On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I am trying to use hbase 0.98 distributed mode with zk 3.4.6 hadoop ha 2.6.(JDK 1.8) I am having following issue and little help in google pages also I tried to start zk first after clearing zk data dir and tried to start master first and rs later and no luck I used mycluster/hbase in hbase-site.xml and no luck to me.tried to put hdfs-site.xml/core-site.xml in $Hbase_home/conf also. I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of hadoop where as we are using 2.6 and tried to copy those hadoop jars ..but no luck. A New new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. My bash_profile on classpath:export CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. my hbase_site.xml propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property and hadoop core-site is same. Any incompatibility between JDK1.8 with hbase or hadoop 2.6? Please suggest any solution. Thanks,Sridhar
RE: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase
Hi Esteban, I pasted logs in github.com: Thanks,Sridhar From: este...@cloudera.com Date: Tue, 7 Apr 2015 08:44:55 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org Sridhar, What do you see in the HBase Master logs? The exception you are getting from the HBase Master is just a side effect and not the real cause? Is it possible for you to upload the HBase Master logs to a site like pastebin.com or gist.github.com so we can look at? cheers, esteban. -- Cloudera, Inc. On Tue, Apr 7, 2015 at 8:03 AM, sridhararao mutluri drm...@hotmail.com wrote: Team, The port the HBase Master should bind to 6 Thanks,Sridhar From: serega.shey...@gmail.com Date: Tue, 7 Apr 2015 16:40:54 +0200 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase To: user@hbase.apache.org CC: bus...@cloudera.com property namehbase.master/name valuehdfs://cluster1:6/value /property what is it? 2015-04-07 16:34 GMT+02:00 sridhararao mutluri drm...@hotmail.com: Hi, This is my hbase-site.xml: configuration propertynamehbase.master/name valuehdfs://cluster1:6/value /property propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property namehbase.cluster.distributed/name valuetrue/value /property property namehbase.zookeeper.property.clientPort/name value2181/value /property property namehbase.zookeeper.quorum/name valuecluster1,cluster2,cluster3/value /property property namehbase.zookeeper.property.dataDir/name value/hadoop/hdfs/zookeeper/data/zk1/value /property/configuration Thanks,Sridhar Date: Tue, 7 Apr 2015 07:09:44 -0700 Subject: Re: Hbase 0.98 Distributed Mode with hadoop 2.6 HA:Issues of Hbase From: yuzhih...@gmail.com To: user@hbase.apache.org CC: bus...@cloudera.com bq. propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property Looks like there is a property missing at the end of the line. You showed snippet from shell output. Have you checked master log ? Cheers On Tue, Apr 7, 2015 at 5:16 AM, sridhararao mutluri drm...@hotmail.com wrote: Hi Team, I am trying to use hbase 0.98 distributed mode with zk 3.4.6 hadoop ha 2.6.(JDK 1.8) I am having following issue and little help in google pages also I tried to start zk first after clearing zk data dir and tried to start master first and rs later and no luck I used mycluster/hbase in hbase-site.xml and no luck to me.tried to put hdfs-site.xml/core-site.xml in $Hbase_home/conf also. I noticed all hadoop.*jars in $HBASE_HOME/lib are 2.2 of hadoop where as we are using 2.6 and tried to copy those hadoop jars ..but no luck. A New new error is coming: hbase(main):002:0 create 'cars', 'vi' ERROR: java.io.IOException: Table Namespace Manager not ready yet, try again laterat org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)ZK are running on 3 servers.I tired to stop hbase and then stopped zk in 3 clusters and cleared zk data and started fresh. My bash_profile on classpath:export CLASSPATH=$CLASSPATH:/home/hadoop/hadoop-2.6.0/lib/*:.export CLASSPATH=$CLASSPATH:/home/hadoop/hbase-0.98.4/lib/*:. my hbase_site.xml propertynamehbase.rootdir/name valuehdfs://mycluster/hbase/value /property property and hadoop core-site is same. Any incompatibility between JDK1.8 with hbase or hadoop 2.6? Please suggest any solution. Thanks,Sridhar
