I took a look at opentsdb.log which shows asynchbase logs. Have you considered polling asynchbase mailing list ?
On Wed, Dec 14, 2016 at 4:59 PM, karanalang <karan.al...@gmail.com> wrote: > I've kerberized HDP 2.4 (HBase version - 1.1.2.2.4.0.0-169, openTSDB > version > - 2.2.1) > On startiong OpenTSDB, i'm getting following error -> > > What i understand (from the posts on the internet) is that this is because > of connection issues between HBase & OpenTSDB > > In connecting to HBase .META, the connection is failing. > Any ideas on how to fix this ? > > Here is the command used to start OpenTSDB > /build/tsdb tsd --zkbasedir=/hbase-secure --port=9999 > --zkquorum=sandbox.hortonworks.com:2181 --cachedir=/tmp/tsd > --staticroot=build/staticroot --auto-metric > > Also attaching the hbase-site.xml, opentsdb.log, opentsdb.conf > > > > ------------------------------------------------------------ > ---------------------------------------- > > 2016-12-14 18:55:58,002 INFO [AsyncHBase I/O Worker #4] HBaseClient: > Channel [id: 0x9040cb54, /10.0.2.15:51692 => /10.0.2.15:16020] is > disconnecting: [id: 0x9040cb54, /10.0.2.15:51692 => /10.0.2.15:16020] > CLOSE > 2016-12-14 18:55:58,002 INFO [AsyncHBase I/O Worker #4] HBaseClient: Lost > connection with the .META. region > 2016-12-14 18:55:58,003 INFO [AsyncHBase I/O Worker #4] HBaseClient: > Invalidated cache for .META. as null still seems to be splitting or closing > it. > 2016-12-14 18:55:58,006 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > ZooKeeper: Session: 0x158fa6cce65022b closed > 2016-12-14 18:55:58,007 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > ClientCnxn: EventThread shut down > 2016-12-14 18:55:59,524 INFO [AsyncHBase Timer HBaseClient #1] ZooKeeper: > Initiating client connection, connectString=sandbox.hortonworks.com:2181 > sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@438f3a6 > 2016-12-14 18:55:59,531 INFO [AsyncHBase Timer HBaseClient > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket > connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not > attempt to authenticate using SASL (unknown error) > 2016-12-14 18:55:59,533 INFO [AsyncHBase Timer HBaseClient > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection > established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session > 2016-12-14 18:55:59,535 INFO [AsyncHBase Timer HBaseClient #1] > HBaseClient: > Need to find the .META. region > 2016-12-14 18:55:59,536 INFO [AsyncHBase Timer HBaseClient > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session > establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, > sessionid = 0x158fa6cce65022c, negotiated timeout = 5000 > 2016-12-14 18:55:59,539 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 > 2016-12-14 18:55:59,545 INFO [AsyncHBase I/O Worker #1] HBaseClient: > Channel [id: 0x860a8aa5, /10.0.2.15:51700 :> /10.0.2.15:16020] is > disconnecting: [id: 0x860a8aa5, /10.0.2.15:51700 :> /10.0.2.15:16020] > DISCONNECTED > 2016-12-14 18:55:59,548 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > HBaseClient: Need to find the .META. region > 2016-12-14 18:55:59,553 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > ZooKeeper: Session: 0x158fa6cce65022c closed > 2016-12-14 18:55:59,553 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > ZooKeeper: Initiating client connection, > connectString=sandbox.hortonworks.com:2181 sessionTimeout=5000 > watcher=org.hbase.async.HBaseClient$ZKClient@438f3a6 > 2016-12-14 18:55:59,576 INFO [AsyncHBase Timer HBaseClient > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Opening socket > connection to server sandbox.hortonworks.com/10.0.2.15:2181. Will not > attempt to authenticate using SASL (unknown error) > 2016-12-14 18:55:59,579 INFO [AsyncHBase Timer HBaseClient > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Socket connection > established to sandbox.hortonworks.com/10.0.2.15:2181, initiating session > 2016-12-14 18:55:59,585 INFO [AsyncHBase Timer HBaseClient > #1-SendThread(sandbox.hortonworks.com:2181)] ClientCnxn: Session > establishment complete on server sandbox.hortonworks.com/10.0.2.15:2181, > sessionid = 0x158fa6cce65022d, negotiated timeout = 5000 > 2016-12-14 18:55:59,586 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > ClientCnxn: EventThread shut down > 2016-12-14 18:55:59,594 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > HBaseClient: Connecting to .META. region @ 10.0.2.15:16020 > 2016-12-14 18:55:59,603 INFO [AsyncHBase I/O Worker #2] HBaseClient: > Channel [id: 0x6c0b2673, /10.0.2.15:51704 :> /10.0.2.15:16020] is > disconnecting: [id: 0x6c0b2673, /10.0.2.15:51704 :> /10.0.2.15:16020] > DISCONNECTED > 2016-12-14 18:55:59,606 WARN [AsyncHBase Timer HBaseClient #1-EventThread] > HBaseClient: Probe Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, > 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, > 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, > 44, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, > 110, 99, 72, 66, 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, > 95, > 60], family=null, qualifiers=null, attempt=0, > region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", > stop_key="")) failed > org.hbase.async.NonRecoverableException: Too many attempts: > Exists(table="hbase:meta", key=[116, 115, 100, 98, 44, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, 97, > 115, > 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60, 44, 58, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 65, 115, 121, 110, 99, 72, 66, > 97, 115, 101, 126, 112, 114, 111, 98, 101, 126, 60, 59, 95, 60], > family=null, qualifiers=null, attempt=11, > region=RegionInfo(table="hbase:meta", region_name="hbase:meta,,1", > stop_key="")) > at org.hbase.async.HBaseClient.tooManyAttempts(HBaseClient. > java:2056) > [asynchbase-1.7.2.jar:na] > at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient. > java:1920) > [asynchbase-1.7.2.jar:na] > at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient. > java:1944) > [asynchbase-1.7.2.jar:na] > at org.hbase.async.HBaseClient$1RetryRpc.call(HBaseClient. > java:1927) > [asynchbase-1.7.2.jar:na] > at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) > [async-1.4.0.jar:na] > at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) > [async-1.4.0.jar:na] > at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) > [async-1.4.0.jar:na] > at > org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient. > java:3645) > [asynchbase-1.7.2.jar:na] > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java: > 558) > [zookeeper-3.4.5.jar:3.4.5-1392090] > at org.apache.zookeeper.ClientCnxn$EventThread.run( > ClientCnxn.java:495) > [zookeeper-3.4.5.jar:3.4.5-1392090] > 2016-12-14 18:55:59,607 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > HBaseClient: Need to find the .META. region > 2016-12-14 18:55:59,611 INFO [AsyncHBase Timer HBaseClient #1-EventThread] > ZooKeeper: Session: 0x158fa6cce65022d closed > ------------------------------------------------------------ > ----------------- > > hbase-site.txt > <http://apache-hbase.679495.n3.nabble.com/file/n4085136/hbase-site.txt> > opentsdb.zip > <http://apache-hbase.679495.n3.nabble.com/file/n4085136/opentsdb.zip> > hbase-site.txt > <http://apache-hbase.679495.n3.nabble.com/file/n4085136/hbase-site.txt> > > > > -- > View this message in context: http://apache-hbase.679495.n3. > nabble.com/Regionserver-IP-in-Meta-tp4077711p4085136.html > Sent from the HBase User mailing list archive at Nabble.com. >
