On 2019/04/04 15:15:37, Josh Elser <[email protected]> wrote:
> Looks like your RegionServer process might have died if you can't
> connect to its RPC port.
>
> Did you look in the RegionServer log for any mention of an ERROR or
> FATAL log message?
>
> On 4/4/19 8:20 AM, [email protected] wrote:
> > I have installed Hadoop single node
> > http://intellitech.pro/tutorial-hadoop-first-lab/ and Hbase
> > http://intellitech.pro/hbase-installation-on-ubuntu/ successfully. I am
> > using a Java agent to connect to the Hbase. After a random time period
> > Hbase stop working and the java agent gives following error message.
> >
> > Call exception, tries=7, retries=7, started=8321 ms ago, cancelled=false,
> > msg=Call to db-2.c.xxx-dev.internal/xx.xx.0.21:16201 failed on connection
> > exception:
> > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
> > Connection refused: db-2.c.xxx-dev.internal/xx.xx.0.21:16201, details=row
> > 'xxx,00000000001:1553904000000,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > hostname=db-2.c.xxx-dev.internal,16201,1553683263844, seqNum=-1
> > Here are the Hbase and zookeeper logs
> >
> > hbase-hduser-regionserver-db-2.log
> >
> > [main] zookeeper.ZooKeeperMain: Processing delete 2019-03-30 02:11:44,089
> > DEBUG [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Reading reply
> > sessionid:0x169bd98c099006e, packet:: clientPath:null serverPath:null
> > finished:false header:: 1,2 replyHeader:: 1,300964,0 request::
> > '/hbase/rs/db-2.c.stl-cardio-dev.internal%2C16201%2C1553683263844,-1
> > response:: null
> > hbase-hduser-zookeeper-db-2.log
> >
> > server.FinalRequestProcessor: sessionid:0x169bd98c099004a type:getChildren
> > cxid:0x28e3ad zxid:0xfffffffffffffffe txntype:unknown
> > reqpath:/hbase/splitWAL
> > my hbase-site.xml file is as follows
> >
> > <configuration>
> > //Here you have to set the path where you want HBase to store its files.
> > <property>
> > <name>hbase.rootdir</name>
> > <value>hdfs://localhost:9000/hbase</value>
> > </property>
> > <property>
> > <name>hbase.zookeeper.quorum</name>
> > <value>localhost</value>
> > </property>
> > //Here you have to set the path where you want HBase to store its built
> > in zookeeper files.
> > <property>
> > <name>hbase.zookeeper.property.dataDir</name>
> > <value>${hbase.tmp.dir}/zookeeper</value>
> > </property>
> > <property>
> > <name>hbase.cluster.distributed</name>
> > <value>true</value>
> > </property>
> > <property>
> > <name>hbase.zookeeper.property.clientPort</name>
> > <value>2181</value>
> > </property>
> > </configuration>
> > when I restart the Hbase it will start working again and stop working after
> > few days. I am wondering what would be the fix for this.
> >
> > Thanks.
> > BR,
> > Melanka
> >
> Hi Josh,
Sorry for the late reply. I restarted the Hbase on 05/04/2019 and it was again
down on 06/04/2019 at 00.06 AM.
Log from hbase-root-regionserver-db-2 is as follows.
2019-04-04 04:42:26,047 DEBUG [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Reading reply sessionid:0x169d86a879b00bf, packet::
clientPath:null serverPath:null finished:false header:: 67,2 replyHeader::
67,776370,0 request::
'/hbase/rs/db-2.c.stl-cardio-dev.internal%2C16201%2C1554352093266,-1
response:: null
2019-04-04 04:42:26,047 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher:
regionserver:16201-0x169d86a879b00bf, quorum=localhost:2181, baseZNode=/hbase
Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected,
path=/hbase/rs/db-2.c.stl-cardio-dev.internal,16201,1554352093266
2019-04-04 04:42:26,047 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher:
regionserver:16201-0x169d86a879b00bf, quorum=localhost:2181, baseZNode=/hbase
Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected,
path=/hbase/rs
2019-04-04 04:42:26,050 DEBUG
[regionserver/db-2.c.xxx-dev.internal/xx.xxx.0.21:16201] zookeeper.ZooKeeper:
Closing session: 0x169d86a879b00bf
2019-04-04 04:42:26,050 DEBUG
[regionserver/db-2.c.xxx-dev.internal/xx.xx.0.21:16201] zookeeper.ClientCnxn:
Closing client for session: 0x169d86a879b00bf
2019-04-04 04:42:26,056 DEBUG [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Reading reply sessionid:0x169d86a879b00bf, packet::
clientPath:null serverPath:null finished:false header:: 68,-11 replyHeader::
68,776371,0 request:: null response:: null
2019-04-04 04:42:26,056 DEBUG
[regionserver/db-2.c.xxx-dev.internal/xx.xxx.0.21:16201] zookeeper.ClientCnxn:
Disconnecting client for session: 0x169d86a879b00bf
2019-04-04 04:42:26,056 INFO
[regionserver/db-2.c.xxx-dev.internal/xxx.xxx.0.21:16201] zookeeper.ZooKeeper:
Session: 0x169d86a879b00bf closed
2019-04-04 04:42:26,056 INFO
[regionserver/db-2.c.xxx-dev.internal/xxx.xxx.0.21:16201]
regionserver.HRegionServer: stopping server
db-2.c.xxx-dev.internal,16201,1554352093266; zookeeper connection closed.
2019-04-04 04:42:26,056 INFO
[regionserver/db-2.c.xxx-dev.internal/xxx.0.21:16201]
regionserver.HRegionServer: regionserver/db-2.c.xxx-dev.internal/xxx.0.21:16201
exiting
2019-04-04 04:42:26,056 ERROR [main] regionserver.HRegionServerCommandLine:
Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2831)
2019-04-04 04:42:26,057 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down for session: 0x169d86a879b00bf
2019-04-04 04:42:26,063 INFO [Thread-5] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@35a9782c
2019-04-04 04:42:26,067 INFO [Thread-5] regionserver.ShutdownHook: Starting fs
shutdown hook thread.
2019-04-04 04:42:26,073 INFO [Thread-5] regionserver.ShutdownHook: Shutdown
hook finished.
Log from the hbase-hduser-regionserver-db-2 is ass follows.
2019-04-05 13:48:14,734 DEBUG [main] zookeeper.ZooKeeperMain: Processing delete
2019-04-05 13:48:14,754 DEBUG [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Reading reply sessionid:0x169eb8b7c230010, packet::
clientPath:null serverPath:null finished:false header:: 1,2 replyHeader::
1,783888,0 request::
'/hbase/rs/db-2.c.xxx-dev.internal%2C16201%2C1554434982329,-1 response:: null
