Hello Everyone,
We added a node to our cluster, and startup the datanode, tasktracker,
regionserver, but the regionserver failed.And we noted that there was some
exception in hbase log as following:
2010-03-02 19:18:06,565 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server cactus208/127.0.0.1:2222
2010-03-02 19:18:06,570 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x0 to sun.nio.ch.selectionkeyi...@7d95d4fe
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
2010-03-02 19:18:06,572 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown input
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-03-02 19:18:06,572 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
2010-03-02 19:18:06,689 WARN
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set watcher on
ZNode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.watchMasterAddress(ZooKeeperWrapper.java:304)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.watchMasterAddress(HRegionServer.java:385)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reinitializeZooKeeper(HRegionServer.java:315)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reinitialize(HRegionServer.java:306)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:276)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.doMain(HRegionServer.java:2472)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2540)
2010-03-02 19:18:06,689 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to set watcher on
ZooKeeper master address. Retrying.
2010-03-02 19:18:07,417 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server cactus209/172.16.1.209:2222
2010-03-02 19:18:07,418 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
172.16.1.208:39575 remote
=cactus209/172.16.1.209:2222]
2010-03-02 19:18:07,421 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
...
...
...
2010-03-02 19:23:37,084 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
172.16.1.207:60000 that we are up
2010-03-02 19:23:37,102 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Unhandled exception.
Aborting...
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:459)
at java.lang.Thread.run(Thread.java:619)
2010-03-02 19:23:37,103 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=0, stores=0, storefiles=0, storefileInd
exSize=0, memstoreSize=0, usedHeap=25, maxHeap=2991, blockCacheSize=5147928,
blockCacheFree=622254440, blockCacheCount=0, blockCacheHitRatio=0
2010-03-02 19:23:37,104 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60020
2010-03-02 19:23:37,104 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 60020
2010-03-02 19:23:37,104 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60020: exiting
2010-03-02 19:23:37,104 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2010-03-02 19:23:37,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020: exiting
2010-03-02 19:23:37,107 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2010-03-02 19:23:37,110 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020: exiting
2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020: exiting
2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60020: exiting
2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60020: exiting
2010-03-02 19:23:37,111 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60020: exiting
2010-03-02 19:23:37,112 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020: exiting
2010-03-02 19:23:37,112 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60020: exiting
2010-03-02 19:23:37,112 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60020: exiting
2010-03-02 19:23:37,114 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
regionserver/127.0.0.1:60020.compactor exiting
The version we are using is hbase0.20.1.Anyone can give some
suggestions?Thank you very much.
LvZheng
