I have version 0.20.2. The client requests hang.
And, I looked in the region server logs...here is the likely culprit:
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:332)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
per.java:318)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
oKeeperWrapper.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
2)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:328)
... 4 more
java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:332)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap
per.java:318)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo
oKeeperWrapper.java:231)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44
2)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee
perWrapper.java:328)
... 4 more
Is there zk state on the region servers? Maybe when wiping the system, I
should have wiped something on them.
Thanks,
Adam
On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <[email protected]> wrote:
> Those node exists are usually harmless, I would like the see what's on
> the HBase region server and master side.
>
> You say it cannot do any operation but you don't see exceptions
> client-side? It just hangs? Which version is it?
>
> Thx,
>
> J-D
>
> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein
> <[email protected]> wrote:
>> Hi,
>> I set up a new Hbase cluster and am seeing an error I haven't seen before.
>> When I start up Hbase I have this NodeExists error in the zookeeper log:
>>
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:10:47,503 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0002 type:create cxid:0xc
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>>
>> I've read that may be harmless. But then when I run the java client, it
>> cannot do any operations. And I see a similar NodeExists error in the
>> zookeeper log.
>>
>>
>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Connected to /203.104.21.210:36928 lastZxid 0
>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Creating new session 0x1262918396b0004
>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Finished init of 0x1262918396b0004 valid:true
>> 2010-01-13 11:11:24,973 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0004 type:create cxid:0x2
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> Exception causing close of session 0x1262918396b0004 due to
>> java.io.IOException: Read error
>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> closing session:0x1262918396b0004 NIOServerCnxn:
>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181
>> remote=/203.104.21.210:36928]
>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Connected to /203.104.21.212:54837 lastZxid 0
>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Creating new session 0x1262918396b0005
>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> Finished init of 0x1262918396b0005 valid:true
>> 2010-01-13 11:11:42,711 WARN
>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
>> processing sessionid:0x1262918396b0005 type:create cxid:0x1
>> zxid:0xfffffffffffffffe txntype:unknown n/a
>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
>> NodeExists
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcess
>> or.java:245)
>> at
>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.ja
>> va:114)
>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.SessionTrackerImpl:
>> Expiring session 0x1262918396b0004
>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer:
>> Expiring session 0x1262918396b0004
>> 2010-01-13 11:12:06,002 INFO
>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
>> termination request for id: 0x1262918396b0004
>>
>> I've tried wiping the whole setup, including the zookeeper files that Hbase
>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm
>> not seeing any exceptions on the client-side.
>>
>> Thanks,
>> Adam
>>
>>
