No, nothing at the master, other than a bunch of NodeExists exceptions in the zookeeper log.
Something else I've found is that I can add records to the table through the shell, without problems. It's only from the java client, on a separate box, that I'm having problems. Thanks, Adam On 1/13/10 12:08 PM, "Jean-Daniel Cryans" <[email protected]> wrote: > It means the root region likely wasn't assigned, anything in the > master about it? > > J-D > > On Wed, Jan 13, 2010 at 11:43 AM, Adam Silberstein > <[email protected]> wrote: >> I have version 0.20.2. The client requests hang. >> And, I looked in the region server logs...here is the likely culprit: >> >> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/root-region-server >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >> perWrapper.java:332) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap >> per.java:318) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo >> oKeeperWrapper.java:231) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44 >> 2) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/root-region-server >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >> perWrapper.java:328) >> ... 4 more >> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/root-region-server >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >> perWrapper.java:332) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap >> per.java:318) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo >> oKeeperWrapper.java:231) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44 >> 2) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /hbase/root-region-server >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >> perWrapper.java:328) >> ... 4 more >> >> Is there zk state on the region servers? Maybe when wiping the system, I >> should have wiped something on them. >> >> Thanks, >> Adam >> >> On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <[email protected]> wrote: >> >>> Those node exists are usually harmless, I would like the see what's on >>> the HBase region server and master side. >>> >>> You say it cannot do any operation but you don't see exceptions >>> client-side? It just hangs? Which version is it? >>> >>> Thx, >>> >>> J-D >>> >>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein >>> <[email protected]> wrote: >>>> Hi, >>>> I set up a new Hbase cluster and am seeing an error I haven't seen before. >>>> When I start up Hbase I have this NodeExists error in the zookeeper log: >>>> >>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = >>>> NodeExists >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> s >>>> or.java:245) >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> a >>>> va:114) >>>> 2010-01-13 11:10:47,503 WARN >>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc >>>> zxid:0xfffffffffffffffe txntype:unknown n/a >>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = >>>> NodeExists >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> s >>>> or.java:245) >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> a >>>> va:114) >>>> >>>> I've read that may be harmless. But then when I run the java client, it >>>> cannot do any operations. And I see a similar NodeExists error in the >>>> zookeeper log. >>>> >>>> >>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> Connected to /203.104.21.210:36928 lastZxid 0 >>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> Creating new session 0x1262918396b0004 >>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> Finished init of 0x1262918396b0004 valid:true >>>> 2010-01-13 11:11:24,973 WARN >>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2 >>>> zxid:0xfffffffffffffffe txntype:unknown n/a >>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = >>>> NodeExists >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> s >>>> or.java:245) >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> a >>>> va:114) >>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn: >>>> Exception causing close of session 0x1262918396b0004 due to >>>> java.io.IOException: Read error >>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> closing session:0x1262918396b0004 NIOServerCnxn: >>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181 >>>> remote=/203.104.21.210:36928] >>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> Connected to /203.104.21.212:54837 lastZxid 0 >>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> Creating new session 0x1262918396b0005 >>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>> Finished init of 0x1262918396b0005 valid:true >>>> 2010-01-13 11:11:42,711 WARN >>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1 >>>> zxid:0xfffffffffffffffe txntype:unknown n/a >>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = >>>> NodeExists >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> s >>>> or.java:245) >>>> at >>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> a >>>> va:114) >>>> 2010-01-13 11:12:06,002 INFO >>>> org.apache.zookeeper.server.SessionTrackerImpl: >>>> Expiring session 0x1262918396b0004 >>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer: >>>> Expiring session 0x1262918396b0004 >>>> 2010-01-13 11:12:06,002 INFO >>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session >>>> termination request for id: 0x1262918396b0004 >>>> >>>> I've tried wiping the whole setup, including the zookeeper files that Hbase >>>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm >>>> not seeing any exceptions on the client-side. >>>> >>>> Thanks, >>>> Adam >>>> >>>> >> >>
