Oh, so HBase is working fine but you have a remote client connectivity problem. You gave the good hbase.zookeeper.quorum to HTable?
J-D On Wed, Jan 13, 2010 at 1:04 PM, Adam Silberstein <[email protected]> wrote: > No, nothing at the master, other than a bunch of NodeExists exceptions in > the zookeeper log. > > Something else I've found is that I can add records to the table through the > shell, without problems. It's only from the java client, on a separate box, > that I'm having problems. > > Thanks, > Adam > > > On 1/13/10 12:08 PM, "Jean-Daniel Cryans" <[email protected]> wrote: > >> It means the root region likely wasn't assigned, anything in the >> master about it? >> >> J-D >> >> On Wed, Jan 13, 2010 at 11:43 AM, Adam Silberstein >> <[email protected]> wrote: >>> I have version 0.20.2. The client requests hang. >>> And, I looked in the region server logs...here is the likely culprit: >>> >>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: >>> KeeperErrorCode = NoNode for /hbase/root-region-server >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >>> perWrapper.java:332) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap >>> per.java:318) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo >>> oKeeperWrapper.java:231) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44 >>> 2) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >>> KeeperErrorCode = NoNode for /hbase/root-region-server >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >>> perWrapper.java:328) >>> ... 4 more >>> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: >>> KeeperErrorCode = NoNode for /hbase/root-region-server >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >>> perWrapper.java:332) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddress(ZooKeeperWrap >>> per.java:318) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readRootRegionLocation(Zo >>> oKeeperWrapper.java:231) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:44 >>> 2) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >>> KeeperErrorCode = NoNode for /hbase/root-region-server >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:102) >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKee >>> perWrapper.java:328) >>> ... 4 more >>> >>> Is there zk state on the region servers? Maybe when wiping the system, I >>> should have wiped something on them. >>> >>> Thanks, >>> Adam >>> >>> On 1/13/10 11:33 AM, "Jean-Daniel Cryans" <[email protected]> wrote: >>> >>>> Those node exists are usually harmless, I would like the see what's on >>>> the HBase region server and master side. >>>> >>>> You say it cannot do any operation but you don't see exceptions >>>> client-side? It just hangs? Which version is it? >>>> >>>> Thx, >>>> >>>> J-D >>>> >>>> On Wed, Jan 13, 2010 at 11:25 AM, Adam Silberstein >>>> <[email protected]> wrote: >>>>> Hi, >>>>> I set up a new Hbase cluster and am seeing an error I haven't seen before. >>>>> When I start up Hbase I have this NodeExists error in the zookeeper log: >>>>> >>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode >>>>> = >>>>> NodeExists >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> > s >>>>> or.java:245) >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> > a >>>>> va:114) >>>>> 2010-01-13 11:10:47,503 WARN >>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>>> processing sessionid:0x1262918396b0002 type:create cxid:0xc >>>>> zxid:0xfffffffffffffffe txntype:unknown n/a >>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode >>>>> = >>>>> NodeExists >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> > s >>>>> or.java:245) >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> > a >>>>> va:114) >>>>> >>>>> I've read that may be harmless. But then when I run the java client, it >>>>> cannot do any operations. And I see a similar NodeExists error in the >>>>> zookeeper log. >>>>> >>>>> >>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> Connected to /203.104.21.210:36928 lastZxid 0 >>>>> 2010-01-13 11:11:24,661 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> Creating new session 0x1262918396b0004 >>>>> 2010-01-13 11:11:24,664 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> Finished init of 0x1262918396b0004 valid:true >>>>> 2010-01-13 11:11:24,973 WARN >>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>>> processing sessionid:0x1262918396b0004 type:create cxid:0x2 >>>>> zxid:0xfffffffffffffffe txntype:unknown n/a >>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode >>>>> = >>>>> NodeExists >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> > s >>>>> or.java:245) >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> > a >>>>> va:114) >>>>> 2010-01-13 11:11:33,088 WARN org.apache.zookeeper.server.NIOServerCnxn: >>>>> Exception causing close of session 0x1262918396b0004 due to >>>>> java.io.IOException: Read error >>>>> 2010-01-13 11:11:33,089 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> closing session:0x1262918396b0004 NIOServerCnxn: >>>>> java.nio.channels.SocketChannel[connected local=/203.104.21.210:2181 >>>>> remote=/203.104.21.210:36928] >>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> Connected to /203.104.21.212:54837 lastZxid 0 >>>>> 2010-01-13 11:11:42,709 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> Creating new session 0x1262918396b0005 >>>>> 2010-01-13 11:11:42,711 INFO org.apache.zookeeper.server.NIOServerCnxn: >>>>> Finished init of 0x1262918396b0005 valid:true >>>>> 2010-01-13 11:11:42,711 WARN >>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>>> processing sessionid:0x1262918396b0005 type:create cxid:0x1 >>>>> zxid:0xfffffffffffffffe txntype:unknown n/a >>>>> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode >>>>> = >>>>> NodeExists >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces>>>> > s >>>>> or.java:245) >>>>> at >>>>> > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j>>>> > a >>>>> va:114) >>>>> 2010-01-13 11:12:06,002 INFO >>>>> org.apache.zookeeper.server.SessionTrackerImpl: >>>>> Expiring session 0x1262918396b0004 >>>>> 2010-01-13 11:12:06,002 INFO org.apache.zookeeper.server.ZooKeeperServer: >>>>> Expiring session 0x1262918396b0004 >>>>> 2010-01-13 11:12:06,002 INFO >>>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session >>>>> termination request for id: 0x1262918396b0004 >>>>> >>>>> I've tried wiping the whole setup, including the zookeeper files that >>>>> Hbase >>>>> puts in /tmp and formatting HDFS. But I get the same problem again. I'm >>>>> not seeing any exceptions on the client-side. >>>>> >>>>> Thanks, >>>>> Adam >>>>> >>>>> >>> >>> > >
