That's only the first part. It could be that the user was still pointing to his local host and not to the developer's cloud. (A common mistake.)
But it lead to my checking the logs on the quorum leader and seeing the error. (Scroll down to the bottom.) I have to apologize, I tried to format the e-mail and unfortunately the mailing list kind of screwed that up. The error(s) that concern me is: > > Checking the logs, I found that I had an error ... KeeperErrorCode = NodeExistsI set the logging to DEBUG and restarted both HBase and Hadoop. (When both down checked that no zombie processes were left running.) > > Now I get the following block in the zookeeper log:2010-02-26 14:35:36,435 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown n/a > > org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = > > NodeExists > > at > > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) > > at > > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) > > 2010-02-26 14:35:36,435 DEBUG org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a > > 2010-02-26 14:35:36,436 DEBUG org.apache.zookeeper.server.quorum.Leader: Proposing:: sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a > > 2010-02-26 14:35:36,439 DEBUG org.apache.zookeeper.server.quorum.Leader: > > Count for zxid: 0x6300000019 is 1 > > 2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.Leader: > > Count for zxid: 0x6300000019 is 2 > > 2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.CommitProcessor: Committing request:: sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a > > 2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.Leader: > > outstanding is 0 > > 2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 txntype:-1 n/a Thx -Mike > Date: Fri, 26 Feb 2010 14:22:16 -0800 > Subject: Re: Need help in understanding this zookeeper error... > From: [email protected] > To: [email protected] > > Looks like your client wasn't configured with hbase.zookeeper.quorum > which defaults to localhost. > > J-D > > On Fri, Feb 26, 2010 at 2:15 PM, Michael Segel > <[email protected]> wrote: > > > > Setting up a development cluster. > > > > Using Cloudera's latest release which has HBase-20.3. > > We have 3 nodes running ZooKeeper which is managed by HBase. > > We have a quorum set up. > > > > One of the developers ran in to this error: > > 10/02/26 07:31:48 WARN zookeeper.ClientCnxn: Exception closing > > session 0x0 to > > [email protected]: Connection > > refused at > > sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)10/02/26 > > 07:31:48 WARN zookeeper.ClientCnxn: Ignoring exception > > during shutdown inputjava.nio.channels.ClosedChannelException at > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > > at > > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)10/02/26 > > 07:31:48 WARN zookeeper.ClientCnxn: Ignoring exception > > during shutdown outputjava.nio.channels.ClosedChannelException at > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > > at > > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) at > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)10/02/26 > > 07:31:49 WARN zookeeper.ZooKeeperWrapper: Failed to create > > /hbase -- check quorum servers, > > currently=localhost:2181org.apache.zookeeper.KeeperException$ConnectionLossException: > > KeeperErrorCode = ConnectionLoss for /hbase > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [SNIP] > > The strange thing is that we have a quorum but the error says > > currently=localhost. > > Checking the logs, I found that I had an error ... KeeperErrorCode = > > NodeExistsI set the logging to DEBUG and restarted both HBase and Hadoop. > > (When both down checked that no zombie processes were left running.) > > Now I get the following block in the zookeeper log:2010-02-26 14:35:36,435 > > WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when > > processing sessionid:0x270bfd523f0003 type:create cxid:0x2 > > zxid:0xfffffffffffffffe txntype:unknown n/a > > org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = > > NodeExists > > at > > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) > > at > > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) > > 2010-02-26 14:35:36,435 DEBUG > > org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: > > sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 > > txntype:-1 n/a > > 2010-02-26 14:35:36,436 DEBUG org.apache.zookeeper.server.quorum.Leader: > > Proposing:: sessionid:0x270bfd523f0003 type:create cxid:0x2 > > zxid:0x6300000019 txntype:-1 n/a > > 2010-02-26 14:35:36,439 DEBUG org.apache.zookeeper.server.quorum.Leader: > > Count for zxid: 0x6300000019 is 1 > > 2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.Leader: > > Count for zxid: 0x6300000019 is 2 > > 2010-02-26 14:35:36,445 DEBUG > > org.apache.zookeeper.server.quorum.CommitProcessor: Committing request:: > > sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 > > txntype:-1 n/a > > 2010-02-26 14:35:36,445 DEBUG org.apache.zookeeper.server.quorum.Leader: > > outstanding is 0 > > 2010-02-26 14:35:36,445 DEBUG > > org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: > > sessionid:0x270bfd523f0003 type:create cxid:0x2 zxid:0x6300000019 > > txntype:-1 n/a > > > > Now this happens when I run a command in the hbase shell. (Like list, > > status, ...) > > I'd like to understand what is happening and what I can do to fix this? > > I'm not sure why the user had Zookeeper errors showing local host but the > > larger concern is what is happening to cause this? > > Earlier I had i/o errors mentioning /hbase/root but they went away when I > > took everything down.Since this is still a development machine, I may just > > blow away /hbase in HDFS and see if I can come up clean. > > TIA. > > -Mike > > > > > > _________________________________________________________________ > > Hotmail: Free, trusted and rich email service. > > http://clk.atdmt.com/GBL/go/201469228/direct/01/ _________________________________________________________________ Hotmail: Trusted email with powerful SPAM protection. http://clk.atdmt.com/GBL/go/201469227/direct/01/
