Hi again !
Well.. to make this easier, Ive created a script which greps the hotspots
in log files out and cleans them, restarting the entire cluster.
Overall, it appears that my region servers cant see my hbase-master.... due
to "closed connections", or something of that sort.
These errors are now reproducible so hopefully someone can shed some light.
----------------->
EXCEPTIONS
hbase-master
-------------------------------------
tcp 0 0 :::3888 :::*
LISTEN 0 172888 4674/java
2013-05-21 18:59:16,537 INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of
this process is 4729@hbase-master
2013-05-21 18:59:16,562 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server hbase-master/192.168.122.200:2222. Will not
attempt to authenticate using SASL (unknown error)
2013-05-21 18:59:16,628 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to hbase-master/192.168.122.200:2222, initiating
session
2013-05-21 18:59:16,669 INFO org.apache.zookeeper.ClientCnxn: Unable to
read additional data from server sessionid 0x0, likely server has closed
socket, closing socket connection and attempting reconnect
2013-05-21 18:59:16,842 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase-master
2013-05-21 18:59:16,842 INFO org.apache.hadoop.hbase.util.RetryCounter:
Sleeping 2000ms before retry #1...
Tue May 21 18:59:15 EDT 2013 Starting regionserver on hbase-master
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
--
2013-05-21 18:59:14,579 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
(n.peerEPoch), LOOKING (my state)
2013-05-21 18:59:14,580 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection
request /192.168.122.203:33665
2013-05-21 18:59:14,580 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0
(n.peerEPoch), LOOKING (my state)
2013-05-21 18:59:14,582 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while
waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
--
2013-05-21 18:59:16,146 INFO org.apache.zookeeper.server.quorum.Learner:
Getting a diff from the leader 0x0
2013-05-21 18:59:16,190 INFO
org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0
to /tmp/hbase-root/zookeeper/version-2/snapshot.0
2013-05-21 18:59:16,524 INFO
org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0
to /tmp/hbase-root/zookeeper/version-2/snapshot.0
2013-05-21 18:59:16,594 INFO
org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
connection from /192.168.122.200:40223
2013-05-21 18:59:16,661 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x0 due to java.io.IOException:
ZooKeeperServer not running
2013-05-21 18:59:16,661 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /192.168.122.200:40223 (no session
established for client)
---------------------------------------
hbase-regionserver1
-------------------------------------
tcp 0 0 :::3888 :::*
LISTEN 0 94882 2306/java
2013-05-21 18:59:14,442 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port:
0.0.0.0/0.0.0.0:3888
2013-05-21 18:59:14,451 INFO org.apache.zookeeper.server.quorum.QuorumPeer:
LOOKING
2013-05-21 18:59:14,453 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id
= 1, proposed zxid=0x0
2013-05-21 18:59:14,458 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
0 at election address hbase-master/192.168.122.200:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:679)
2013-05-21 18:59:14,461 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
(n.peerEPoch), LOOKING (my state)
2013-05-21 18:59:14,467 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
2 at election address hbase-regionserver2/192.168.122.202:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:679)
2013-05-21 18:59:14,468 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
3 at election address hbase-regionserver3/192.168.122.203:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
at java.lang.Thread.run(Thread.java:679)
2013-05-21 18:59:14,498 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection
request /192.168.122.203:57959
2013-05-21 18:59:14,508 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0
(n.peerEPoch), LOOKING (my state)
2013-05-21 18:59:14,508 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
0 at election address hbase-master/192.168.122.200:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:679)
2013-05-21 18:59:14,509 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
(n.peerEPoch), LOOKING (my state)
2013-05-21 18:59:14,510 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
2 at election address hbase-regionserver2/192.168.122.202:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
2013-05-21 18:59:14,801 INFO org.apache.zookeeper.server.ZooKeeperServer:
Server environment:user.dir=/root/hbaseinstall/hbase-0.94.7
2013-05-21 18:59:14,802 INFO org.apache.zookeeper.server.ZooKeeperServer:
Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout
180000 datadir /tmp/hbase-root/zookeeper/version-2 snapdir
/tmp/hbase-root/zookeeper/version-2
2013-05-21 18:59:14,802 INFO org.apache.zookeeper.server.quorum.Learner:
FOLLOWING - LEADER ELECTION TOOK - 349
2013-05-21 18:59:14,804 WARN org.apache.zookeeper.server.quorum.Learner:
Unexpected exception, tries=0, connecting to hbase-regionserver3/
192.168.122.203:2888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
---------------------------------------
hbase-regionserver2
-------------------------------------
tcp 0 0 :::3888 :::*
LISTEN 0 95070 2364/java
2013-05-21 18:59:14,511 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port:
0.0.0.0/0.0.0.0:3888
2013-05-21 18:59:14,517 INFO org.apache.zookeeper.server.quorum.QuorumPeer:
LOOKING
2013-05-21 18:59:14,518 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id
= 2, proposed zxid=0x0
2013-05-21 18:59:14,522 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
0 at election address hbase-master/192.168.122.200:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
2013-05-21 18:59:14,788 INFO org.apache.zookeeper.server.ZooKeeperServer:
Server environment:user.dir=/root/hbaseinstall/hbase-0.94.7
2013-05-21 18:59:14,790 INFO org.apache.zookeeper.server.ZooKeeperServer:
Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout
180000 datadir /tmp/hbase-root/zookeeper/version-2 snapdir
/tmp/hbase-root/zookeeper/version-2
2013-05-21 18:59:14,790 INFO org.apache.zookeeper.server.quorum.Learner:
FOLLOWING - LEADER ELECTION TOOK - 272
2013-05-21 18:59:14,794 WARN org.apache.zookeeper.server.quorum.Learner:
Unexpected exception, tries=0, connecting to hbase-regionserver3/
192.168.122.203:2888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
---------------------------------------
hbase-regionserver3
-------------------------------------
tcp 0 0 :::3888 :::*
LISTEN 0 160970 2411/java
2013-05-21 18:59:14,481 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port:
0.0.0.0/0.0.0.0:3888
2013-05-21 18:59:14,488 INFO org.apache.zookeeper.server.quorum.QuorumPeer:
LOOKING
2013-05-21 18:59:14,490 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id
= 3, proposed zxid=0x0
2013-05-21 18:59:14,495 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
0 at election address hbase-master/192.168.122.200:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
at java.lang.Thread.run(Thread.java:679)
2013-05-21 18:59:14,507 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to
2 at election address hbase-regionserver2/192.168.122.202:3888
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
--
2013-05-21 18:59:14,539 INFO
org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3
(n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0
(n.peerEPoch), LOOKING (my state)
2013-05-21 18:59:14,563 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection
request /192.168.122.200:42394
2013-05-21 18:59:14,574 INFO
org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection
request /192.168.122.200:42395
2013-05-21 18:59:14,575 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while
waiting for message on queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
2013-05-21 18:59:14,576 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Send worker leaving
thread
2013-05-21 18:59:14,576 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Connection broken for
id 0, my id = 3, error =
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.net.SocketInputStream.read(SocketInputStream.java:199)
at java.io.DataInputStream.readInt(DataInputStream.java:388)
---------------------------------------
On Tue, May 21, 2013 at 5:33 PM, Jean-Daniel Cryans <jdcry...@apache.org>wrote:
> I see:
>
> 2013-05-21 17:15:07,914 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN,
> server=hbase-regionserver1,60020,1369170595340, region=70236052/-ROOT-
>
> Over and over. Look in the region server logs, you should see fat
> stack traces on why it's failing to open the -ROOT- region. Maybe
> related to your gluster setup.
>
> J-D
>
> On Tue, May 21, 2013 at 2:18 PM, Jay Vyas <jayunit...@gmail.com> wrote:
> > https://gist.github.com/anonymous/5623327 <-- all logs for starting up
> > hbase and master
> >
> >
> > On Tue, May 21, 2013 at 5:13 PM, Mohammad Tariq <donta...@gmail.com>
> wrote:
> >
> >> No prob. I was referring to this :
> >>
> >> 127.0.0.1 hbase-master
> >> 192.168.122.200 hbase-master
> >>
> >> I was thinking that this is your HBase master. Correct me if i'm wrong.
> >> Could you please show me your logs?
> >>
> >> Warm Regards,
> >> Tariq
> >> cloudfront.blogspot.com
> >>
> >>
> >> On Wed, May 22, 2013 at 2:36 AM, Jay Vyas <jayunit...@gmail.com> wrote:
> >>
> >> > Hmmm... what do you mean "have your hostname in there"? sorry -- just
> >> > curious about which hostname you are referring to...?
> >> >
> >> > Im now getting a new exception:
> >> >
> >> > 13/05/21 17:02:44 INFO
> >> client.HConnectionManager$HConnectionImplementation:
> >> > getMaster attempt 2 of 7 failed; retrying after sleep of 1002
> >> >
> >> >
> >> >
> >> > On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq <donta...@gmail.com>
> >> > wrote:
> >> >
> >> > > OK..You already have your hostname in there. But it is appearing
> twice.
> >> > > Comment out "127.0.0.1 hbase-master". This might be a reason. I did
> not
> >> > > notice that you are on a distributed setup. RS IPs and hostnames are
> >> > fine.
> >> > >
> >> > > Warm Regards,
> >> > > Tariq
> >> > > cloudfront.blogspot.com
> >> > >
> >> > >
> >> > > On Wed, May 22, 2013 at 2:21 AM, Jay Vyas <jayunit...@gmail.com>
> >> wrote:
> >> > >
> >> > > > Hi kevin : So you don't have any region servers defined in your
> >> > > /etc/hosts
> >> > > > ?
> >> > > >
> >> > > >
> >> > > > On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq <
> donta...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Sorry, my bad. By that I meant "127.0.0.1 hostname"..To me it
> >> > seems
> >> > > > like
> >> > > > > HBase is not able to connect to localhost using 127.0.0.1
> >> > > > >
> >> > > > > Warm Regards,
> >> > > > > Tariq
> >> > > > > cloudfront.blogspot.com
> >> > > > >
> >> > > > >
> >> > > > > On Wed, May 22, 2013 at 2:12 AM, Jay Vyas <jayunit...@gmail.com
> >
> >> > > wrote:
> >> > > > >
> >> > > > > > Thanks, but adding 127.0.0.1 localhost to the top seems
> >> > redundant...
> >> > > > > > right? I did so but still no luck :(.
> >> > > > > >
> >> > > > > > 1) OS? This is fedora 16.....
> >> > > > > >
> >> > > > > > 2) any thoughts on why the "PleaseHoldException" is being
> >> > triggered ?
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq <
> >> > donta...@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > OS?Add "127.0.0.1 localhost" and see if it makes any
> >> > difference.
> >> > > > > > >
> >> > > > > > > Warm Regards,
> >> > > > > > > Tariq
> >> > > > > > > cloudfront.blogspot.com
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Wed, May 22, 2013 at 1:57 AM, Jay Vyas <
> >> jayunit...@gmail.com>
> >> > > > > wrote:
> >> > > > > > >
> >> > > > > > > > #This is my /etc/hosts file ---
> >> > > > > > > > 127.0.0.1 localhost localhost.localdomain localhost4
> >> > > > > > > > localhost4.localdomain4
> >> > > > > > > > #::1 localhost localhost.localdomain localhost6
> >> > > > > > > > localhost6.localdomain6
> >> > > > > > > > 127.0.0.1 hbase-master
> >> > > > > > > > 192.168.122.200 hbase-master
> >> > > > > > > > 192.168.122.201 hbase-regionserver1
> >> > > > > > > > 192.168.122.202 hbase-regionserver2
> >> > > > > > > > 192.168.122.203 hbase-regionserver3
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq <
> >> > > > donta...@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hello Jay,
> >> > > > > > > > >
> >> > > > > > > > > Please change the line containing "127.0.1.1" in
> your
> >> > > > > > /etc/hosts
> >> > > > > > > to
> >> > > > > > > > > "127.0.0.1" and see if it works.
> >> > > > > > > > >
> >> > > > > > > > > Warm Regards,
> >> > > > > > > > > Tariq
> >> > > > > > > > > cloudfront.blogspot.com
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Wed, May 22, 2013 at 1:47 AM, Jay Vyas <
> >> > > jayunit...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi folks:
> >> > > > > > > > > >
> >> > > > > > > > > > Hope someone can shed some light on this -
> >> > > > > > > > > >
> >> > > > > > > > > > I cannot run "hbase shell" create commands because of
> the
> >> > > > > > > > > > PleaseHoldException on a fresh install of hbase. Im
> not
> >> > > > finding
> >> > > > > > much
> >> > > > > > > > in
> >> > > > > > > > > > the error logs, and all nodes appear to be up and
> >> running,
> >> > > > > > including
> >> > > > > > > > > hbase
> >> > > > > > > > > > master.
> >> > > > > > > > > >
> >> > > > > > > > > > Version : hbase-0.94.7
> >> > > > > > > > > >
> >> > > > > > > > > > Error: I am getting the "Please hold exception"... on
> my
> >> > > hbase
> >> > > > > > shell.
> >> > > > > > > > > When
> >> > > > > > > > > > running "create table 't1','f1'"... But...
> >> > > > > > > > > >
> >> > > > > > > > > > But wait :) theres more ! ... clearly, the hbase
> master
> >> is
> >> > > > > > running :
> >> > > > > > > > > >
> >> > > > > > > > > > [root@hbase-master ~]# jps
> >> > > > > > > > > > 11896 HQuorumPeer
> >> > > > > > > > > > 12914 Jps
> >> > > > > > > > > > 9894 Main
> >> > > > > > > > > > 5879 Main
> >> > > > > > > > > > ******************
> >> > > > > > > > > > 12279 HMaster
> >> > > > > > > > > > ******************
> >> > > > > > > > > > 5779 Main
> >> > > > > > > > > > 11714 ZKServerTool
> >> > > > > > > > > > 12058 HRegionServer
> >> > > > > > > > > > 12860 Main
> >> > > > > > > > > > 8369 Main
> >> > > > > > > > > >
> >> > > > > > > > > > And finally - here is a dump of the output from the
> shell
> >> > ---
> >> > > > any
> >> > > > > > > > > thoughts?
> >> > > > > > > > > >
> >> > > > > > > > > > [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/
> >> > > > > > > > > > hbase shell -d <<EOF
> >> > > > > > > > > > create 't1','f1'
> >> > > > > > > > > > EOF
> >> > > > > > > > > >
> >> > > > > > > > > > Setting DEBUG log level...
> >> > > > > > > > > >
> >> > > > > > > > > > HBase Shell; enter 'help<RETURN>' for list of
> supported
> >> > > > commands.
> >> > > > > > > > > > Type "exit<RETURN>" to leave the HBase Shell
> >> > > > > > > > > > Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013
> >> > > > > > > > > >
> >> > > > > > > > > > create 't1','f1'
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:zookeeper.version=3.4.5-1392090, built on
> >> > > > 09/30/2012
> >> > > > > > > 17:52
> >> > > > > > > > > GMT
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > environment:
> >> > > > > > > > host.name
> >> > > > > > > > > > =localhost
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:java.version=1.6.0_24
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:java.vendor=Sun Microsystems Inc.
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > >
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> environment:java.class.path=/root/hbaseinstall/hbase-0.94.7/bin/../conf:/usr/lib/jvm/jre-1.6.0-openjdk.x86_64/lib/tools.jar:/root/hbaseinstall/hbase-0.94.7/bin/..:/root/hbaseinstall/hbase-0.94.7/bin/../hbase-0.94.7.jar:/root/hbaseinstall/hbase-0.94.7/bin/../hbase-0.94.7-tests.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/activation-1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/asm-3.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/avro-1.5.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/avro-ipc-1.5.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-beanutils-1.7.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-beanutils-core-1.8.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-cli-1.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-codec-1.4.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-collections-3.2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-configuration-1.6.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-digester-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-el-1.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-httpclient-3.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-io-2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-lang-2.5.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-logging-1.1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-math-2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-net-1.4.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/core-3.1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/glusterfs-0.0.1-20130520-1810.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/guava-11.0.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/hadoop-core-1.0.4.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/high-scale-lib-1.1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/httpclient-4.1.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/httpcore-4.1.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-core-asl-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-jaxrs-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-mapper-asl-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-xc-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jamon-runtime-2.3.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jasper-compiler-5.5.23.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jasper-runtime-5.5.23.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jaxb-api-2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jaxb-impl-2.2.3-1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jersey-core-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jersey-json-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jersey-server-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jettison-1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jetty-6.1.26.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jetty-util-6.1.26.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jruby-complete-1.6.5.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jsp-2.1-6.1.14.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jsp-api-2.1-6.1.14.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jsr305-1.3.9.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/junit-4.10-HBASE-1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/libthrift-0.8.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/log4j-1.2.16.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/metrics-core-2.1.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/netty-3.2.4.Final.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/protobuf-java-2.4.0a.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/servlet-api-2.5-6.1.14.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/slf4j-api-1.4.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/slf4j-log4j12-1.4.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/snappy-java-1.0.3.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/stax-api-1.0.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/velocity-1.7.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/xmlenc-0.52.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/zookeeper-3.4.5.jar:
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> environment:java.library.path=/root/hbaseinstall/hbase-0.94.7/bin/../lib/native/Linux-amd64-64
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:java.io.tmpdir=/tmp
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:java.compiler=<NA>
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > environment:
> >> > > > > > > os.name
> >> > > > > > > > > > =Linux
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:os.arch=amd64
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:os.version=3.1.0-7.fc16.x86_64
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > environment:
> >> > > > > > > > user.name
> >> > > > > > > > > > =root
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:user.home=/root
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
> >> > > > > > > > > > environment:user.dir=/root
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Initiating
> >> > client
> >> > > > > > > > connection,
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> connectString=hbase-regionserver3:2222,hbase-regionserver2:2222,hbase-regionserver1:2222,hbase-master:2222
> >> > > > > > > > > > sessionTimeout=180000 watcher=hconnection
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn:
> >> > > > > > > > > > zookeeper.disableAutoWatchReset is false
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.RecoverableZooKeeper:
> >> The
> >> > > > > > identifier
> >> > > > > > > > of
> >> > > > > > > > > > this process is 12762@hbase-master
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ClientCnxn: Opening
> >> socket
> >> > > > > > > connection
> >> > > > > > > > to
> >> > > > > > > > > > server hbase-regionserver3/192.168.122.203:2222. Will
> >> not
> >> > > > > attempt
> >> > > > > > to
> >> > > > > > > > > > authenticate using SASL (unknown error)
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ClientCnxn: Socket
> >> > > connection
> >> > > > > > > > > established
> >> > > > > > > > > > to hbase-regionserver3/192.168.122.203:2222,
> initiating
> >> > > > session
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Session
> >> > > > > establishment
> >> > > > > > > > > request
> >> > > > > > > > > > sent on hbase-regionserver3/192.168.122.203:2222
> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ClientCnxn: Session
> >> > > > > establishment
> >> > > > > > > > > complete
> >> > > > > > > > > > on server hbase-regionserver3/192.168.122.203:2222,
> >> > > sessionid
> >> > > > =
> >> > > > > > > > > > 0x33ec8aac0f40002, negotiated timeout = 180000
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 1,3 replyHeader::
> >> 1,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master/hbaseid,F response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> s{17179869216,51539607572,1368812077009,1369166263798,12,0,0,0,59,0,17179869216}
> >> > > > > > > > > >
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 2,4 replyHeader::
> >> 2,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master/hbaseid,F response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> #ffffffff0001231323237394068626173652d6d617374657234333565333230302d613533612d346634662d393838342d393735633736333966343233,s{17179869216,51539607572,1368812077009,1369166263798,12,0,0,0,59,0,17179869216}
> >> > > > > > > > > >
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 3,3 replyHeader::
> >> 3,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master/master,T response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> s{51539607570,51539607570,1369166263477,1369166263477,0,0,0,17672084227293186,54,0,51539607570}
> >> > > > > > > > > >
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 4,4 replyHeader::
> >> 4,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master/master,T response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> #ffffffff0001231323237394068626173652d6d6173746572006c6f63616c686f73742c36303030302c31333639313636323633313032,s{51539607570,51539607570,1369166263477,1369166263477,0,0,0,17672084227293186,54,0,51539607570}
> >> > > > > > > > > >
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 5,3 replyHeader::
> >> 5,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master/root-region-server,T response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> s{51539607579,51539607579,1369166269726,1369166269726,0,0,0,0,52,0,51539607579}
> >> > > > > > > > > >
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 6,4 replyHeader::
> >> 6,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master/root-region-server,T response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> #ffffffff0001231323035384068626173652d6d61737465726c6f63616c686f73742c36303032302c31333639313636323335353932,s{51539607579,51539607579,1369166269726,1369166269726,0,0,0,0,52,0,51539607579}
> >> > > > > > > > > >
> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
> >> reply
> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
> >> > > > > > serverPath:null
> >> > > > > > > > > > finished:false header:: 7,3 replyHeader::
> >> 7,51539607605,0
> >> > > > > > request::
> >> > > > > > > > > > '/hbase-master,F response::
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> s{17179869186,17179869186,1368811379390,1368811379390,0,88,0,0,0,12,51539607579}
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > --
> >> > > > > > > > > > Jay Vyas
> >> > > > > > > > > > http://jayunit100.blogspot.com
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Jay Vyas
> >> > > > > > > > http://jayunit100.blogspot.com
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Jay Vyas
> >> > > > > > http://jayunit100.blogspot.com
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Jay Vyas
> >> > > > http://jayunit100.blogspot.com
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Jay Vyas
> >> > http://jayunit100.blogspot.com
> >> >
> >>
> >
> >
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
