This log message is actually a bad message:
https://issues.apache.org/jira/browse/ZOOKEEPER-588
2010-03-16 23:40:17,459 WARN org.apache.zookeeper.server.Request: Ignoring
exception during toString
java.nio.BufferUnderflowException
In 3.3.0 we've removed it. Prior to that someone had wrapped some bad
tostring code with a try/catch rather than do it right. And then logged
as warning if the try/catch triggered. So this is just a bogus message,
it would have nothing to do with your real problem.
also note the messages that follow
2010-03-16 23:40:17,634 DEBUG
org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
sessionid:0x276a6c04f00000 type:createSession cxid:0x0 zxid:0x8900000006
txntype:-10 n/a
this indicates the actual operation was successful (session creation)
Still, there's a problem with ZK other than this issue right? If you can
provide the logs I'll take a look. Or if you find something else that's
hinky I'd be willing to provide feedback.
regarding this issue:
> Zookeeper doesn't need _that_ much ;)
JD is right. There was some issues initially when ZK was integrated with
Hbase, but most of that was due to GC in the ZK end-user client code,
not the ZK service/client itself per say. We worked with hbase to
address this and afaik it's pretty good now.
ZK doesn't need much to run. See this document for details on minimal
(single cpu) and we can still handle 15k+ ops/second - way more than
hbase needs at the moment. http://bit.ly/4ekN8G Obviously if you do
something like run hadoop/hbase/zkserver on an oversubscribed "small"
ec2 node you're bound to see issues, but in general the ZK service
itself doesn't need much HP to run. See this for addl detail on types of
things "not to do" we've accumulated over time http://bit.ly/5WwS44 ;-)
Regards,
Patrick
Jean-Daniel Cryans wrote:
Zookeeper doesn't need _that_ much ;)
You say you are loosing your zk server... can we see the error? Pastebin?
Thx
J-D
On Tue, Mar 16, 2010 at 11:48 PM, Michael Segel
<michael_se...@hotmail.com> wrote:
Unfortunately I can't up the ulimit easily. :-( I'll have to get an admin to do
that.
I did update the xceivers and set it to 2048 based on something I saw.
But I'm losing my zookeeper on the node. Getting an IO error.
I had the handler count high at 50 but reset it back down to 25 (default value)
From what I've read, I definitely will move the zookeeper nodes when I can find
additional machines to add to the cluster.
Again any input welcome.
Thx
-Mike
Date: Tue, 16 Mar 2010 20:30:27 -0800
Subject: Re: Weird HBase Shell issue with count
From: st...@duboce.net
To: hbase-user@hadoop.apache.org
Oh, you've read the 'getting started' and the hbase requirements where
it specifies upping ulimit and xceivers in your cluster?
St.Ack
On Tue, Mar 16, 2010 at 8:29 PM, Stack <st...@duboce.net> wrote:
Is DEBUG enabled in the log4j.properties that the client can see? If
not, enable it. If so, can you see the regions loading as the count
progresses? Which region does it stop at? Can you try to do a get on
its startkey? Does it work?
St.Ack
On Tue, Mar 16, 2010 at 8:25 PM, Michael Segel
<michael_se...@hotmail.com> wrote:
Ok,
Still trying to track down some issues.
I opened up an hbase shell and decided to use count to count the number of
rows in a table.
As it was running, count was flying along until it hit 150,000 then stopped.
Just stood there, nothing.
I started to check the other nodes in the cloud to see what is happening and
the load on the data nodes, which are also region servers jumped up, where one
1 node jumped up to 2.71 ... other nodes saw some jump but again it doesn't
make sense why the count suddenly died.
I'm going to check the logs, but has anyone seen something like this?
Thx
-Mike
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_1
_________________________________________________________________
The New Busy is not the old busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID27925::T:WLMTAGL:ON:WL:en-US:WM_HMP:032010_3
