Setting the heap size
Hi all, We are setting up a small Hadoop 13 node cluster running 1 HDFS master, 9 region severs for HBase and 3 map reduce nodes, and are just installing zookeeper to perform the HBase coordination and to manage a few simple process locks for other tasks we run. Could someone please advise what kind on heap we should give to our single ZK node and also (ahem) how does one actually set this? It's not immediately obvious in the docs or config. Thanks, Tim
Re: Setting the heap size
On Thu, Oct 28, 2010 at 2:52 AM, Tim Robertson timrobertson...@gmail.com wrote: We are setting up a small Hadoop 13 node cluster running 1 HDFS master, 9 region severs for HBase and 3 map reduce nodes, and are just installing zookeeper to perform the HBase coordination and to manage a few simple process locks for other tasks we run. Could someone please advise what kind on heap we should give to our single ZK node and also (ahem) how does one actually set this? It's not immediately obvious in the docs or config. The amount of heap necessary will be dependent on the application(s) using ZK, also configuration of the heap is dependent on what packaging you are using to start ZK. Are you using zkServer.sh from our distribution? If so then you probably want to set JVMFLAGS env variable. We pass this through to the jvm, see -Xmx in the man page (http://www.manpagez.com/man/1/java/) Given this is Hbase (which I'm reasonably familiar with) the default heap should be fine. However you might want to check with the Hbase team on that. I'd also encourage you to enter a JIRA on the (lack of) doc issue you highlighted: https://issues.apache.org/jira/browse/ZOOKEEPER Regards, Patrick
Re: Setting the heap size
Tim, one other thing you might want to be aware of: http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_supervision Patrick On Thu, Oct 28, 2010 at 9:11 AM, Patrick Hunt ph...@apache.org wrote: On Thu, Oct 28, 2010 at 2:52 AM, Tim Robertson timrobertson...@gmail.com wrote: We are setting up a small Hadoop 13 node cluster running 1 HDFS master, 9 region severs for HBase and 3 map reduce nodes, and are just installing zookeeper to perform the HBase coordination and to manage a few simple process locks for other tasks we run. Could someone please advise what kind on heap we should give to our single ZK node and also (ahem) how does one actually set this? It's not immediately obvious in the docs or config. The amount of heap necessary will be dependent on the application(s) using ZK, also configuration of the heap is dependent on what packaging you are using to start ZK. Are you using zkServer.sh from our distribution? If so then you probably want to set JVMFLAGS env variable. We pass this through to the jvm, see -Xmx in the man page (http://www.manpagez.com/man/1/java/) Given this is Hbase (which I'm reasonably familiar with) the default heap should be fine. However you might want to check with the Hbase team on that. I'd also encourage you to enter a JIRA on the (lack of) doc issue you highlighted: https://issues.apache.org/jira/browse/ZOOKEEPER Regards, Patrick
Client seeing wrong data on nodeDataChanged
I'm trying to debug an issue that maybe you fellas have some ideas for figuring. In short: Client 1 updates a znode setting its content to X, then X again, then Y, and then finally it deletes the znode. Client 1 is watching the znode and I can see that its getting three nodeDataChanged events and a nodeDeleted. Client 2 is also watching the znode. It gets notified three times: two nodeDataChanged events(only) and a nodeDeleted event. I'd expect 3 nodeDataChanged events but understand a client might skip states. The problem is that when client 2 looks at the data in the znode on nodeDataChanged, for both cases the data is Y. Not X and then Y, but Y both times. This is unexpected. This is 3.3.1 on a 5 node ensemble. I have full zk logging enabled. Would it help posting these? St.Ack
Getting a node exists code on a sequence create
HI everyone, Is there any situation in which creating a new ZK node with the SEQUENCE flag should result in a node exists error? I'm seeing this happening after a failure of a ZK node that appeared to have been the master; when the new master takes over, my app is unable to create a new SEQUENCE node under an existing parent node. I'm using Zookeeper 3.2.2. Here's a representative log snippet: -- 3050756 [ProcessThread:-1] TRACE org.apache.zookeeper.server.PrepRequestProcessor - :Psessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0xfffe txntype:unknown /zkrsm/_record 3050756 [ProcessThread:-1] WARN org.apache.zookeeper.server.PrepRequestProcessor - Got exception when processing sessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) 3050756 [ProcessThread:-1] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Processing request:: sessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0x5027e txntype:-1 n/a 3050756 [ProcessThread:-1] DEBUG org.apache.zookeeper.server.quorum.Leader - Proposing:: sessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0x5027e txntype:-1 n/a 3050756 [SyncThread:0] TRACE org.apache.zookeeper.server.quorum.Leader - Ack zxid: 0x5027e 3050757 [SyncThread:0] TRACE org.apache.zookeeper.server.quorum.Leader - outstanding proposal: 0x5027e 3050757 [SyncThread:0] TRACE org.apache.zookeeper.server.quorum.Leader - outstanding proposals all 3050757 [SyncThread:0] DEBUG org.apache.zookeeper.server.quorum.Leader - Count for zxid: 0x5027e is 1 3050757 [FollowerHandler-/172.16.0.28:48776] TRACE org.apache.zookeeper.server.quorum.Leader - Ack zxid: 0x5027e 3050757 [FollowerHandler-/172.16.0.28:48776] TRACE org.apache.zookeeper.server.quorum.Leader - outstanding proposal: 0x5027e 3050757 [FollowerHandler-/172.16.0.28:48776] TRACE org.apache.zookeeper.server.quorum.Leader - outstanding proposals all 3050757 [FollowerHandler-/172.16.0.28:48776] DEBUG org.apache.zookeeper.server.quorum.Leader - Count for zxid: 0x5027e is 2 3050757 [FollowerHandler-/172.16.0.28:48776] DEBUG org.apache.zookeeper.server.quorum.CommitProcessor - Committing request:: sessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0x5027e txntype:-1 n/a 3050757 [CommitProcessor:0] DEBUG org.apache.zookeeper.server.FinalRequestProcessor - Processing request:: sessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0x5027e txntype:-1 n/a 3050757 [CommitProcessor:0] TRACE org.apache.zookeeper.server.FinalRequestProcessor - :Esessionid:0x12bf518350f0001 type:create cxid:0x4cca0691 zxid:0x5027e txntype:-1 n/a 3050757 [FollowerHandler-/172.16.0.28:41062] TRACE org.apache.zookeeper.server.quorum.Leader - Ack zxid: 0x5027e 3050757 [FollowerHandler-/172.16.0.28:41062] TRACE org.apache.zookeeper.server.quorum.Leader - outstanding proposals all 3050757 [FollowerHandler-/172.16.0.28:41062] DEBUG org.apache.zookeeper.server.quorum.Leader - outstanding is 0 -- I'm still a n00b at understanding ZK log messages, so maybe there's something obvious going on. I looked in the JIRA and did my best to search the mailing list archives, but couldn't find anything related to this. Any ideas? Thanks very much, Jeremy
Re: Client seeing wrong data on nodeDataChanged
Client 2 is not guaranteed to see X if it doesn't get to asking before the value has been updated to Y. On Thu, Oct 28, 2010 at 2:39 PM, Stack st...@duboce.net wrote: Client 2 is also watching the znode. It gets notified three times: two nodeDataChanged events(only) and a nodeDeleted event. I'd expect 3 nodeDataChanged events but understand a client might skip states. The problem is that when client 2 looks at the data in the znode on nodeDataChanged, for both cases the data is Y. Not X and then Y, but Y both times. This is unexpected.
Re: Client seeing wrong data on nodeDataChanged
On Thu, Oct 28, 2010 at 7:32 PM, Ted Dunning ted.dunn...@gmail.com wrote: Client 2 is not guaranteed to see X if it doesn't get to asking before the value has been updated to Y. Right, but I wouldn't expect the watch to be triggered twice with value Y. Anyways, I think we have a handle on whats going on: at the time of the above incident, the master process is experiencing a flood of zk changes and our thought is that we're not paying sufficient attention to the order of receipt. Will be back if this is not the issue. Thanks, St.Ack
Re: Client seeing wrong data on nodeDataChanged
On Thu, Oct 28, 2010 at 9:56 PM, Stack st...@duboce.net wrote: On Thu, Oct 28, 2010 at 7:32 PM, Ted Dunning ted.dunn...@gmail.com wrote: Client 2 is not guaranteed to see X if it doesn't get to asking before the value has been updated to Y. Right, but I wouldn't expect the watch to be triggered twice with value Y. It may not have been. It may have been triggered with the change from x to x, but by the time the client got around to looking when the value was already y. The trigger and the value are not connected.
