Non Hadoop scheduling frameworks

2010-08-23 Thread Todd Nine
Hi all, We're using Zookeeper for Leader Election and system monitoring. We're also using it for synchronizing our cluster wide jobs with barriers. We're running into an issue where we now have a single job, but each node can fire the job independently of others with different criteria in the

Re: Non Hadoop scheduling frameworks

2010-08-23 Thread Todd Nine
> Just to be clear, are you looking at solving UC1 and UC2 via zookeeper? Or > is this a broader question for scheduling on cassandra nodes? For the latter > this probably isnt the right mailing list. > > Thanks > mahadev > > > On 8/23/10 4:02 PM, "Todd Nine" wr

Re: Non Hadoop scheduling frameworks

2010-08-25 Thread Todd Nine
ming more research I think job semantics such as partitioning and parallel processing are outside the scope of how the scheduler should work. Those semantics are more internal to the job itself, and I think they should remain outside of the scope of this project. todd SENIOR SOFTWARE ENGINEE

What roles do "even" nodes play in the ensamble

2010-08-25 Thread Todd Nine
Hey guys, Forgive me if this is documented somewhere, but I can't find an answer. Our application is not enormous, so we will be using 4 application nodes that will also initially run Zookeeper. As our load increases, Zookeeper will be moved to nodes that only run ZK and no other processes. Gi

Re: What roles do "even" nodes play in the ensamble

2010-08-25 Thread Todd Nine
ENIOR SOFTWARE ENGINEER todd nine| spidertracks ltd | 117a the square po box 5203 | palmerston north 4441 | new zealand P: +64 6 353 3395 | M: +64 210 255 8576 E: t...@spidertracks.co.nz W: www.spidertracks.com On Wed, 2010-08-25 at 17:57 -0400, Dave Wright wrote: > > > > 1

Re: What roles do "even" nodes play in the ensamble

2010-08-25 Thread Todd Nine
but from my > > understanding all 4 nodes participate in the voting and you need a majority > > of 3 to elect a leader. > > > > -Dave > > > > On Wed, Aug 25, 2010 at 6:09 PM, Todd Nine > > wrote: > > > > > Thanks for that Dave. If I do no

Re: What roles do "even" nodes play in the ensamble

2010-08-25 Thread Todd Nine
en or odd number. With zookeeper there is no voting on > each > transaction, only leader changes. > > -Dave Wright > > > > On Wed, Aug 25, 2010 at 6:22 PM, Todd Nine > wrote: > > Do I get any read performance increase (similar to an > observer) since > > the node will not have a voting role? > > > > > > > >

Re: What roles do "even" nodes play in the ensamble

2010-08-25 Thread Todd Nine
'more things available to go wrong'). > > > If you have four servers and one dies, the 'majority' that still needs > to be alive is still three - it doesn't drop down to two. The majority > is of all voting servers, alive or dead. > > > Hope this hel

Receiving create events for self with synchronous create

2010-08-26 Thread Todd Nine
Hi all, I'm running into a strange issue I could use a hand with. I've implemented leader election, and this is working well. I'm now implementing a follower queue with ephemeral nodes. I have an interface IClusterManager which simply has the api "clusterChanged". I don't care if nodes are ad

Re: Receiving create events for self with synchronous create

2010-08-26 Thread Todd Nine
able to make out the sequence > of steps. >Can you be more clear on what you are trying to do in terms of > zookeeper api? > > Thanks > mahadev > On 8/26/10 5:58 PM, "Todd Nine" wrote: > > > Hi all, > I'm running into a stra

Strage issue with chroot

2010-08-30 Thread Todd Nine
Hi all, I'm running into a very strange issue with using chroot. I've defined the following URL in my java client. zookeeper.connections=localhost:2181/com/spidertracks/aviator When I run my unit tests against Zookeeper, all works as expected when I create the path "/cluster/election/node_xx"

getting created child on NodeChildrenChanged event

2010-08-31 Thread Todd Nine
Hi all, I'm writing a distributed queue monitoring class for our leader node in the cluster. We're queueing messages per input hardware device, this queue is then assigned to a node with the least load in our cluster. To do this, I maintain 2 Persistent Znode with the following format. data qu

Re: getting created child on NodeChildrenChanged event

2010-08-31 Thread Todd Nine
was chosen for Zookeeper > and I don't think it's likely to be implemented. > > -Dave Wright > > On Tue, Aug 31, 2010 at 3:49 AM, Todd Nine wrote: > > > Hi all, > > I'm writing a distributed queue monitoring class for our leader node in > > t