Re: Non Hadoop scheduling frameworks

2010-08-23 Thread Ted Dunning
These are pretty easy to solve with ZK. Ephemerality, exclusive create, atomic update and file versions allow you to implement most of the semantics you need. I don't know of any recipes available for this, but they would be worthy additions to ZK. On Mon, Aug 23, 2010 at 11:33 PM, Todd Nine wr

Re: Parent nodes & multi-step transactions

2010-08-23 Thread Ted Dunning
My own opinion is that lots of these structure sorts of problems are solved by putting the structure into a single znode. Atomic creation and update come for free at that point and we can even make the node ephemeral which we can't really do if there are children. It is tempting to use children a

Re: Parent nodes & multi-step transactions

2010-08-23 Thread Dave Wright
For my $0.02, I really think it would be nice if ZK supported "lightweight transactions". By that, I simply mean that a batch of create/update/delete requests could be submitted in a single request, and be processed atomically (if any of the requests would fail, none are applied). I know transactio

Re: Non Hadoop scheduling frameworks

2010-08-23 Thread Todd Nine
Solving UC1 and UC2 via zookeeper or some other framework if one is recommended. We don't run Hadoop, just ZK and Cassandra as we don't have a need for map/reduce. I'm searching for any existing framework that can perform standard time based scheduling in a distributed environment. As I said ear

Re: Non Hadoop scheduling frameworks

2010-08-23 Thread Mahadev Konar
Hi Todd, Just to be clear, are you looking at solving UC1 and UC2 via zookeeper? Or is this a broader question for scheduling on cassandra nodes? For the latter this probably isnt the right mailing list. Thanks mahadev On 8/23/10 4:02 PM, "Todd Nine" wrote: Hi all, We're using Zookeeper

Non Hadoop scheduling frameworks

2010-08-23 Thread Todd Nine
Hi all, We're using Zookeeper for Leader Election and system monitoring. We're also using it for synchronizing our cluster wide jobs with barriers. We're running into an issue where we now have a single job, but each node can fire the job independently of others with different criteria in the

Re: Parent nodes & multi-step transactions

2010-08-23 Thread Gustavo Niemeyer
Hi Mahadev, >  Usually the paradigm I like to suggest is to have something like > > /A/init > > Every client watches for the existence of this node and this node is only > created after /A has been initialized with the creation of /A/C or other > stuff. > > Would that work for you? Yeah, this is

Re: Parent nodes & multi-step transactions

2010-08-23 Thread Mahadev Konar
Hi Gustavo, Usually the paradigm I like to suggest is to have something like /A/init Every client watches for the existence of this node and this node is only created after /A has been initialized with the creation of /A/C or other stuff. Would that work for you? Thanks mahadev On 8/23/10 7:

Parent nodes & multi-step transactions

2010-08-23 Thread Gustavo Niemeyer
Greetings, We (a development team at Canonical) are stumbling into a situation here which I'd be curious to understand what is the general practice, since I'm sure this is somewhat of a common issue. It's quite easy to describe it: say there's a parent node A somewhere in the tree. That node was