Re: GSoC 2010: ZooKeeper Failure Detector Module
Hi Abmar, cool project! I wrote an implementation of the Phi Accrual Detector which I would be happy to donate. It has the nice property that its parameters are based on expectation odds rather than something more arbitrary like # retries. You can steal it at http://github.com/arosien/failure. It works well but let me know if you have any suggestions. .. Adam On Wed, May 5, 2010 at 1:28 PM, Abmar Barros wrote: > Hello all, > > As Henry Robinson first introduced me, my name is Abmar Barros and I am one > of the GSoC accepted students for ZooKeeper. > In the next 4 months, I will be being mentored by Flavio Junqueira in order > to improve ZooKeeper's failure detector module. In this project I will > abstract the failure detection code to a separate module, in order to make > the code cleaner and to ease the implementation of new methods. Also, I will > implement several failure detectors and I will compare their appropriateness > for ZooKeeper. > > I have created a page ( > http://wiki.apache.org/hadoop/ZooKeeper/GSoCFailureDetector) for this > project in the ZooKeeper wiki and also there is a related JIRA issue ( > https://issues.apache.org/jira/browse/ZOOKEEPER-702). > > Suggestions and specific requirements are always welcome :) > > Best regards > > -- > Abmar Barros > MSc candidate on Computer Science at Federal University of Campina Grande - > www.ufcg.edu.br > OurGrid Team Member - www.ourgrid.org > Paraíba - Brazil >
[jira] Created: (ZOOKEEPER-751) Recipe heading refers to 'recoverable' but should be 'revocable'
Recipe heading refers to 'recoverable' but should be 'revocable'
Key: ZOOKEEPER-751
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-751
Project: Zookeeper
Issue Type: Improvement
Components: documentation
Affects Versions: 3.3.0
Reporter: Adam Rosien
Priority: Minor
http://hadoop.apache.org/zookeeper/docs/r3.3.0/recipes.html#sc_recoverableSharedLocks
uses the heading "recoverable" locks, but the text refers to "revocable".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-641) Improve details about group membership recipe
Improve details about group membership recipe - Key: ZOOKEEPER-641 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-641 Project: Zookeeper Issue Type: Improvement Components: documentation Affects Versions: 3.2.1 Reporter: Adam Rosien Regarding http://eng.kaching.com/2010/01/actually-implementing-group-management.html Patrick Hunt asked for a more complete group membership recipe from the one listed at http://hadoop.apache.org/zookeeper/docs/r3.0.0/recipes.html#sc_outOfTheBox. The relevant text from the blog post: One type of group management system using ZooKeeper: * A group contains some logical service. The *meaning* of belonging to a group is typically "the instance is available for use by clients over the network". * Services can join and leave the group. The special case of a service crashing or a network outage needs to be handled as leaving the group. * Joined services share metadata about how to communicate with it, i.e., its IP address, base URL, etc. * Clients can ask what instances are in the group, i.e., available. * Clients are notified when group membership changes so they can mutate their local state. These map onto ZooKeeper as: * A group is a (permanent) node in the ZooKeeper hierarchy. Clients and services must be told the path to this node. * A services joins the group by creating an ephemeral node whose parent is the group node. By using an ephemeral node, if the service dies then the service is automatically removed from the group. * The ephemeral node's data contains the service metadata in some format like JSON, XML, Avro, Protobufs, Thrift, etc. ZooKeeper has no equivalent of HTTP's "Content-Type" header to identify the metadata representation, so services and clients must agree upon the format in some manner. * Clients can query for the children of the group node to identify the members of the group. * Clients can place a watch on the group node to be notified if nodes have joined or left the group. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: blog post about zk and group membership
Added https://issues.apache.org/jira/browse/ZOOKEEPER-641 for improved group membership recipe. .. Adam On Mon, Jan 11, 2010 at 9:49 AM, Patrick Hunt wrote: > Adam, this is awesome! I've retweeted it on my twitter account: > http://twitter.com/phunt/status/7637054275 > > A few comments: > > 1) you are correct about the "off the shelf" part, but really alot of that > is our marketing blurb, we paint with a broad stroke. I think it would make > sense to include this sort of detail in the "recipes" page though, would you > be interested to create a JIRA and submit some content? (you could re > purpose alot of what you have in your post) > > 2) re zkclient - I have not been following that project closely of late, > however Mahadev and I did review their code a few months ago. At that time > we had some concerns about the semantic impact of some of the design choices > they made (which were fine, just not well documented and therefor easy for > users to miss). In particular (again, at the time, may be addressed by now) > watch notifications could be lost during disconnected/expired sessions, and > the user would not be made aware of this. > > 3) excellent point/suggestion on the "rogue" service > > Patrick > > Adam Rosien wrote: >> >> I've written a blog post about using ZooKeeper for group membership at >> KaChing: >> >> http://bit.ly/6yraP6 >> http://eng.kaching.com/2010/01/actually-implementing-group-management.html >> >> Any comments or feedback is welcome. >> >> .. Adam >
Fwd: blog post about zk and group membership
-- Forwarded message -- From: Adam Rosien Date: Mon, Jan 11, 2010 at 10:27 AM Subject: Re: blog post about zk and group membership To: zookeeper-u...@hadoop.apache.org Yes, your Disqus comment went through. We filed a ticket with them about some javascript issues that messed up the UI. Data wrappers would certainly work but I feel like it's still a black-hole to fall into that something like node-metadata (to do something like HTTP's Content-Type") would help solve. That too can be a black hole :(. I need more practical experience to find the right balance. .. Adam On Mon, Jan 11, 2010 at 10:20 AM, Patrick Hunt wrote: > Adam Rosien wrote: >> >> Re "off the shelf", my title of "Actually Implementing" was a *bit* >> snarky to try and gain some readers, but it wasn't intended to be >> sharp criticism; more documented recipes is a good thing for everyone. >> I'd be happy to submit a JIRA with a group membership recipe. >> > > Ya, I got that, no worries. It was a valid point and something I've been > asked before. Figured I could get some (addl) free work out of the deal. ;-) > > Disqus seems to be having troubles, I've tried like 3 times to comment on > your blog post, did it finally make it through? > >> Re zkclient, I agree that documentation is a big thing, which I hope >> to help out with as I use it more. I'm talking now with one of the >> developers about zkclient's implementation of data serialization since >> that is another big area; zk pushes all serialization semantics into >> the clients, which is ok but requires its own practical strategies. > > Yea, that's an interesting point. I suspect that a wrapper around the data > (zk contrib? subclass/proxy of ZooKeeper?) that put content-type type > information into the "header" of the data portion might be useful. The > wrapper could encode/decode based on the data type in ZK, vs the data type > the user would like. The wrapper could support whichever encode/decode the > user would like (say xml/json to start) etc... Donno how useful in general > but in a larger system with a bunch of heterogeneous clients accessing it > might be interesting. > > Patrick > >> >> On Mon, Jan 11, 2010 at 9:49 AM, Patrick Hunt wrote: >>> >>> Adam, this is awesome! I've retweeted it on my twitter account: >>> http://twitter.com/phunt/status/7637054275 >>> >>> A few comments: >>> >>> 1) you are correct about the "off the shelf" part, but really alot of >>> that >>> is our marketing blurb, we paint with a broad stroke. I think it would >>> make >>> sense to include this sort of detail in the "recipes" page though, would >>> you >>> be interested to create a JIRA and submit some content? (you could re >>> purpose alot of what you have in your post) >>> >>> 2) re zkclient - I have not been following that project closely of late, >>> however Mahadev and I did review their code a few months ago. At that >>> time >>> we had some concerns about the semantic impact of some of the design >>> choices >>> they made (which were fine, just not well documented and therefor easy >>> for >>> users to miss). In particular (again, at the time, may be addressed by >>> now) >>> watch notifications could be lost during disconnected/expired sessions, >>> and >>> the user would not be made aware of this. >>> >>> 3) excellent point/suggestion on the "rogue" service >>> >>> Patrick >>> >>> Adam Rosien wrote: >>>> >>>> I've written a blog post about using ZooKeeper for group membership at >>>> KaChing: >>>> >>>> http://bit.ly/6yraP6 >>>> >>>> http://eng.kaching.com/2010/01/actually-implementing-group-management.html >>>> >>>> Any comments or feedback is welcome. >>>> >>>> .. Adam >
Re: blog post about zk and group membership
Thanks Patrick. Re "off the shelf", my title of "Actually Implementing" was a *bit* snarky to try and gain some readers, but it wasn't intended to be sharp criticism; more documented recipes is a good thing for everyone. I'd be happy to submit a JIRA with a group membership recipe. Re zkclient, I agree that documentation is a big thing, which I hope to help out with as I use it more. I'm talking now with one of the developers about zkclient's implementation of data serialization since that is another big area; zk pushes all serialization semantics into the clients, which is ok but requires its own practical strategies. .. Adam On Mon, Jan 11, 2010 at 9:49 AM, Patrick Hunt wrote: > Adam, this is awesome! I've retweeted it on my twitter account: > http://twitter.com/phunt/status/7637054275 > > A few comments: > > 1) you are correct about the "off the shelf" part, but really alot of that > is our marketing blurb, we paint with a broad stroke. I think it would make > sense to include this sort of detail in the "recipes" page though, would you > be interested to create a JIRA and submit some content? (you could re > purpose alot of what you have in your post) > > 2) re zkclient - I have not been following that project closely of late, > however Mahadev and I did review their code a few months ago. At that time > we had some concerns about the semantic impact of some of the design choices > they made (which were fine, just not well documented and therefor easy for > users to miss). In particular (again, at the time, may be addressed by now) > watch notifications could be lost during disconnected/expired sessions, and > the user would not be made aware of this. > > 3) excellent point/suggestion on the "rogue" service > > Patrick > > Adam Rosien wrote: >> >> I've written a blog post about using ZooKeeper for group membership at >> KaChing: >> >> http://bit.ly/6yraP6 >> http://eng.kaching.com/2010/01/actually-implementing-group-management.html >> >> Any comments or feedback is welcome. >> >> .. Adam >
Re: feedback zkclient
I'm on the fence about checked vs. unchecked, but I think Javadoc that declares what exceptions are thrown would help users figure out if they might want to catch something. Additionally some of the ZkClient method semantics are confusing. For example, create() returns the path of the created node, but throws a (runtime) ZkNodeExistsException when the node already exists. Creating a node that already exists is a relatively non-exceptional circumstance IMHO. Wouldn't it be simpler to return the path to the client, essentially pretending that the creation succeeded? Also, is it correct that if you want to "use" a node that may not exist, you'd have to call create() and then wait for the node to actually be created via (in zkclient) waitUntilExists()? (Obviously the node can be removed before it is used, and that situation needs to be taken care of anyway.) Perhaps create() and waitUntilExists() should be combined, or offer a combined version. As a naive zk user a wrapper like that would remove a lot of error potential. Apologies if this is not the best place for zkclient API discussions. .. Adam On Thu, Oct 1, 2009 at 9:19 AM, Ted Dunning wrote: > I think that another way to say this is that zkClient is going a bit for the > Spring philosophy that if the caller can't (or won't) be handling the > situation, then they shouldn't be forced to declare it. The Spring > jdbcTemplate is a grand example of the benefits of this. > > First implementations of this policy generally are a bit too broad, though, > so this should be examined carefully. > > On Thu, Oct 1, 2009 at 8:05 AM, Peter Voss wrote: > >> 5) there's a lot of wrapping of exceptions, looks like this is done in >>> order to make them unchecked. Is this wise? How much "simpler" does it >>> really make things? Esp things like interrupted exception? As you mentioned, >>> one of your intents is to simplify things, but perhaps too simple? Some >>> short, clear examples of usage would be helpful here to compare/contrast, I >>> took a very quick look at some of the tests but that didn't help much. Is >>> there a test(s) in particular that I should look at to see how zkclient is >>> used, and the benefits incurred? >>> >> >> Checked exceptions are very painful when you are assembling together a >> larger number of libraries (which is true for most enterprise applications). >> Either you wind up having a general "throws Exception" (which I don't really >> like, because it's too general) at most of your interfaces, or you have to >> wrap checked exceptions into runtime exceptions. >> >> We didn't want a library to introduce yet another checked exception that >> you MUST catch or rethrow. I know that there are different opinions about >> that, but that's the idea behind this. >> >> Similar situation for the InterruptedException. ZkClient also converts this >> to a runtime exception and makes sure that the interrupted flag doesn't get >> cleared. There are just too many existing libraries that have a "catch >> (Exception e)" somewhere that totally ignores that this would reset the >> interrupt flag, if e is an InterruptedException. Therefore we better avoid >> having all of the methods throwing that exception. > > > > > -- > Ted Dunning, CTO > DeepDyve >
