Hi Steve, Please see my answers inline.
On Wed, May 7, 2014 at 1:07 AM, Steve Loughran <[email protected]> wrote: > Hi, > > I'm starting to build a service registry in the now-in-incubation Slider > project, with an ultimate goal for a derivative or rework of this to make > its way into YARN proper, so that there'll be one we can all use to publish > and share information about YARN-deployed and static apps in a Hadoop > cluster. > > To this end I'm trying to make sure I understand the various uses other > projects are making of their ZK-based registries. > > I see that Twill is using it to publish (hostname, port) pairs, and lets > you poll or watch for this, and want to know a bit more > > > 1. What are the common ways you use this? Is it within a distributed > app, or between apps? Could be both. E.g. one could starts an app inside a cluster, hosting some network endpoints and with those endpoints announced through discovery service. Any other services, whether they are run inside YARN or not could discovery those endpoints and be able to talk to them. > 2. Do you ever use this outside the cluster itself? Yes > 3. Do you listen for changes, or just grab it at startup? It's even base with client side cache updated by ZK watchers > 4. If you could publish more information, what would you publish? I would say just a byte array > 5. If we had a well-known URL a service registry, one that would support > GET lookups and 302 redirects, what would you serve up? A Web UI for > people, a REST API for app management, core app functionality... Do you mean a hosted web service for discovery? It would be useful for simple use cases. However, the clients would need to poll for changes? > 6. What security model do you rely on? do you assume that whoever > publishes a service is being honest, do you rely on ZK permissions, or > something else? Our current solution uses ZK, hence security is also relies on ZK (auth + ACL) > 7. Finally: did you deliberately choose not to use Apache Curator's > discovery mechanism, and if so -why not? Currently there are some dependencies issue with using Curator. Also Twill uses ZK for things other than discovery as well. > > > Any insight to these or other registry-issues would be really helpful. I > don't want to build the "single best ever registry service for humanity" as > it would take too long, but I'd like to start with something that we can > evolve, which suits YARN app needs and hooks into the user experience and > management tools. For now we're using Curator, with some workarounds for > hadoop 2.4 jackson versions, a custom payload that publishes the real info, > and our AM serving up content -configurations and other artifacts- indexed > off the registry entry. > > > -Steve > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
