Hi Steve, not sure how long ago you posted this, I still get emails that were sent during the outage, so apologies if this response comes with a little delay. Let me start with answers to your list of questions (In some cases, this reflects the way that Twill is used at Continuuity, there may be others who use it differently):
1. What are the common ways you use this? Is it within a distributed > app, or between apps? We use it within an app (one runnable discovers the location of another runnable in the same app) and between different Twill apps. We also use it to discover services provided by a Twill app from outside of the cluster. 2. Do you ever use this outside the cluster itself? Yes, we also use it outside of the cluster. Twill's registry is not limited to use within Twill. 3. Do you listen for changes, or just grab it at startup? Twill's discovery service client watches for changes in ZooKeeper and provides live updates. 4. If you could publish more information, what would you publish? I think that really depends on the application. From the perspective of the Twill framework, it is really about the location, but I see that generic payloads would be of good use. 5. If we had a well-known URL a service registry, one that would support > GET lookups and 302 redirects, what would you serve up? A Web UI for > people, a REST API for app management, core app functionality... Not sure if REST is the right protocol for a service discovery, as you cannot watch for changes, you would have to keep polling, which seems inappropriate to me. WebSockets might be better for this. 302 redirects also assume that the registered services are all HTTP, which may not be the case. 6. What security model do you rely on? do you assume that whoever > publishes a service is being honest, do you rely on ZK permissions, or > something else? Security is not quite evolved in Twill yet. We currently rely on ZK permissions. 7. Finally: did you deliberately choose not to use Apache Curator's > discovery mechanism, and if so -why not? This was deliberate... mostly to avoid the transitive dependencies that Curator would bring in (and version conflicts with other libraries we use). We also found Curator's logging very chatty and flooding our logs whereas it was not very resilient to ZK timeout or connection loss. However, that was with a Curator version from one or two years ago, and I cannot say whether we would make same decision today as Curator has become more mature. Any insight to these or other registry-issues would be really helpful. I > don't want to build the "single best ever registry service for humanity" as > it would take too long, but I'd like to start with something that we can > evolve, which suits YARN app needs and hooks into the user experience and > management tools. For now we're using Curator, with some workarounds for > hadoop 2.4 jackson versions, a custom payload that publishes the real info, > and our AM serving up content -configurations and other artifacts- indexed > off the registry entry. In Twill, it the runnable itself that publishes to the registry, and not the AM. Doing it directly from the runnable ensures that the (ephemeral) node in ZK disappears right away the runnable dies that actually provides the service. Doing it in the AM bears the risk that the AM's view of what's live and what's not may be out of date, and the AM also can't make use of ephemeral nodes because the AM would remain alive even when a runnable dies. Generally speaking, I think that having a service registry in YARN would be useful. Yet I am not sure whether Twill would use that: Twill's APIs are generic and do not have dependencies on YARN. Right now, the only implementation is on top of YARN, but we might as well run in multiple threads of a single JVM or in Mesos. In these cases we would not want to depend on a YARN registry. -Andreas. On Wed, May 7, 2014 at 10:07 AM, Steve Loughran <[email protected]>wrote: > Hi, > > I'm starting to build a service registry in the now-in-incubation Slider > project, with an ultimate goal for a derivative or rework of this to make > its way into YARN proper, so that there'll be one we can all use to publish > and share information about YARN-deployed and static apps in a Hadoop > cluster. > > To this end I'm trying to make sure I understand the various uses other > projects are making of their ZK-based registries. > > I see that Twill is using it to publish (hostname, port) pairs, and lets > you poll or watch for this, and want to know a bit more > > > 1. What are the common ways you use this? Is it within a distributed > app, or between apps? > 2. Do you ever use this outside the cluster itself? > 3. Do you listen for changes, or just grab it at startup? > 4. If you could publish more information, what would you publish? > 5. If we had a well-known URL a service registry, one that would support > GET lookups and 302 redirects, what would you serve up? A Web UI for > people, a REST API for app management, core app functionality... > 6. What security model do you rely on? do you assume that whoever > publishes a service is being honest, do you rely on ZK permissions, or > something else? > 7. Finally: did you deliberately choose not to use Apache Curator's > discovery mechanism, and if so -why not? > > > Any insight to these or other registry-issues would be really helpful. I > don't want to build the "single best ever registry service for humanity" as > it would take too long, but I'd like to start with something that we can > evolve, which suits YARN app needs and hooks into the user experience and > management tools. For now we're using Curator, with some workarounds for > hadoop 2.4 jackson versions, a custom payload that publishes the real info, > and our AM serving up content -configurations and other artifacts- indexed > off the registry entry. > > > -Steve > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
