On 16 May 2014 08:12, Andreas Neumann <[email protected]> wrote: > Hi Steve, > > not sure how long ago you posted this, I still get emails that were sent > during the outage, so apologies if this response comes with a little delay. >
yes, it did get lost, but thanks fore replying > Let me start with answers to your list of questions (In some cases, this > reflects the way that Twill is used at Continuuity, there may be others who > use it differently): > > 1. What are the common ways you use this? Is it within a distributed > > app, or between apps? > > > We use it within an app (one runnable discovers the location of another > runnable in the same app) and between different Twill apps. We also use it > to discover services provided by a Twill app from outside of the cluster. > > OK. Helix has the notion of internal vs external here (a bit like private/public, but json introspection doesn't like those words) > 2. Do you ever use this outside the cluster itself? > > > Yes, we also use it outside of the cluster. Twill's registry is not limited > to use within Twill. > > 3. Do you listen for changes, or just grab it at startup? > > > Twill's discovery service client watches for changes in ZooKeeper and > provides live updates. > > 4. If you could publish more information, what would you publish? > > > I think that really depends on the application. From the perspective of the > Twill framework, it is really about the location, but I see that generic > payloads would be of good use. > OK. We're playing with publishing key-value configs, but that's the kind of thing that isn't ready for standardisation. I think for the YARN core we will initially define -the notion of an app having multiple components, each potentially publishing their own data -the notion of an "instance" entry that can represent the primary accessor to the application (this could be registered by any component, its just the default thing to bind to -the notion of internal/external -the notion of a service/node having 1* endpoint (web, IPC, REST, ...), with information for client apps and UIs. I think we may want to evolve some common names here, for things like JMX, log4j management, ... -a way for components to publish 1* JSON document of their own choice. Anything can do their own, or build defacto standards with others. > > 5. If we had a well-known URL a service registry, one that would support > > GET lookups and 302 redirects, what would you serve up? A Web UI for > > people, a REST API for app management, core app functionality... > > > Not sure if REST is the right protocol for a service discovery, as you > cannot watch for changes, you would have to keep polling, which seems > inappropriate to me. WebSockets might be better for this. 302 redirects > also assume that the registered services are all HTTP, which may not be the > case. > > 6. What security model do you rely on? do you assume that whoever > > publishes a service is being honest, do you rely on ZK permissions, or > > something else? > > > Security is not quite evolved in Twill yet. We currently rely on ZK > permissions. > > 7. Finally: did you deliberately choose not to use Apache Curator's > > discovery mechanism, and if so -why not? > > > This was deliberate... mostly to avoid the transitive dependencies that > Curator would bring in (and version conflicts with other libraries we use). > yeah, we get that. I patched Hadoop to at least use the same jackson version, > We also found Curator's logging very chatty and flooding our logs whereas > it was not very resilient to ZK timeout or connection loss. However, that > was with a Curator version from one or two years ago, and I cannot say > whether we would make same decision today as Curator has become more > mature. > They have some good plugin retry policy handlers, but I haven't tested failure. Logs are an emergent problem in long-lived YARN apps > > Any insight to these or other registry-issues would be really helpful. I > > don't want to build the "single best ever registry service for humanity" > as > > it would take too long, but I'd like to start with something that we can > > evolve, which suits YARN app needs and hooks into the user experience and > > management tools. For now we're using Curator, with some workarounds for > > hadoop 2.4 jackson versions, a custom payload that publishes the real > info, > > and our AM serving up content -configurations and other artifacts- > indexed > > off the registry entry. > > > In Twill, it the runnable itself that publishes to the registry, and not > the AM. Doing it directly from the runnable ensures that the (ephemeral) > node in ZK disappears right away the runnable dies that actually provides > the service. Doing it in the AM bears the risk that the AM's view of what's > live and what's not may be out of date, and the AM also can't make use of > ephemeral nodes because the AM would remain alive even when a runnable > dies. > makes sense. Anything published by a component that is transient to that component instance/container should be owned by that component -else you are missing all the watchability features. > > Generally speaking, I think that having a service registry in YARN would be > useful. Yet I am not sure whether Twill would use that: Twill's APIs are > generic and do not have dependencies on YARN. Right now, the only > implementation is on top of YARN, but we might as well run in multiple > threads of a single JVM or in Mesos. In these cases we would not want to > depend on a YARN registry. > > -Andreas. > > I think what makes sense there is making sure that the YARN one is >= your needs, and just use it a back end. If we annotate service endpoints with binding info (protocol, URL, auth types, certs &c) it'd be good if we could say "must" for some of these, so it would be nice if at least at registration time Twill could state these things -you could just discard that info when bonding to back ends that don't cache it. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
