> On 24 Dec 2015, at 13:40, Jon Maron <jma...@hortonworks.com> wrote: > > >> On Dec 23, 2015, at 6:27 PM, Manoj Samel <manojsamelt...@gmail.com> wrote: >> >> Jon, as to your last comment - >> >> As I understand, the idea of pushing slider registry to more general yarn >> registry was to make it easier for clients of application to query, without >> going through slider (URIs) at all. Most client would care about info for >> live application. If client query on ZK registry can't tell whether the >> application is live, then its not clear what is the value of registry for >> clients of the application- there has to be some path in registry that is >> tied to liveliness of the application. > > As Steve indicated, there is a patch that will be merged post the next > release that does, in fact, purge the repository entries during a stop > (SLIDER-1017). That should address the concern about a mismatch of > availability of information between resources (ZK vs AM). >
Jon, you can put the patch into the develop branch now if you want; the 0.90 stuff has split off. > I think the registry being moved to Yarn was more an effort to make its > facilities available as a resource for all yarn applications rather than > slider specifically. But I would also agree that its utility as a “liveness” > indicator is enhanced by this fix. > The goal was/is for something generic across not just YARN, but hadoop apps in general, where even the NN and stuff are discoverable. Someone (that'll be me?) has to sit down and finish the YARN-913 stuff. the original implementation used ZK transient nodes to model things, so when a container went away, so did the entry —which turns out to be pretty brittle once you start testing by killing ZK services. They're all persistent now. Part of YARN-913 is a patch to do cleanup of nodes in the RM when containers and apps go away, but the YARN team don't actually like it in its present form. I've put it into HDP, but I think we need to accept that it's not going to stay. I do want the always-create-the-user-path feature in there though, isolated from the rest. -Steve
