Hi Steve, Regarding your note "I've put it into HDP, but I think we need to accept that it's not going to stay"
Would like to understand development process for slider w.r.t. code changes in other parts of hadoop. E.g. do you put it in HDP first and then merge into general hadoop branch ? Is there a timeline for when a slider change that got into HDP gets into general hadoop codeline ? We are unable to uptake HDP, hence asking Thanks, Manoj On Tue, Dec 29, 2015 at 2:59 AM, Steve Loughran <ste...@hortonworks.com> wrote: > > > On 24 Dec 2015, at 13:40, Jon Maron <jma...@hortonworks.com> wrote: > > > > > >> On Dec 23, 2015, at 6:27 PM, Manoj Samel <manojsamelt...@gmail.com> > wrote: > >> > >> Jon, as to your last comment - > >> > >> As I understand, the idea of pushing slider registry to more general > yarn > >> registry was to make it easier for clients of application to query, > without > >> going through slider (URIs) at all. Most client would care about info > for > >> live application. If client query on ZK registry can't tell whether the > >> application is live, then its not clear what is the value of registry > for > >> clients of the application- there has to be some path in registry that > is > >> tied to liveliness of the application. > > > > As Steve indicated, there is a patch that will be merged post the next > release that does, in fact, purge the repository entries during a stop > (SLIDER-1017). That should address the concern about a mismatch of > availability of information between resources (ZK vs AM). > > > > Jon, > > you can put the patch into the develop branch now if you want; the 0.90 > stuff has split off. > > > > > I think the registry being moved to Yarn was more an effort to make its > facilities available as a resource for all yarn applications rather than > slider specifically. But I would also agree that its utility as a > “liveness” indicator is enhanced by this fix. > > > > The goal was/is for something generic across not just YARN, but hadoop > apps in general, where even the NN and stuff are discoverable. Someone > (that'll be me?) has to sit down and finish the YARN-913 stuff. > > the original implementation used ZK transient nodes to model things, so > when a container went away, so did the entry —which turns out to be pretty > brittle once you start testing by killing ZK services. They're all > persistent now. Part of YARN-913 is a patch to do cleanup of nodes in the > RM when containers and apps go away, but the YARN team don't actually like > it in its present form. I've put it into HDP, but I think we need to accept > that it's not going to stay. I do want the always-create-the-user-path > feature in there though, isolated from the rest. > > -Steve > > >
