Improvement jira filed - https://issues.apache.org/jira/browse/SLIDER-1066
Thanks all for your time and insights. Manoj On Mon, Jan 18, 2016 at 6:56 PM, Gour Saha <[email protected]> wrote: > Yes please file a jira. Let’s discuss it there further. > > On 1/18/16, 3:57 PM, "Manoj Samel" <[email protected]> wrote: > > >Point take. > > > >For now, we have chosen to go with single app. And exactly to your point - > >we don't want the AM to restart when we know that there is no change to be > >picked up so restarting in this case has more downside (AM issue etc.) - > >hence the feature ask. > > > >Any pros/conn on making this possible ? Should I file a feature jira ? > > > >Thanks > > > >On Mon, Jan 18, 2016 at 3:05 PM, Gour Saha <[email protected]> wrote: > > > >> Inline. > >> > >> On 1/14/16, 4:52 PM, "Manoj Samel" <[email protected]> wrote: > >> > >> >Hi Gour, > >> > > >> >The use case is described in > >> >https://issues.apache.org/jira/browse/SLIDER-1046 > >> > > >> >Regarding your latest proposals - > >> > > >> >1. Can a "slider update" be used - As I recall, slider update is only > >>when > >> >application is stopped. We don't stop the application when adding new > >>user > >> > >> [Gour] good point. I forgot that. > >> > > >> >2. Can a new slider app be used for each user - Hummm .. this is > >> >interesting one ;^). After some thoughts, I see couple of issues > >> > > >> >Issue 1 - With large number of users, we have equally large number of > >>AMs. > >> >Even if AM footprint is small, it could grow in future and # of AM > >>will be > >> >a issue in any case. > >> > > >> >Issue 2 - Large number of applications will become issue. E.g. when > >>doing > >> >application upgrade, instead of doing one AM upgrade followed by > >>rolling > >> >upgrade of users, it will be larger number of AM upgrades. I think > >>this is > >> >less manageable. Also other operations like shutting down the app (in > >> >event > >> >of cluster downtime etc.) will be getting cumbersome > >> > > >> >Single app with different components per user seems still the best > >>choice > >> >... > >> > >> [Gour] Makes sense, but keep in mind that a single app is also subject > >>to > >> a single point of failure for all users (say in the unforeseen case of > >>an > >> issue during Slider AM or application upgrade). Note, multiple > >> applications upgrade will be performed by a script or orchestrator, just > >> like you would have to do to the equal number of components for a single > >> app as well. > >> > > >> >Thanks, > >> > > >> >Manoj > >> > > >> >On Wed, Jan 13, 2016 at 2:08 PM, Gour Saha <[email protected]> > >>wrote: > >> > > >> >> Can you look into ³slider update² (not upgrade) and then subsequently > >> >> followed by appropriate ³slider flex² to bring up components for new > >> >>users? > >> >> > >> >> It might be helpful if you can provide a brief explanation of your > >> >> usecase. Just trying to see if mapping a new user to a new slider app > >> >> (with a single component definition) makes more sense, compared to > >>your > >> >> current approach. The overhead of a Slider AM is very low. > >> >> > >> >> -Gour > >> >> > >> >> On 1/13/16, 10:35 AM, "Manoj Samel" <[email protected]> > wrote: > >> >> > >> >> >Flex isn't option for couple of reasons in above use case > >> >> > > >> >> >1. For each new user, different parameters need to be passed when > >> >> >starting. > >> >> >New component allows control over component name and parameters > >> >> >2. More importantly, when user is 'decomissioned', the component for > >> >>that > >> >> >user should be stopped for good. This is achieved at present by > >>naming > >> >> >each > >> >> >component after respective user and stopping that when user is gone. > >> >> > > >> >> >Thanks, > >> >> > > >> >> >Manoj > >> >> > > >> >> >On Wed, Jan 13, 2016 at 10:11 AM, Steve Loughran > >> >><[email protected]> > >> >> >wrote: > >> >> > > >> >> >> you should just be flexing here > >> >> >> > >> >> >> > On 13 Jan 2016, at 09:19, Manoj Samel <[email protected] > > > >> >> >>wrote: > >> >> >> > > >> >> >> > Hi, > >> >> >> > > >> >> >> > In recent thread about existing components taking long time to > >> >> >>heartbeat > >> >> >> > after upgrade; it was mentioned that "slider AM restarts itself > >>to > >> >> >>pick > >> >> >> up > >> >> >> > Yarn parameters" > >> >> >> > > >> >> >> > If the upgrade simply consist of starting a new component, is > >>the > >> >>AM > >> >> >> > restart still needed ? I.e. if the hadoop cluster configuration > >>has > >> >> >>not > >> >> >> > changed and the Yarn parameters in the slider applications have > >>not > >> >> >> changed > >> >> >> > for any components and upgrade is being used just to start a new > >> >> >> component; > >> >> >> > is AM restart still needed ? > >> >> >> > > >> >> >> > The use case is for service running large number of components. > >> >>When a > >> >> >> new > >> >> >> > user is enabled for service, a new component is started for that > >> >>user. > >> >> >> > However, no changes are done for hadoop cluster or in the slider > >> >> >> > application configuration for any existing components. When > >>large > >> >> >>number > >> >> >> of > >> >> >> > users (i.e. components) are already running, AM restart means > >>these > >> >> >> > existing components have to be informed of new AM and vice > >>versa. > >> >> >> > > >> >> >> > In such cases, if AM restart is not needed, can it be avoided > >>e.g. > >> >>by > >> >> >> > having a new option in Upgrade command ? > >> >> >> > > >> >> >> > Thanks, > >> >> >> > > >> >> >> > Manoj > >> >> >> > >> >> >> > >> >> > >> >> > >> > >> > >
