Hi Martin, Great work with the proposal!
On Wed, Apr 30, 2014 at 9:56 AM, Martin Eppel (meppel) <[email protected]>wrote: > Hi Lakmal > > > > Below is a further enhanced proposal to add grouping to apache stratos: > > > > In a nutshell: > > > > By grouping cartridges together we can define characteristics which apply > to all cartridges alike most importantly it will allow us also to define > dependencies between cartridges. Groups should also be flexible enough to > not only contain single cartridges but also groups, allowing a hierarchical > structure. > > > > Dependencies are defined at the group level, describing the dependency > relationship of immediate children. > > Other group specific properties could be defined which will apply to all > immediate group members. > > > > The subscription model will be extended to subscribe to a group in > addition to a cartridge (either or). To generalize we added the concept of > a Subscribable which can be either a group or cartridge. > > > > Since a Subscribale (or more specifically a group) doesn’t have scaling > characteristics, it seemed appropriate to add the concept of Scalable, > describing the scalable nature of a cartridge. > > > > Below is a more detailed description and a corresponding diagram: > > > > “Class diagram” of the logical model showing the existing description > files in the Diadem model, and the proposed changes. > > > > Description: see attachment Slide1.png > > > > Some notes on the new dependency system: > > > > * A Group can be subscribed or a Scalable can be subscribed, whereas > > today, a Cartridge is subscribed. > Shouldn't this rather be a Subscribable can be subscribed, as a Subscribable can be either a group or a cartridge? > o The new Scalable entity is needed because currently > > Subscriptions own autoscale policies, but that doesn’t work when > > a Group contains more than one Cartridge, because we want one > > autoscale policy per Cartridge. However we can’t couple the > > autoscale policy to the Cartridge because it may be re-used in > > other Subscriptions where a different autoscale policy is desired > > * children is the set of Subscribables in the Group. > > o This supports hierarchical Groups. > > * collocate says “these Children must be physically next to each other” > > * startupOrder is a Set of asymmetric pairs (A, B) where A points to B > > but B doesn’t point to A. > > o These pairs define a DAG, which we use to perform a topological > > sort of the children and get a /partial/ ordering. > > o Any member of “children” not in a pair is a singleton started in > > parallel to everything else. > > * The killBehaviour says that when node X dies, we must kill: > > o Everything else (for applications where the loss of any member > > of “children” cannot be recovered with less impact > > o Nothing else (for applications where any element can be restarted) > > o Or just the things “upstream” in the startup order (for > > conventional tiered applications) > > > > We think this describes all the use-cases we’ll ever see with the > exception of things like “n instances of A depends on B”; that can be > considered in the future as an enhancement. > > > > Startup use cases considered > > > > * All start in parallel: don’t specify any pairs => everything is > > equivalent => all start in parallel > > * All start one-after-another (total order): specify a set of pairs > > such that there is a single contiguous list covering all the nodes, > > e.g. A -> B -> C -> D -> E > > * Some start-up dependencies: specify as necessary e.g. [[MiddleTier > > -> Database], [Logging]] > > > > Kill use cases considered > > > > * One dies => all die: set killBehaviour to KillAll > > * One dies => nothing happens: set killBehaviour to killNone > > * One dies => its dependancies die (aka “restart from here”): specify > > a startupOrder DAG and killBehaviour=StartupOrder. E.g. [[MiddleTier > > -> Database], [Logging]], Database dies => MiddleTier is restarted, > > Logging untouched. > Also, we will need to come up with a plan for rollback as well, in the case a group being an atomic unit. There might be cases of errors in one or more subscribables in spinning up, where the entire group becomes un-usable and we need to terminate the whole group (and maybe try again). WDYT? > > > Example instantiation > > > > Description: see attachment Slide2.png > > > > Thanks > > > > Martin > > > > > > *From:* Lakmal Warusawithana [mailto:[email protected]] > *Sent:* Sunday, March 30, 2014 9:09 PM > > *To:* [email protected] > *Subject:* Re: [Discuss] Grouping of services (cartridges) > > > > > > > > On Mon, Mar 31, 2014 at 7:47 AM, Martin Eppel (meppel) <[email protected]> > wrote: > > I totally agree, I am also not in favor of complicating existing > components (e.g. autoscaler). > > > > However, I am not sure what the alternative might be to solve the > requirements (e.g. item 1) mentioned below. > > The suggestion I made to enhance the autoscaler / rules are based on my > understanding that the autoscaler already handles similar actions (e.g. VM > startup, scaling and termination). > > It seems to me a logical extension to add additional knowledge to handle > the boot sequencing, dependent scaling and dependent termination to the > autoscaler defined by optional properties in the autoscaler policy. > > > > From my current point of view, CAMP seems a fairly complex specification > which might require quite some changes to adopt. > > > > Sorry I could not go through CAMP in detail. Should spend sometime. > > > > > > I do agree that alternatives might exist and should be discussed !? > Alternatives ? > > > > +1 for alternatives, we should not take CAMP as it is, we should see how > we can match with existing Stratos workflow IMO. > > > > > > Thanks > > > > Martin > > > > *From:* damitha kumarage [mailto:[email protected]] > *Sent:* Sunday, March 30, 2014 7:59 AM > *To:* [email protected] > > > *Subject:* Re: [Discuss] Grouping of services (cartridges) > > > > Please see my inline comment, > > > > On Sat, Mar 29, 2014 at 11:53 AM, Isuru Haththotuwa <[email protected]> > wrote: > > Hi Martin, > > On Fri, Mar 28, 2014 at 11:19 PM, Martin Eppel (meppel) <[email protected]> > wrote: > > Hi, > > > > I think this property will be a quite useful piece to solve the grouping > issue: > > > > I also would like to suggest to add the serviceGroup to the topology map > (in case it is not yet available in the topology map). This will help to > tie together cartridges (or services) in the autoscaler and , for example > enable synchronized auto scaling behavior of services within a service > group, like synchronized scaling, sequenced boot up, etc …. > > > > In addition the autoscaler should be enhanced to add additional (but > optional properties) in the auto scaling policy related to a service group > to govern the respective auto scaling behavior. > > > > For example, related properties should identify a service group and other > related properties to define dependencies between the various cartridges in > the service group like boot sequence, scale up / down ratios, termination > dependencies, etc … . The property set (or json structure ) should be > fairly flexible as we are just about to explore this new feature and should > be easily expandable. > > > > I would think that these additions will also prove useful to integrate in > the long term with CAMP (or other spec) but will help to solve more > immediate requirements > > > > Yes, these are very valid points in coming up with a proper grouping > architecture for services. Thank you for bringing them up. > > As I understand, what Sajith has done here is enabling static grouping of > services, by using a property for that in the cartridge deployment time. > What we are trying to achieve in long term is dynamic grouping of services, > so that we can group any available service at runtime seamlessly, according > to CAMP specification (or some other suitable way). > > > I see three things here > 1) Grouping and resolve inter-dependancies between cartridges. > > 2) Composite Artifact deployment (May be if required, according to the > above grouping and inter-dependancy( of cartridges. There can also be > intra-dependancies inside a cartridge in case of artifact deployment) > > 3) Improved nonitoring and health check of above intricacies. > > Instead of adopt CAMP to solve above, I think it is better to discuss and > find ways that fits most naturally to the existing Stratos > architecture(Unless CAMP is widely adopted and we are compelled to > adhere). Is there a way we can solve this without doing major changes to > existing components? For example without complicating autoscaler > policies/logic as suggested by Martin above? > > Damitha > > > > > > > > -- > __________________________________________________________________ > Damitha Kumarage > http://people.apache.org/ > __________________________________________________________________ > > > > > > -- > > Lakmal Warusawithana > > Software Architect; WSO2 Inc. > > Mobile : +94714289692 > > Blog : http://lakmalsview.blogspot.com/ > -- Thanks and Regards, Isuru H. +94 716 358 048* <http://wso2.com/>*
