[re-sending to dev list since I had accidentally have removed dev list] ---------- Forwarded message ---------- From: Isuru Haththotuwa <isu...@apache.org> Date: Thu, Sep 4, 2014 at 12:14 PM Subject: Re: [Discuss] High Level Flow for Composite Application Deployment To: "Martin Eppel (meppel)" <mep...@cisco.com>
Hi Martin, On Thu, Sep 4, 2014 at 1:33 AM, Martin Eppel (meppel) <mep...@cisco.com> wrote: > Hi Isuru, > > > > +1 > > > > I generally agree with flow to avoid multiple, expensive conversions . I > have a couple of questions, some are just to clarify the flow and data > model, some of them we need to answer as we go deeper into the design / > implementation: > > > > Will we maintain backwards compatibility for single cartridge > subscriptions ? > As I understand we can support this using an Application with a single Subscribable. With Composite App changes, we will need to modify some of the logic related to Cluster Monitors in autoscaler. Therefore, it might be difficult to support single Subscription as it is. > > > Any parsing of the composite application is done at the CC, Rest API will > just “forward” the composite application to CC ? > Correct. > > > ApplicationCreatedEvent has all the subscription related information which > provides the data to spin up the VMs. All the grouping related information > (like dependencies) will be retrieved by autoscaler from topology model ? > Correct. This is what I thought as well. > > > Before spinning up any VMs the dependencies conditions need to be > satisfied, I would think that the application monitor will calculate the > dependencies (since we are using a hierarchical model) which can then be > queried by the autoscaler, WDYT ? > I agree. Each level would be responsible for calculating and performing dependency checks for the immediate children. > > > For group scaling will the scaling information (e.g group scaling policy) > be part of subscription information ( Application Created event) or some > other means ? > Currently a user can specify scaling policies (group/individual) in the Composite App definition itself. > > > When a group scales, multiple instances of the group (definition / type) > can exist (group instance), will we have a group monitor for each group > instance or one for each group (type) only ? > Still there should only one Group Monitor as I understand, and it should take care of scaling up/down of the group in the relevant order. > > > When we scale groups, subscription information (for cartridge > subscriptions within the group) won’t change for the new group instance – > will it be necessary to create a new Cluster , ClusterMonitor, etc or can > we think of a way to just add on to the existing model instances (e.g. > adding more VM instances in a cluster to scale up by group) ? > As I understand, we don't need to create a new cluster, but can add additional members to Group(s). > > > > > Thanks > > > > > > Martin > > > > *From:* isu...@wso2.com [mailto:isu...@wso2.com] *On Behalf Of *Isuru > Haththotuwa > *Sent:* Tuesday, September 02, 2014 11:44 PM > *To:* dev > *Subject:* [Discuss] High Level Flow for Composite Application Deployment > > > > This is to discuss $subject. Sorry for the long mail. > > In the current single subscription based mechanism, SM creates the > Subscription and sends the relevant information (cluster id, host name, > payload, etc.) to CC. This is not feasible in the Composite App scenario. > There are several reasons for this: > > 1. Composite App structure is a complex nested object. If we parse > this at SM and we send over to CC via an API call, we would need to > recursively traverse the tree structure and convert to the stub object. > Again in CC, it will need to convert to the Topology model. > 2. If we send the same information from SM to CC via a new event, > again from the SM we need to convert it to the event model and from CC we > need to convert it to the Topology model (because SM does not publish to > the Topology). > > Therefore, in both scenarios mentioned above, there are two expensive > conversions happening. > > To overcome this, I came up with the following changes. > > 1. Composite App definition will be directly parsed at CC, and the > topology will be updated from there. > 2. CC will send an event (Application Created) to notify the > listeners. This will have the relevant Subscription related information > (cluster id, hostname, etc. - what is currently being generated at SM). > 3. The payload information will be published to the meta data service. > 4. SM will build a information model using the information sent in > Application Created event and the information queried from the meta data > service to expose to outside over the rest API. > > > > In Composite App implementation, mainly would need two flows; deploying a > Group definition(s) and deploying Composite App definition. > > 1. > > Main steps: > > - send the Group Definition from rest API to CC > - validate definition and persist in CC side > > Validation here includes verifying whether a all the defined cartridges > and sub groups are already deployed. > > > 2. > > > > > Main steps: > > - send the Group Definition from rest API to CC > - CC traverses and parses the app definition > - CC generates a unique key for the app > - Application Created event sent from CC. This will have all the > Subscription related information > - Update meta data service with relevant payload data. These data > should be accessible by providing the relevant rest URL. Authentication is > performed using the generated app specific key. > > By using the meta data service, minimum amount of information should be > sent by payload (the meta data service endpoint, application key, etc.). > Other information can be queries using the meta data service. > > After the Application Created event is sent by CC, Autoscaler will respond > to this event as follows: > > > > > Main steps: > > - Autoscaler acts upon Application Created event; > i. Traverse the tree, create Application Monitors and Group > Monitors (hierarchical). It will find the order in which clusters should be > created and members spawned (according to the dependency ordering defined > in the Group Definition). > ii. create cluster monitor for the particular cluster(s) > - Autoscaler asks CC to spawn the cluster with minimum no of instances. > - AS will wait for Cluster Created event to mark that cluster as > created. When a particular cluster has minimum number of instances > activate, notify relevant parent Monitor and mark Cluster Status as ACTIVE > - When a Group's all dependent clusters and sub groups are active, > notify relevant parent Group Monitor and mark Group Status as ACTIVE > - When an application's all groups and clusters are active, The App > Monitor will be notified, and the App Status would be marked as ACTIVE > > The App Monitor, Group Monitor and the Cluster Monitors forms a hierarchy. > Therefore, it should be possible to register an Observer for each Monitor > so that interested parties can be notified. In this case, parent Monitors > should be notified by the child monitors about the changes happening (state > changes, etc.). This will be useful in performing Group based scaling and > error handling (When a member of a Cluster goes down, how to handle the > termination of dependent members and re-spawn, etc.). > > This is a considerable amount of change from what we have in Stratos > currently. Please share your feedback and questions. > > -- > > Thanks and Regards, > > Isuru H. > > +94 716 358 048 > > -- > <%2B94%20716%20358%20048> > Thanks and Regards, > > Isuru H. > <%2B94%20716%20358%20048> > +94 716 358 048 > > -- > <%2B94%20716%20358%20048> > Thanks and Regards, > > Isuru H. > <%2B94%20716%20358%20048> > +94 716 358 048* <http://wso2.com/>* > > > * <http://wso2.com/>* > > >
