I'd say right now we're looking at "static service composition" which is only about "substituting service templates", not "substituting service". If a service is already running, it will not be used.
I think what Tal meant was that each service template - whether the top level one or one of the substituting templates - needs to resolve its inner reqs&caps internally first, and then resolve substitution reqs&caps across service templates. On Wed, Aug 16, 2017 at 12:00 PM, D Jayachandran < d.jayachand...@ericsson.com> wrote: > Hi Tal, > > Thanks for organizing the points. > So if I understand correctly we are looking only at "Static service > composition" which includes "substituting service template" and > "substituting service". > > As you said with "substituting service template" approach ,we will have > all the nodes aggregated from other service templates and a single workflow > would be triggered to perform life-cycle operation on all the nodes. > Am not sure why the workflows needs to be "boundary aware" for nodes being > substituted ? I see nodes are already part of the composed service, Could > you please help me understand this ? > > > Regards, > DJ > -----Original Message----- > From: Tal Liron [mailto:t...@cloudify.co] > Sent: Saturday, August 12, 2017 4:52 AM > To: dev@ariatosca.incubator.apache.org > Subject: Re: Service Composition / Substitution Mapping > > You are correct -- to participate in this "multi-VIM" scenario, the > Openstack plugin would have to know how to translate the TOSCA properties > to a flavor ID. This could all be done in 100% TOSCA via policies (say, an > aria.Openstack). > > Doing this automatically might not be a good idea, or even necessary. > Worst case is you get a validation error if the ARIA plugin can't find a > flavor in the table to match your requirements, in which you case you can > go and manually find the right ID and add it to the table. > > And I agree about being fine with imperfection: the rule of thumb for our > work is always to allow for sensible defaults even if no explicit policy is > given. > > Anyway, we've gone way off the topic of this thread. We can talk about it > more once it comes closer to an implementation. > > On Fri, Aug 11, 2017 at 3:52 PM, DeWayne Filppi <dewa...@cloudify.co> > wrote: > > > Interesting. Take Openstack (please <rim shot>). If you model a > > compute OS as explicitly as you like, there still has to be a "match" > > to an Openstack image ID. Or are you saying you must supply the image > ID for > > OSs. Likewise, you can't supply RAM and CPUs without a flavor ID. > > Openstack does allow for custom flavors, but I doubt the plugin is > > doing that. Much better to have a "caps-init" interface in some low > > down base type that can be triggered at service creation to support > > reqs/caps (IMHO). Then the plugin can verify whether the service can > > be construct based on fuzzy constraints. Maybe this is a case that > > the "general solution" is a nightmare of complexity, but having a > > plugin scan the available flavors to make sure a requirement can be > > met doesn't seem that tough. If TOSCA provided a formal lifecycle > > interface for it, then orchestrators or just plugins could determine > > how tricky they wanted to be. IOW, let not the perfect be the enemy of > the good. > > > > DeWayne > > > > > > On Fri, Aug 11, 2017 at 1:26 PM, Tal Liron <t...@cloudify.co> wrote: > > > > > OK, that's a whole different can of worms. :) > > > > > > TOSCA's Compute capabilities (Container and OperatingSystem) are > > explicit. > > > You specify which OS you want, how much RAM you want, how many CPUs, > etc. > > > ARIA's explicit node types (for example, the AWS Compute node) are > > likewise > > > explicit. So there is not querying here: the plugin will attempt to > > > spin > > up > > > exactly the virtual machine you asked for. If it fails, the workflow > > > will fail. > > > > > > This is not good enough, I think, for real world scenarios. There > > > are two possible solutions: > > > > > > 1) Support ranges or fallbacks. So instead of saying "I need 4 GB of > RAM" > > > you can say "I would like 4 GB of RAM, but 2 GB would also be OK." > > There's > > > no easy way to do this now without totally changing how the > > > capability types are designed. But, it may be possible to override > > > this via > > policies. > > > So, the capabilities would perhaps specify the minimal requirements, > > while > > > policies specify preferences. Some aspects of this were discussed in > > > the OPEN-O project. DeWayne, has any of this survived in ONAP, or > > > have we not reached that point in the discussion yet? > > > > > > 2) Incorporate this into the bigger topic of resource orchestration. > > This a > > > huge challenge for the industry. The problem field contains not just > > > "I need X amount of RAM" but also "I want all my virtual machines > > > and containers running on the same high-performance network backend > > > and have these two nodes on the same bare-metal machine or at least > > > in the same > > data > > > center rack with a NUMA interconnect, and I also don't want all this > > > to cost more that $100 per hour." That's not crazy: these are real > > > world requirements for high-performance VNFs and service chaining. > > > Resource orchestration requires a full map of what is available in > > > the data > > centers, > > > a negotiation-based algorithm for properly allocating and placing > > > resources, connection to billing services, etc. Of course resource > > > orchestration is not within the scope of ARIA, but it would be great > > > for ARIA to have plugins for them (and TOSCA be able to model > > > resource requirement policies) when these become available. > > > > > > > > > > > > On Fri, Aug 11, 2017 at 3:02 PM, DeWayne Filppi > > > <dewa...@cloudify.co> > > > wrote: > > > > > > > For the "resource realization" part, I was not even considering > > > > multicloud/vim. I was considering single cloud even outside of > > > > composition. Just reqs and caps. If my node "requires" a compute > > > > node with Suse Linux version X with a minimum of 4GB RAM, how does > the > > > > orchestrator match that without querying the cloud via the plugin? > If > > > it > > > > does query it, which it seems it must in addition to the implicit > > > > quota query, how is this done? TOSCA seems to not really care, > > > > which is fine > > > and > > > > perhaps a good idea. But ARIA has to care. > > > > > > > > DeWayne > > > > > > > > On Fri, Aug 11, 2017 at 12:34 PM, Tal Liron <t...@cloudify.co> wrote: > > > > > > > > > In my opinion, the new composite service should indeed be a > > > > > single > > > > service, > > > > > so the ARIA CLI will show it as one (if a substituting service > > already > > > > > existed, it would be "dissolved" into the new top-level > > > > > service). The composition will show its traces if you look more > > > > > deeply, because > > > you'll > > > > > see that some node templates come from a different service > template. > > > > > Perhaps our CLI can detect this and tell the user which service > > > templates > > > > > were used to create the nodes in current service. > > > > > > > > > > The rest of what you described I think is not related directly > > > > > to > > > service > > > > > composition, but resource realization. The way we currently > > > > > handle > > this > > > > is > > > > > by being very explicit: you have to use the derived Compute > > > > > types > > > > included > > > > > in our AWS or Openstack plugins, for example. > > > > > > > > > > I think we should still keep this, because sometimes you want to > > > > > be > > > very > > > > > explicit (and use specific features of the platform), but I > > > > > actually > > > have > > > > > some ideas for multi-VIM support that would work differently: > > > > > the > > idea > > > is > > > > > to use the basic Compute type, with detailed properties in its > > > Container > > > > > and OperatingSystem capabilities. There would then be a policy, > > perhaps > > > > > aria.Realization, that would hint at what kind of Compute nodes > > > > > you > > are > > > > > looking for (you can apply the policy to specific nodes or > > > > > groups, or generally). It might be possible to still have strong > > platform-specific > > > > > feature support here, perhaps by implementing a general string > > > key-value > > > > > dict that could include hints for the specific plugin to use. > > > > > > > > > > Anyway, just ideas at this point... > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 2:25 PM, DeWayne Filppi > > > > > <dewa...@cloudify.co > > > > > > > > wrote: > > > > > > > > > > > Good stuff. Obviously a "fancy include" was the wrong metaphor. > > The > > > > > > lifecycles are linked though. When a create the referencing > > service > > > > > (aria > > > > > > service create), ARIA will match the other service template, > > > > > > and > > run > > > > > create > > > > > > service on it, and then continue on. Uninstall of the > > > > > > referencer > > > will > > > > > > destroy the referenced service. After such an instantiation, > > > > > > would > > > > > listing > > > > > > all services produce two (perhaps with some parent indicator) > > > > > > or > > one? > > > > > From > > > > > > what I've seen, reqs and caps don't have the equivalent of a > > > lifecycle > > > > > > interface associated. Maybe I missed it. It is implicit in > > > > > > the orchestrator to match stuff up at service creation time, > > > > > > I'm > > > assuming, > > > > > and > > > > > > using plugins somehow? The sort of flagship scenario is > > > > > > matching > > > cloud > > > > > > compute resources without specifying an image. Since > > > > > > discovering > > > > > whether a > > > > > > particular cloud has a given capability, I guessing a plugin > > > > > > for > > that > > > > > cloud > > > > > > has to search cloud inventory and perhaps quota to provide > > > > > > info for > > > the > > > > > > match. Since plugins (at least the Cloudify variety) tend to > > > > > > be > > > > > triggered > > > > > > due to operations that they connect to lifecycle methods, and > > > > > > there > > > is > > > > no > > > > > > "materialize_capabilities" interface that I noticed, some > > > > > > other > > kind > > > of > > > > > > magic must be performed. Maybe this is an > > > > > > orchestrator-specific > > > detail > > > > > > outside of TOSCA. > > > > > > > > > > > > DeWayne > > > > > > > > > > > > On Fri, Aug 11, 2017 at 10:57 AM, Tal Liron <t...@cloudify.co> > > wrote: > > > > > > > > > > > > > OK, let me try to organize this differently. Three potential > > > > > conceptions > > > > > > > here: > > > > > > > > > > > > > > 1) A "fancy include," as you say. All that would happen here > > > > > > > is > > > that > > > > > the > > > > > > > TOSCA parser would automatically find the service template > > > > > > > to > > > include > > > > > > from > > > > > > > some kind of repository of recognized service templates, and > > > > > > > just > > > > > include > > > > > > > that. The language in the TOSCA spec suggests that this is > > > > > > > not > > the > > > > > > > intention: it is something that happens at the "orchestrator," > > not > > > > the > > > > > > > "parser." > > > > > > > > > > > > > > 2) Static service composition. This happens not at the > > > > > > > parsing > > > stage, > > > > > but > > > > > > > rather the instantiation stage, where reqs-and-caps happens. > > > > > > > I > > > think > > > > > this > > > > > > > is what is intended: substitution mapping is specifically > > > > > > > about > > > > mapping > > > > > > > reqs-and-caps. And this is also where things like > > > > > > > scalability and > > > > > > placement > > > > > > > happen: think of a requirement matching a capability, but > > > > > > > that > > > > > capability > > > > > > > not having enough capacity. So, the node might be able to > > > > > > > scale > > > out: > > > > in > > > > > > the > > > > > > > case of substitution, this would mean duplicating an entire > > service > > > > > > > instance. My understanding is that this is what is intended > > > > > > > by > > > TOSCA, > > > > > and > > > > > > > it's entirely within the scope of what we can do. We've > > > > > > > recently > > > just > > > > > > > refactored the instantiation phase into a new "topology" > > > > > > > module > > > where > > > > > > > exactly all this logic is concentrated. So think of it this > > > > > > > way > > -- > > > > ARIA > > > > > > has > > > > > > > three parts: "parser," "topology manager," and "workflow > engine" > > > > > > > ("orchestrator"). > > > > > > > > > > > > > > (I think I might have confused some thing here a bit when > > > mentioning > > > > a > > > > > > > "logical proxy node." I do not mean an actual piece of proxy > > > software > > > > > > > running on a machine somewhere. I mean just a data point in > > > > > > > the ARIA-generated topology that can be used as a barrier of > > > > > > > sorts > > when > > > > > > > constructing the task graph -- because the task graph > > > > > > > follows relationships, the edges of the topology. It could > > > > > > > be that we > > > > discover > > > > > in > > > > > > > our POC that this is not needed, because actually the > > substitution > > > > node > > > > > > is > > > > > > > already part of our topology data model and we might be able > > > > > > > to > > > > easily > > > > > > take > > > > > > > that into account when generating the task graph.) > > > > > > > > > > > > > > 3) Live service composition. I think this is a fantasy of > > > > > > > some > > > > people: > > > > > > that > > > > > > > ARIA would be able to take existing, "running" service > > > > > > > chains and > > > run > > > > > > > workflows with them. I do think this is a solvable problem, > > > > > > > but > > not > > > > via > > > > > > > substitution mapping per se. A solution could involve > > > > > > > deployment > > > of a > > > > > > proxy > > > > > > > service (which could actually be encapsulated in a > > > > > > > substitution > > > > > mapping, > > > > > > > but doesn't have to be), or configuring specialized virtual > > > > > > > ports > > > via > > > > > an > > > > > > > SDN controller, or via logical proxy nodes created via an > > > inspection > > > > > > tool, > > > > > > > etc. I cannot believe that there is a one-size-fits-all > > > > > > > solution > > to > > > > > this > > > > > > > problem. The dream of NFV might be to have everything > > > > > > > connect to > > > each > > > > > > other > > > > > > > like LEGO blocks, but there are far too many protocols, > > > > configuration, > > > > > > and > > > > > > > security standards. I think point #2, what I called "static > > service > > > > > > > composition," is a realistic compromise and within the scope > > > > > > > of > > > what > > > > > > TOSCA > > > > > > > promises. > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 12:08 PM, DeWayne Filppi < > > > > dewa...@cloudify.co> > > > > > > > wrote: > > > > > > > > > > > > > > > To my eyes, the spec doesn't speak of runtime service > > > substituion, > > > > > but > > > > > > > > parsetime template composition. IOW, substitution mapping > > > > > > > > is a > > > > fancy > > > > > > > > "include", or the equivalent of an interface definition. > > > > > > > > Is it > > > > > > > understood > > > > > > > > by the ARIA team that this includes proxying of running > > services? > > > > > IOW, > > > > > > > if > > > > > > > > my template requires a database service that my template > > > > > > > > does > > > *not* > > > > > > want > > > > > > > to > > > > > > > > control the lifecycle of, I can "substitution map" an > > > > > > > > instance > > > of a > > > > > > > > template (i.e. a running service)? This would be a lovely > > > feature, > > > > > but > > > > > > > > it's not really a "substitution map", rather more of a > > > > > > > > "service > > > > > proxy" > > > > > > > (as > > > > > > > > implemented as a plugin in Cloudify). Just trying to > clarify. > > > > > Maybe > > > > > > > the > > > > > > > > community thinks that "substitution map" as something that > > occurs > > > > > > beyond > > > > > > > > parsetime, or should. > > > > > > > > > > > > > > > > On Fri, Aug 11, 2017 at 9:52 AM, Tal Liron > > > > > > > > <t...@cloudify.co> > > > > wrote: > > > > > > > > > > > > > > > > > Well, DJ, it's all just opinion at this stage and > > > > > > > > > feedback is > > > > > > welcome! > > > > > > > > > > > > > > > > > > Option 1: > > > > > > > > > > To look at satisfying nodes present in a > > substituting > > > > > > > service, > > > > > > > > > > Have these nodes part of the newly created service and > > remove > > > > the > > > > > > > > > > substituting service(nodes with different ID's. Also > > > > > > > > > > we are > > > > very > > > > > > much > > > > > > > > in > > > > > > > > > > favor of UUID ) > > > > > > > > > > With this approach I guess the substituting > > > > > > > > > > service > > > > > should > > > > > > > not > > > > > > > > > > have any associated workflows running. If at all an > > workflow > > > > > > > execution > > > > > > > > is > > > > > > > > > > already triggered I hope this service will not be > > considered > > > > for > > > > > > > > > > substitution. > > > > > > > > > > I hope this is the correct approach when we > > > > > > > > > > are > > > looking > > > > > at > > > > > > a > > > > > > > > > > service for the substitution > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, this is a good idea. It would be easy to discover > > > > > > > > > this > > > > > according > > > > > > > to > > > > > > > > > the stored node state -- just make sure that all nodes > > > > > > > > > are > > in a > > > > > > stable > > > > > > > > > state before composition. > > > > > > > > > > > > > > > > > > This leads to a general issue: before composition, the > > > > substituting > > > > > > > > > services must be validated in some way before > > > > > > > > > composition > > > begins. > > > > > > > > > > > > > > > > > > Also, let's all start using TOSCA terminology here: the > > > > containing > > > > > > > > service > > > > > > > > > is called the "top-level service," and the contained > > > > > > > > > services > > > are > > > > > > > called > > > > > > > > > "substituting services." > > > > > > > > > > > > > > > > > > Also, we keep using trivial examples, but it's > > > > > > > > > definitely > > > > possible > > > > > > for > > > > > > > a > > > > > > > > > top-level service to require several substituting > > > > > > > > > services at > > > the > > > > > > same > > > > > > > > > time. I can definitely see such things happening in NFV > > > > > > > > > with > > > even > > > > > > > simple > > > > > > > > > service chains. Basically every VNF could be a > > > > > > > > > substituting > > > > > service. > > > > > > > > > > > > > > > > > > So, actually one of the validations would be to make > > > > > > > > > sure you > > > do > > > > > not > > > > > > > > create > > > > > > > > > circular composition: if the top-level also has > > > > > > > > > substitution > > > > > > mappings, > > > > > > > > you > > > > > > > > > need to make sure that one of the substituting ones > > > > > > > > > doesn't > > > > require > > > > > > it. > > > > > > > > :) > > > > > > > > > Not a very likely situation, but it would lead to failure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Option 2: > > > > > > > > > > While creating a service look at the req-caps > > > > > > > > > > at > > the > > > > > > > > > > service-template level and create a service including > > > > > > > > > > the > > > nodes > > > > > > > > provided > > > > > > > > > by > > > > > > > > > > the substituting service-template. With this approach > > > > > > > > > > there > > > > would > > > > > > not > > > > > > > > be > > > > > > > > > > any service created from the service-template which is > > > > > providing > > > > > > > the > > > > > > > > > > substitution functionality. The service-template would > > remain > > > > the > > > > > > > same > > > > > > > > > but > > > > > > > > > > only the service would be added with extra nodes. > > > > > > > > > > > > > > > > > > > > Are you considering both option 1 & 2 for the > > implementation > > > ? > > > > If > > > > > > not > > > > > > > > > > which one do you feel which take priority. I see > > > > > > > > > > option 2 > > at > > > > this > > > > > > > stage > > > > > > > > > > could be the best possible approach Also could you > > > > > > > > > > please let me know a tentative time for this > > > > > feature > > > > > > > to > > > > > > > > be > > > > > > > > > > available? > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think both options 1 and 2 make sense and are useful, > > > > > > > > > and > > > > > actually > > > > > > > one > > > > > > > > is > > > > > > > > > a subset of the other. > > > > > > > > > > > > > > > > > > With option #2 (substituting a service template), it > > > > > > > > > means > > new > > > > > nodes > > > > > > > are > > > > > > > > > instantiated and the composed service would include all > > nodes. > > > > So, > > > > > an > > > > > > > > > "install" workflow would install everything at once. In > > > > > > > > > this > > > case > > > > > we > > > > > > do > > > > > > > > > need to fix the lifecycle workflows to be "boundary aware," > > so > > > > that > > > > > > > > > workflows of substituting service nodes are part of > > > > > > > > > their own > > > > task > > > > > > > > graph. I > > > > > > > > > think that possibly using a logical proxy node in > > > > > > > > > between > > might > > > > > solve > > > > > > > > this > > > > > > > > > situation automatically. > > > > > > > > > > > > > > > > > > With option #1 (substituting a service) the substituting > > nodes > > > > > might > > > > > > > > > already be installed. Or not (they might have been > > instantiated > > > > but > > > > > > > still > > > > > > > > > not installed). So, the lifecycle workflows should only > > > > > > > > > work > > on > > > > the > > > > > > > nodes > > > > > > > > > that have not yet been installed. > > > > > > > > > > > > > > > > > > The point is that we just need to beef up the lifecycle > > > workflows > > > > > to > > > > > > > > > properly work with boundaries and with a mix of nodes in > > > > different > > > > > > > > states. > > > > > > > > > If they can do that, then they can handle any kind of > > > > > > > > > service > > > > > > > > composition, > > > > > > > > > whether it's option #1 or option #2. > > > > > > > > > > > > > > > > > > I don't think we can provide a timeline yet, but I will > > > > > > > > > say > > > that > > > > we > > > > > > are > > > > > > > > in > > > > > > > > > the research phase and may have a POC soon. Avia is in > > > > > > > > > charge > > > of > > > > > this > > > > > > > > JIRA, > > > > > > > > > so I will let him chime in with the current state of > things. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
