[openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Stan Lagun]

Hello,

I’m one of the engineer working on Murano project. Recently we started a
discussion about Murano and Heat Software orchestration and I want to
continue this discussion with more technical details.

In our project we do deployment of complex multi-instance Windows services.
Those services usually require specific multi-VM orchestration that is
currently impossible or at least not that easy to achieve with Heat. As you
are currently doing HOT software orchestration design we would like to
participate in HOT Software orchestration design and contribute into it, so
that the Heat could address use-cases which we believe are very common.

For example here is how deployment of a SQL Server cluster goes:

   1.

   Allocate Windows VMs for SQL Server cluster
   2.

   Enable secondary IP address from user input on all SQL Windows instances
   3.

   Install SQL Server prerequisites on each node
   4.

   Choose a master node and install Failover Cluster on it
   5.

   Configure all nodes so that they know which one of them is the master
   6.

   Install SQL Server on all nodes
   7.

   Initialize AlwaysOn on all nodes except for the master
   8.

   Initialize Primary replica
   9.

   Initialize secondary replicas


All of the steps must take place in appropriate order depending on the
state of other nodes. Some steps require an output from previous steps and
all of them require some input parameters. SQL Server requires an Active
Directory service in order to use Failover mechanism and installation of
Active Directory with primary and secondary controllers is a complex
workflow of its own.

That is why it is necessary to have some central coordination service which
would handle deployment workflow and perform specific actions (create VMs
and other OpenStack resources, do something on that VM) on each stage
according to that workflow. We think that Heat is the best place for such
service.

Our idea is to extend HOT DSL by adding  workflow definition capabilities
as an explicit list of resources, components’ states and actions. States
may depend on each other so that you can reach state X only after you’ve
reached states Y and Z that the X depends on. The goal is from initial
state to reach some final state “Deployed”.

There is such state graph for each of our deployment entities (service,
VMs, other things). There is also an action that must be performed on each
state.
For example states graph from example above would look like this:






The goal is to reach Service_Done state which depends on VM1_Done and
VM2_Done states and so on from initial Service_Start state.

We propose to extend HOT DSL with workflow definition capabilities where
you can describe step by step instruction to install service and properly
handle errors on each step.

We already have an experience in implementation of the DSL, workflow
description and processing mechanism for complex deployments and believe
we’ll all benefit by re-using this experience and existing code, having
properly discussed and agreed on abstraction layers and distribution of
responsibilities between OS components. There is an idea of implementing
part of workflow processing mechanism as a part of Convection proposal,
which would allow other OS projects to benefit by using this.

We would like to discuss if such design could become a part of future Heat
version as well as other possible contributions from Murano team.

Regards,
Stan Lagun

-- 

Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
sla...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Angus Salkeld]

On 09/10/13 00:53 +0400, Stan Lagun wrote:

Hello,

I’m one of the engineer working on Murano project. Recently we started a
discussion about Murano and Heat Software orchestration and I want to
continue this discussion with more technical details.


I hope you are going to be at summit, as I expect this to an important
session we have there:

Related summit sessions:
http://summit.openstack.org/cfp/details/82
http://summit.openstack.org/cfp/details/121
http://summit.openstack.org/cfp/details/78

Related blueprints:
https://blueprints.launchpad.net/heat/+spec/software-configuration-provider
https://blueprints.launchpad.net/heat/+spec/hot-software-config-deps
https://blueprints.launchpad.net/heat/+spec/hot-software-config
https://blueprints.launchpad.net/heat/+spec/windows-instances

Excuse me if you are well aware of these.

-Angus



In our project we do deployment of complex multi-instance Windows services.
Those services usually require specific multi-VM orchestration that is
currently impossible or at least not that easy to achieve with Heat. As you
are currently doing HOT software orchestration design we would like to
participate in HOT Software orchestration design and contribute into it, so
that the Heat could address use-cases which we believe are very common.

For example here is how deployment of a SQL Server cluster goes:

  1.

  Allocate Windows VMs for SQL Server cluster
  2.

  Enable secondary IP address from user input on all SQL Windows instances
  3.

  Install SQL Server prerequisites on each node
  4.

  Choose a master node and install Failover Cluster on it
  5.

  Configure all nodes so that they know which one of them is the master
  6.

  Install SQL Server on all nodes
  7.

  Initialize AlwaysOn on all nodes except for the master
  8.

  Initialize Primary replica
  9.

  Initialize secondary replicas


All of the steps must take place in appropriate order depending on the
state of other nodes. Some steps require an output from previous steps and
all of them require some input parameters. SQL Server requires an Active
Directory service in order to use Failover mechanism and installation of
Active Directory with primary and secondary controllers is a complex
workflow of its own.

That is why it is necessary to have some central coordination service which
would handle deployment workflow and perform specific actions (create VMs
and other OpenStack resources, do something on that VM) on each stage
according to that workflow. We think that Heat is the best place for such
service.

Our idea is to extend HOT DSL by adding  workflow definition capabilities
as an explicit list of resources, components’ states and actions. States
may depend on each other so that you can reach state X only after you’ve
reached states Y and Z that the X depends on. The goal is from initial
state to reach some final state “Deployed”.

There is such state graph for each of our deployment entities (service,
VMs, other things). There is also an action that must be performed on each
state.
For example states graph from example above would look like this:






The goal is to reach Service_Done state which depends on VM1_Done and
VM2_Done states and so on from initial Service_Start state.

We propose to extend HOT DSL with workflow definition capabilities where
you can describe step by step instruction to install service and properly
handle errors on each step.

We already have an experience in implementation of the DSL, workflow
description and processing mechanism for complex deployments and believe
we’ll all benefit by re-using this experience and existing code, having
properly discussed and agreed on abstraction layers and distribution of
responsibilities between OS components. There is an idea of implementing
part of workflow processing mechanism as a part of Convection proposal,
which would allow other OS projects to benefit by using this.

We would like to discuss if such design could become a part of future Heat
version as well as other possible contributions from Murano team.


I am really happy that you want to get involved and this sounds like it
functionally matches quite well to the blueprints at the top.

-Angus



Regards,
Stan Lagun

--

Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
sla...@mirantis.com



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Georgy Okrokvertskhov]

Hi Angus,

We will have representatives from our Team . Alex Tivelkov and I will be on
summit. We definitely will participate in design sessions for these hot
topics.

Before the summit we will work in etherpads to add necessary technical
information to have a solid background for discussions. We already checked
BPs and we think we can add more details to them.

As I see, not all BP have etherpad links published on launchpad. Should we
create them and attach to BP's whiteboards?

Do you have any internal design sessions before summit? It would be nice if
we can participate in them too.

Thanks
Gosha






On Tue, Oct 8, 2013 at 2:46 PM, Angus Salkeld  wrote:

> On 09/10/13 00:53 +0400, Stan Lagun wrote:
>
>> Hello,
>>
>> I’m one of the engineer working on Murano project. Recently we started a
>> discussion about Murano and Heat Software orchestration and I want to
>> continue this discussion with more technical details.
>>
>
> I hope you are going to be at summit, as I expect this to an important
> session we have there:
>
> Related summit sessions:
> http://summit.openstack.org/**cfp/details/82
> http://summit.openstack.org/**cfp/details/121
> http://summit.openstack.org/**cfp/details/78
>
> Related blueprints:
> https://blueprints.launchpad.**net/heat/+spec/software-**
> configuration-provider
> https://blueprints.launchpad.**net/heat/+spec/hot-software-**config-deps
> https://blueprints.launchpad.**net/heat/+spec/hot-software-**config
> https://blueprints.launchpad.**net/heat/+spec/windows-**instances
>
> Excuse me if you are well aware of these.
>
> -Angus
>
>
>> In our project we do deployment of complex multi-instance Windows
>> services.
>> Those services usually require specific multi-VM orchestration that is
>> currently impossible or at least not that easy to achieve with Heat. As
>> you
>> are currently doing HOT software orchestration design we would like to
>> participate in HOT Software orchestration design and contribute into it,
>> so
>> that the Heat could address use-cases which we believe are very common.
>>
>> For example here is how deployment of a SQL Server cluster goes:
>>
>>   1.
>>
>>
>>   Allocate Windows VMs for SQL Server cluster
>>   2.
>>
>>
>>   Enable secondary IP address from user input on all SQL Windows instances
>>   3.
>>
>>
>>   Install SQL Server prerequisites on each node
>>   4.
>>
>>
>>   Choose a master node and install Failover Cluster on it
>>   5.
>>
>>
>>   Configure all nodes so that they know which one of them is the master
>>   6.
>>
>>
>>   Install SQL Server on all nodes
>>   7.
>>
>>
>>   Initialize AlwaysOn on all nodes except for the master
>>   8.
>>
>>   Initialize Primary replica
>>   9.
>>
>>
>>   Initialize secondary replicas
>>
>>
>> All of the steps must take place in appropriate order depending on the
>> state of other nodes. Some steps require an output from previous steps and
>> all of them require some input parameters. SQL Server requires an Active
>> Directory service in order to use Failover mechanism and installation of
>> Active Directory with primary and secondary controllers is a complex
>> workflow of its own.
>>
>> That is why it is necessary to have some central coordination service
>> which
>> would handle deployment workflow and perform specific actions (create VMs
>> and other OpenStack resources, do something on that VM) on each stage
>> according to that workflow. We think that Heat is the best place for such
>> service.
>>
>> Our idea is to extend HOT DSL by adding  workflow definition capabilities
>> as an explicit list of resources, components’ states and actions. States
>> may depend on each other so that you can reach state X only after you’ve
>> reached states Y and Z that the X depends on. The goal is from initial
>> state to reach some final state “Deployed”.
>>
>> There is such state graph for each of our deployment entities (service,
>> VMs, other things). There is also an action that must be performed on each
>> state.
>> For example states graph from example above would look like this:
>>
>>
>>
>>
>>
>>
>> The goal is to reach Service_Done state which depends on VM1_Done and
>> VM2_Done states and so on from initial Service_Start state.
>>
>> We propose to extend HOT DSL with workflow definition capabilities where
>> you can describe step by step instruction to install service and properly
>> handle errors on each step.
>>
>> We already have an experience in implementation of the DSL, workflow
>> description and processing mechanism for complex deployments and believe
>> we’ll all benefit by re-using this experien

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Clint Byrum]

Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700:
> Hello,
> 
> I’m one of the engineer working on Murano project. Recently we started a
> discussion about Murano and Heat Software orchestration and I want to
> continue this discussion with more technical details.
> 
> In our project we do deployment of complex multi-instance Windows services.
> Those services usually require specific multi-VM orchestration that is
> currently impossible or at least not that easy to achieve with Heat. As you
> are currently doing HOT software orchestration design we would like to
> participate in HOT Software orchestration design and contribute into it, so
> that the Heat could address use-cases which we believe are very common.
> 
> For example here is how deployment of a SQL Server cluster goes:
> 
>1.
> 
>Allocate Windows VMs for SQL Server cluster
>2.

Heat does this quite well. :)

> 
>Enable secondary IP address from user input on all SQL Windows instances
>3.

I don't understand what that means.

> 
>Install SQL Server prerequisites on each node
>4.
> 

Automating software installation is a very common aspect of most
configuration tools.

>Choose a master node and install Failover Cluster on it
>5.
> 

Leader election should be built in to services that take advantage of
it. External leader election is really hard without all of the
primitives built into the service. Otherwise you have to resort to
things like STONITH to avoid split brain. If Heat tries to do this, it
will just result in a complicated design with very few happy users.

>Configure all nodes so that they know which one of them is the master
>6.
> 

Anyway, Heat can do this quite well. The template author can just choose
one, and flag it as the master in the resource's Metadata. Point all
others at that one using Ref's to it.

"Configuring" nodes should be something an automated configuration tool
does well. It is pretty straight forward to have a tool fetch the
Metadata of a resource via the Heat API and feed it to a configuration
tool. Or you can even teach your config tool to read the Metadata.

>Install SQL Server on all nodes
>7.
> 

Same as before - any automated software installation tool should
suffice.

>Initialize AlwaysOn on all nodes except for the master
>8.
> 

I don't know what that means.

>Initialize Primary replica
>9.
> 
>Initialize secondary replicas
> 

I also don't know what that entails, but I would suspect that it is the
kind of thing you do via the SQL server API once everything is done.

For that, you just need to poll a waitcondition and when all of the
nodes have signaled the waitcondition, run this.

> 
> All of the steps must take place in appropriate order depending on the
> state of other nodes. Some steps require an output from previous steps and
> all of them require some input parameters. SQL Server requires an Active
> Directory service in order to use Failover mechanism and installation of
> Active Directory with primary and secondary controllers is a complex
> workflow of its own.
> 
> That is why it is necessary to have some central coordination service which
> would handle deployment workflow and perform specific actions (create VMs
> and other OpenStack resources, do something on that VM) on each stage
> according to that workflow. We think that Heat is the best place for such
> service.
> 

I'm not so sure. Heat is part of the Orchestration program, not workflow.

> Our idea is to extend HOT DSL by adding  workflow definition capabilities
> as an explicit list of resources, components’ states and actions. States
> may depend on each other so that you can reach state X only after you’ve
> reached states Y and Z that the X depends on. The goal is from initial
> state to reach some final state “Deployed”.
> 

Orchestration is not workflow, and HOT is an orchestration templating
language, not a workflow language. Extending it would just complect two
very different (though certainly related) tasks.

I think the appropriate thing to do is actually to join up with the
TaskFlow project and consider building it into a workflow service or tools
(it is just a library right now).

> There is such state graph for each of our deployment entities (service,
> VMs, other things). There is also an action that must be performed on each
> state.

Heat does its own translation of the orchestration template into a
workflow right now, but we have already discussed using TaskFlow to
break up the orchestration graph into distributable jobs. As we get more
sophisticated on updates (rolling/canary for instance) we'll need to
be able to reason about the process without having to glue all the
pieces together.

> We propose to extend HOT DSL with workflow definition capabilities where
> you can describe step by step instruction to install service and properly
> handle errors on each step.
> 
> We already have an experience in implementation of the DSL, wo

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Thomas Spatzier]

Excerpts from Clint Byrum's message

> From: Clint Byrum 
> To: openstack-dev ,
> Date: 09.10.2013 03:54
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700:
> > Hello,
> >
> >
> > That is why it is necessary to have some central coordination service
which
> > would handle deployment workflow and perform specific actions (create
VMs
> > and other OpenStack resources, do something on that VM) on each stage
> > according to that workflow. We think that Heat is the best place for
such
> > service.
> >
>
> I'm not so sure. Heat is part of the Orchestration program, not workflow.
>

I agree. HOT so far was thought to be a format for describing templates in
a structural, declaritive way. Adding workflows would stretch it quite a
bit. Maybe we should see what aspects make sense to be added to HOT, and
then how to do workflow like orchestration in a layer on top.

> > Our idea is to extend HOT DSL by adding  workflow definition
capabilities
> > as an explicit list of resources, components’ states and actions.
States
> > may depend on each other so that you can reach state X only after
you’ve
> > reached states Y and Z that the X depends on. The goal is from initial
> > state to reach some final state “Deployed”.
> >

We also would like to add some mechanisms to HOT for declaratively doing
software component orchestration in Heat, e.g. saying that one component
depends on another one, or needs input from another one once it has been
deployed etc. (I BTW started to write a wiki page, which is admittedly far
from complete, but I would be happy to work on it with interested folks -
https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider).
However, we must be careful not to make such features too complicated so
nobody will be able to use it any more. That said, I believe we could make
HOT cover some levels of complexity, but not all. And then maybe workflow
based orchestration on top is needed.

>
> Orchestration is not workflow, and HOT is an orchestration templating
> language, not a workflow language. Extending it would just complect two
> very different (though certainly related) tasks.
>
> I think the appropriate thing to do is actually to join up with the
> TaskFlow project and consider building it into a workflow service or
tools
> (it is just a library right now).
>
> > There is such state graph for each of our deployment entities (service,
> > VMs, other things). There is also an action that must be performed on
each
> > state.
>
> Heat does its own translation of the orchestration template into a
> workflow right now, but we have already discussed using TaskFlow to
> break up the orchestration graph into distributable jobs. As we get more
> sophisticated on updates (rolling/canary for instance) we'll need to
> be able to reason about the process without having to glue all the
> pieces together.
>
> > We propose to extend HOT DSL with workflow definition capabilities
where
> > you can describe step by step instruction to install service and
properly
> > handle errors on each step.
> >
> > We already have an experience in implementation of the DSL, workflow
> > description and processing mechanism for complex deployments and
believe
> > we’ll all benefit by re-using this experience and existing code, having
> > properly discussed and agreed on abstraction layers and distribution of
> > responsibilities between OS components. There is an idea of
implementing
> > part of workflow processing mechanism as a part of Convection proposal,
> > which would allow other OS projects to benefit by using this.
> >
> > We would like to discuss if such design could become a part of future
Heat
> > version as well as other possible contributions from Murano team.
> >
>
> Thanks really for thinking this through. Windows servers are not unique
in
> this regard. Puppet and chef are pretty decent at expressing single-node
> workflows but they are awkward for deferring control and resuming on
other
> nodes, so I do think there is a need for a general purpose distributed
> workflow definition tool.
>
> I'm not, however, convinced that extending HOT would yield a better
> experience for users. I'd prefer to see HOT have a well defined interface
> for where to defer to external workflows. Wait conditions are actually
> decent at that, and I'm sure with a little more thought we can make them
> more comfortable to use.

Good discussion to have, i.e. what extensions we would need in HOT for
making HOT alone more capable, and what we would need to hook it up with
other orchestra

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Steven Hardy]

On Wed, Oct 09, 2013 at 12:53:45AM +0400, Stan Lagun wrote:
> Hello,
> 
> I’m one of the engineer working on Murano project. Recently we started a
> discussion about Murano and Heat Software orchestration and I want to
> continue this discussion with more technical details.

Thanks, we're certainly interested in Murano, and are keen to discuss your
roadmap, and where requirements and integration opportunities exist.

> In our project we do deployment of complex multi-instance Windows services.
> Those services usually require specific multi-VM orchestration that is
> currently impossible or at least not that easy to achieve with Heat. As you
> are currently doing HOT software orchestration design we would like to
> participate in HOT Software orchestration design and contribute into it, so
> that the Heat could address use-cases which we believe are very common.
> 
> For example here is how deployment of a SQL Server cluster goes:
> 
>1.
> 
>Allocate Windows VMs for SQL Server cluster

Heat can already do this, you just define either OS::Nova::Server or
AWS::EC2::Instance resource in your template, or possibly a group of
instances via OS::Heat::InstanceGroup if the configuration is the same for
all VMs

>2.
>Enable secondary IP address from user input on all SQL Windows instances

So this again is already possible, via several resource types,
AWS::EC2::NetworkInterface, AWS::EC2::EIP, OS::Neutron::Port etc..

I suggest using the Neutron resources where possible, if you don't care
about Clouformation portability.

>3.
>Install SQL Server prerequisites on each node

So Heat is already able to do this, via a couple of methods, for Linux VMs,
so we just need the in-instance agent support for windows (cloud-init,
optionally combined with agents like cfn-init from heat-cfntools)

Can you clarify what you're using for in-instance agents currently,
cloudbase-init, and/or some bespoke tools?

>4.
>Choose a master node and install Failover Cluster on it
>5.
>Configure all nodes so that they know which one of them is the master

I'm not sure what's involved in these steps, but it seems like there are
serialization requirements, which can be handled via WaitConditions.

One thing I think we do need to look at is ways to enable expression of
serialization requirements via HOT, which don't require use of the
AWS-compatible WaitCondition resources.

So I think we already have the required functionality, we just need to
build out better native interfaces to it.

>Configure all nodes so that they know which one of them is the master
>6.
> 
>Install SQL Server on all nodes
>7.
> 
>Initialize AlwaysOn on all nodes except for the master
>8.
> 
>Initialize Primary replica
>9.
> 
>Initialize secondary replicas
> 
> 
> All of the steps must take place in appropriate order depending on the
> state of other nodes. Some steps require an output from previous steps and
> all of them require some input parameters. SQL Server requires an Active
> Directory service in order to use Failover mechanism and installation of
> Active Directory with primary and secondary controllers is a complex
> workflow of its own.

So all of this seems possible right now using WaitConditions, but as
mentioned above we should look at ways to provide a better and more
flexible native interface to similar functionality.

> That is why it is necessary to have some central coordination service which
> would handle deployment workflow and perform specific actions (create VMs
> and other OpenStack resources, do something on that VM) on each stage
> according to that workflow. We think that Heat is the best place for such
> service.

Yep, we already do coordination of VM deployment and other openstack
resources, by managing implicit and explicit dependencies between those
resources.

> Our idea is to extend HOT DSL by adding  workflow definition capabilities
> as an explicit list of resources, components’ states and actions. States
> may depend on each other so that you can reach state X only after you’ve
> reached states Y and Z that the X depends on. The goal is from initial
> state to reach some final state “Deployed”.

IMHO there isn't a real need to provide explicit control of the workflow
implied by the resource dependencies for the sort of use-case you describe.

What I think is needed is simply a better native interface to serialization
primitives/resources.

> There is such state graph for each of our deployment entities (service,
> VMs, other things). There is also an action that must be performed on each
> state.
> For example states graph from example above would look like this:
> 
> The goal is to reach Service_Done state which depends on VM1_Done and
> VM2_Done states and so on from initial Service_Start state.
> 
> We propose to extend HOT DSL with workflow definition capabilities where
> you can describe step by step instruction to install service and properly
> handle erro

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Stan Lagun]

> Thanks, we're certainly interested in Murano, and are keen to discuss your
> roadmap, and where requirements and integration opportunities exist
Glad to here it. The same is true from Murano side.

On sample SQL workflow: that was just an example. I didn't want to bother
you with a SQL Server deployment details as it's really not that important.
What I've tried to say is that deployments consists of many steps, the
steps vary depending on instance's role, user input and so on. The step on
one machine
often requires some other machine already be in some state or some output
from a deployment step happened elsewhere.

I do understand that it is doable using Heat alone. Actually we do use Heat
for some parts of workflow. We do not talk to Nova or Neutron directly.
The special use case of Murano is that there is no HOT template author.
Heat is more an administrator's tool who knows how to write HOT templates
and wants to deal with low-level configuration aspects. But Murano is quite
different. In Murano the developers of workflows/scripts/metadata/etc are
not end-users. The user is not doing any sort of programming. He is given a
UI dashboard where he can compose desired environment from available
building blocks (services). Services may depend on each other and UI guides
him how to fulfill all requirements. User also conigures service's (VM's
etc.) settings. The number of instances in SQL Server cluster and which one
of them is going to be the master are such settings.

Because we do not know in advance all the instances and resources that
would be required for the services that user has chosen and the deployment
process is strongly depends on user input we cannot just have some
hardcoded HOT template. So what we do is we dynamically generate HOT
templates my parameterizing and merging several simpler templates together.
Then we use our orchestration engine to send commands to Murano Agents on
VMs to perform deployment steps in correct order with all needed input.
Probably we could do it without orchestration but then we would need to
dynamically generate all that WaitConditions and waiting/signaling scripts
etc. - something that would be error-prone and hard to manage at large
scales.

So we do believe that some external state orchestration would be very very
helpful for complex software services we deal with. Although Murano
currently has such engine it is far from perfect. As we thought on cleaner
and more explicit approach for state orchestration we came to a vision how
to implement it on top of task-orchestration engine like TaskFlow. And then
we came up with idea that we can go even farther and implement
TaskFlow-as-a-Service service with its own REST API etc that could handle
an abstract task orchestration while leaving all deployment-related out of
the scope of such service. That opens many additional opportunities for
integration that would not be available if we just use TaskFlow library
in-process.

But we do believe that it would be better for all of us if it would be Heat
and not Murano who provides such orchestration capabilities.
And yes, I'm completely agree with you that it should not be a part of HOT
templates but something external to it. To my understanding external
orchestration needs to be capable of
1. Process some object (JSON) model describing what services/resources/etc.
are need to be deployed
2. During orchestration invoke some HOT templates (create nested stacks?)
with passing required attributes from that object model as an inputs for
the HOT template
3. Be able to send commands (bash scripts, puppet manifests, chef recipes
etc) to VM in correct order with a parameters taken from object model and
HOT outputs. See https://wiki.openstack.org/wiki/Murano/UnifiedAgent on how
it is in Murano

We are currently communicating with TaskFlow team on possible contributions
to Convection implementation and would be glad to participate in software
orchestration part on the Heat side. I'm not pretend that I know all the
answers or at least our design is good but in Murano we gained much
experience in software orchestration that might be useful for the Heat team
and we would definitely like to share our ideas. I also believe that now is
the time for that as on summit it may be too late because all the principal
decisions are already made.

On Wed, Oct 9, 2013 at 1:24 PM, Steven Hardy  wrote:

> On Wed, Oct 09, 2013 at 12:53:45AM +0400, Stan Lagun wrote:
> > Hello,
> >
> > I’m one of the engineer working on Murano project. Recently we started a
> > discussion about Murano and Heat Software orchestration and I want to
> > continue this discussion with more technical details.
>
> Thanks, we're certainly interested in Murano, and are keen to discuss your
> roadmap, and where requirements and integration opportunities exist.
>
> > In our project we do deployment of complex multi-instance Windows
> services.
> > Those services usually require specific multi-VM orchestration that is
> > currently im

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Georgy Okrokvertskhov]

Hi,

In addition I want to add couple words about flexibility and debugging
capabilities. I believe it is quite important for HOT template engine to
control all aspects of deployment process execution including software
components. Right now I believe Heat lack of control of what is going on
the VM side.  In my opinion, HOT template user should be able to define
what steps are necessary to deploy complex environment and more important,
he should be able to provide a hints to the engine how to deal with errors
during deployment. Centralized orchestration sees the whole picture of the
environment status while scripts on VM usually have quite limited view.
Workflows specification can have on_error actions and centralized
orchestration will be capable to make smart decision on how to handle
errors during deployment.

I think we need to have a design discussion about the architecture of this
centralized orchestration. From the one side, this orchestration should
have the whole information about environment state and as Heat has full
exposure to the environment it sounds reasonable to have such orchestration
as a part of Heat. At the other side, HOT template should be quite simple
to be useful, so additional workflows concepts might overload DSL syntax
and additional independent orchestration level also sounds quite reasonable
and this is what we have now as a Murano project.

It will be nice to have some initial live discussion before the summit as
not all developers will be on summit. What do you think about Google
hangout session at the end of this week or on the next week?

Thanks
Gosha









On Wed, Oct 9, 2013 at 7:52 AM, Stan Lagun  wrote:

> > Thanks, we're certainly interested in Murano, and are keen to discuss
> your
> > roadmap, and where requirements and integration opportunities exist
> Glad to here it. The same is true from Murano side.
>
> On sample SQL workflow: that was just an example. I didn't want to bother
> you with a SQL Server deployment details as it's really not that important.
> What I've tried to say is that deployments consists of many steps, the
> steps vary depending on instance's role, user input and so on. The step on
> one machine
> often requires some other machine already be in some state or some output
> from a deployment step happened elsewhere.
>
> I do understand that it is doable using Heat alone. Actually we do use
> Heat for some parts of workflow. We do not talk to Nova or Neutron directly.
> The special use case of Murano is that there is no HOT template author.
> Heat is more an administrator's tool who knows how to write HOT templates
> and wants to deal with low-level configuration aspects. But Murano is quite
> different. In Murano the developers of workflows/scripts/metadata/etc are
> not end-users. The user is not doing any sort of programming. He is given a
> UI dashboard where he can compose desired environment from available
> building blocks (services). Services may depend on each other and UI guides
> him how to fulfill all requirements. User also conigures service's (VM's
> etc.) settings. The number of instances in SQL Server cluster and which one
> of them is going to be the master are such settings.
>
> Because we do not know in advance all the instances and resources that
> would be required for the services that user has chosen and the deployment
> process is strongly depends on user input we cannot just have some
> hardcoded HOT template. So what we do is we dynamically generate HOT
> templates my parameterizing and merging several simpler templates together.
> Then we use our orchestration engine to send commands to Murano Agents on
> VMs to perform deployment steps in correct order with all needed input.
> Probably we could do it without orchestration but then we would need to
> dynamically generate all that WaitConditions and waiting/signaling scripts
> etc. - something that would be error-prone and hard to manage at large
> scales.
>
> So we do believe that some external state orchestration would be very very
> helpful for complex software services we deal with. Although Murano
> currently has such engine it is far from perfect. As we thought on cleaner
> and more explicit approach for state orchestration we came to a vision how
> to implement it on top of task-orchestration engine like TaskFlow. And then
> we came up with idea that we can go even farther and implement
> TaskFlow-as-a-Service service with its own REST API etc that could handle
> an abstract task orchestration while leaving all deployment-related out of
> the scope of such service. That opens many additional opportunities for
> integration that would not be available if we just use TaskFlow library
> in-process.
>
> But we do believe that it would be better for all of us if it would be
> Heat and not Murano who provides such orchestration capabilities.
> And yes, I'm completely agree with you that it should not be a part of HOT
> templates but something external to it. To my un

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Alex Rudenko]

Hi everyone,

I've read this thread and I'd like to share some thoughts. In my opinion,
workflows (which run on VMs) can be integrated with heat templates as
follows:

   1. workflow definitions should be defined separately and processed by
   stand-alone workflow engines (chef, puppet etc).
   2. the HOT resources should reference workflows which they require,
   specifying a type of workflow and the way to access a workflow definition.
   The workflow definition might be provided along with HOT.
   3. Heat should treat the orchestration templates as transactions (i.e.
   Heat should be able to rollback in two cases: 1) if something goes wrong
   during processing of an orchestration workflow 2) when a stand-alone
   workflow engine reports an error during processing of a workflow associated
   with a resource)
   4. Heat should expose an API which enables basic communication between
   running workflows. Additionally, Heat should provide an API to workflows
   that allows workflows to specify whether they completed successfully or
   not. The reference to these APIs should be passed to the workflow engine
   that is responsible for executing workflows on VMs.

Pros of each point:
1 & 2 - keeps Heat simple and gives a possibility to choose the best
workflows and engines among available ones.
3 - adds some kind of all-or-nothing semantics improving the control and
awareness of what's going on inside VMs.
4 - allows workflow synchronization and communication through Heat API.
Provides the error reporting mechanism for workflows. If a workflow does
not need this functionality, it can ignore it.

Cons:
- Changes to existing workflows making them aware of Heat existence are
required.

These thoughts might show some gaps in my understanding of how Heat works,
but I would like to share them anyway.

Best regards,
Oleksii Rudenko


On Wed, Oct 9, 2013 at 5:37 PM, Georgy Okrokvertskhov <
gokrokvertsk...@mirantis.com> wrote:

> Hi,
>
> In addition I want to add couple words about flexibility and debugging
> capabilities. I believe it is quite important for HOT template engine to
> control all aspects of deployment process execution including software
> components. Right now I believe Heat lack of control of what is going on
> the VM side.  In my opinion, HOT template user should be able to define
> what steps are necessary to deploy complex environment and more important,
> he should be able to provide a hints to the engine how to deal with errors
> during deployment. Centralized orchestration sees the whole picture of the
> environment status while scripts on VM usually have quite limited view.
> Workflows specification can have on_error actions and centralized
> orchestration will be capable to make smart decision on how to handle
> errors during deployment.
>
> I think we need to have a design discussion about the architecture of this
> centralized orchestration. From the one side, this orchestration should
> have the whole information about environment state and as Heat has full
> exposure to the environment it sounds reasonable to have such orchestration
> as a part of Heat. At the other side, HOT template should be quite simple
> to be useful, so additional workflows concepts might overload DSL syntax
> and additional independent orchestration level also sounds quite reasonable
> and this is what we have now as a Murano project.
>
> It will be nice to have some initial live discussion before the summit as
> not all developers will be on summit. What do you think about Google
> hangout session at the end of this week or on the next week?
>
> Thanks
> Gosha
>
>
>
>
>
>
>
>
>
> On Wed, Oct 9, 2013 at 7:52 AM, Stan Lagun  wrote:
>
>> > Thanks, we're certainly interested in Murano, and are keen to discuss
>> your
>> > roadmap, and where requirements and integration opportunities exist
>> Glad to here it. The same is true from Murano side.
>>
>> On sample SQL workflow: that was just an example. I didn't want to bother
>> you with a SQL Server deployment details as it's really not that important.
>> What I've tried to say is that deployments consists of many steps, the
>> steps vary depending on instance's role, user input and so on. The step on
>> one machine
>> often requires some other machine already be in some state or some output
>> from a deployment step happened elsewhere.
>>
>> I do understand that it is doable using Heat alone. Actually we do use
>> Heat for some parts of workflow. We do not talk to Nova or Neutron directly.
>> The special use case of Murano is that there is no HOT template author.
>> Heat is more an administrator's tool who knows how to write HOT templates
>> and wants to deal with low-level configuration aspects. But Murano is quite
>> different. In Murano the developers of workflows/scripts/metadata/etc are
>> not end-users. The user is not doing any sort of programming. He is given a
>> UI dashboard where he can compose desired environment from available
>> building blocks (services). 

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Mike Spreitzer]

I favor separation of concerns.  I think (4), at least, has got nothing to 
do with infrastructure orchestration, the primary concern of today's heat 
engine.  I advocate (4), but as separate functionality.

Regards,
Mike

Alex Rudenko  wrote on 10/09/2013 12:59:22 PM:

> From: Alex Rudenko 
> To: OpenStack Development Mailing List 
, 
> Date: 10/09/2013 01:03 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration 
> proposal for workflows
> 
> Hi everyone,
> 
> I've read this thread and I'd like to share some thoughts. In my 
> opinion, workflows (which run on VMs) can be integrated with heat 
> templates as follows:
> 1. workflow definitions should be defined separately and processed 
> by stand-alone workflow engines (chef, puppet etc). 
> 2. the HOT resources should reference workflows which they require, 
> specifying a type of workflow and the way to access a workflow 
> definition. The workflow definition might be provided along with HOT.
> 3. Heat should treat the orchestration templates as transactions 
> (i.e. Heat should be able to rollback in two cases: 1) if something 
> goes wrong during processing of an orchestration workflow 2) when a 
> stand-alone workflow engine reports an error during processing of a 
> workflow associated with a resource)
> 4. Heat should expose an API which enables basic communication 
> between running workflows. Additionally, Heat should provide an API 
> to workflows that allows workflows to specify whether they completed
> successfully or not. The reference to these APIs should be passed to
> the workflow engine that is responsible for executing workflows on VMs.
> Pros of each point:
> 1 & 2 - keeps Heat simple and gives a possibility to choose the best
> workflows and engines among available ones.
> 3 - adds some kind of all-or-nothing semantics improving the control
> and awareness of what's going on inside VMs.
> 4 - allows workflow synchronization and communication through Heat 
> API. Provides the error reporting mechanism for workflows. If a 
> workflow does not need this functionality, it can ignore it.
> 
> Cons:
> - Changes to existing workflows making them aware of Heat existence 
> are required.
> 
> These thoughts might show some gaps in my understanding of how Heat 
> works, but I would like to share them anyway.
> 
> Best regards,
> Oleksii Rudenko
> 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Clint Byrum]

Excerpts from Georgy Okrokvertskhov's message of 2013-10-09 08:37:36 -0700:
> Hi,
> 
> In addition I want to add couple words about flexibility and debugging
> capabilities. I believe it is quite important for HOT template engine to
> control all aspects of deployment process execution including software
> components. Right now I believe Heat lack of control of what is going on
> the VM side.  In my opinion, HOT template user should be able to define
> what steps are necessary to deploy complex environment and more important,
> he should be able to provide a hints to the engine how to deal with errors
> during deployment. Centralized orchestration sees the whole picture of the
> environment status while scripts on VM usually have quite limited view.
> Workflows specification can have on_error actions and centralized
> orchestration will be capable to make smart decision on how to handle
> errors during deployment.
> 

What you have described above is some of what I'd like to see in HOT.
It is an evolution beyond the limitations of the waitcondition that
keeps things simple. Basically, orchestration providing a hook point
at which a portion of the workflow is deferred to some other tool. The
tool signals back when it is done, or has an error. We have that now,
but now, the errors just halt the process. We definitely need a way to
say something like this:

on_error_code: {default: {rebuild_resources: [ Instance1, Loadbalancer1]}}

The OS::Heat::HARestarter was sort of an attempt at some of this.

I take issue with a tool that wants to control everything. That may be
the easy way out, however I believe that it will lead to a very large,
very complex tool that users will be suspicious of.

For me, I'd like to know where my orchestration ends, and my software
configuration, installation, and state management all begin. The interface
can be such that Heat doesn't have to be omniscient and omnipotent.
It just has to help simplify the task of orchestration and get out of
workflow/config/installation/etc.'s way.

> I think we need to have a design discussion about the architecture of this
> centralized orchestration. From the one side, this orchestration should
> have the whole information about environment state and as Heat has full
> exposure to the environment it sounds reasonable to have such orchestration
> as a part of Heat. At the other side, HOT template should be quite simple
> to be useful, so additional workflows concepts might overload DSL syntax
> and additional independent orchestration level also sounds quite reasonable
> and this is what we have now as a Murano project.
>

It sounds like you have learned a lot on this journey and it would
definitely be valuable to collaborate with you so that we can make sure
Heat accommodates the use cases you have uncovered.

> It will be nice to have some initial live discussion before the summit as
> not all developers will be on summit. What do you think about Google
> hangout session at the end of this week or on the next week?
> 

I find that summit sessions are most useful when we are at a point where
there are just a few decision points to get through. If we come with too
much already done, then group-think will take over and less-well-formed
ideas will get squelched. If we come without clearly defined decisions
to make, then we'll bike-shed for the full 40 minutes.

So, given that, I think a brief pre-summit discussion is a good idea to
help us figure out where we may have conflicting views and then we can
come ready to hash those out in a high bandwidth environment.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Steven Hardy  wrote on 10/09/2013 05:24:38 AM:

>
> So as has already been mentioned, Heat defines an internal workflow,
based
> on the declarative model defined in the template.
>
> The model should define dependencies, and Heat should convert those
> dependencies into a workflow internally.  IMO if the user also needs to
> describe a workflow explicitly in the template, then we've probably
failed
> to provide the right template interfaces for describing depenendencies.

I agree with Steven here, models should define the dependencies and Heat
should realize/enforce them. An important design issue is granularity at
which dependencies are defined and enforced. I am aware of the
wait-condition
and signal constructs in Heat, but I find them a bit low-level as they are
prone
to the classic dead-lock and race condition problems.  I would like to have
higher level constructs that support finer-granularity dependences which
are needed for software orchestration. Reading through the various
disucssion
on this topic in this mailing list, I see that many would like to have such
higher level constructs for coordination.

In our experience with software orchestration using our own DSL and also
with
some extensions to Heat, we found that the granularity of VMs or Resources
to be
too coarse for defining dependencies for software orchestration. For
example, consider
a two VM app, with VMs vmA, vmB, and a set of software components (ai's and
bi's)
to be installed on them:

vmA = base-vmA + a1 + a2 + a3
vmB = base-vmB + b1 + b2 + b3

let us say that software component b1 of vmB, requires a config value
produced by
software component a1 of vmA. How to declaratively model this dependence?
Clearly,
modeling a dependence between just base-vmA and base-vmB is not enough.
However,
defining a dependence between the whole of vmA and vmB is too coarse. It
would be ideal
to be able to define a dependence at the granularity of software
components, i.e.,
vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what
value
is passed between vmB.b1 and vmA.a1, so that the communication can be
facilitated
by the orchestration engine.

We found that such finer granular modeling of the dependencies provides two
valuable benefits:

1. Faster total (resources + software setup) deployment time. For the
example described
above, a coarse-granularity dependence enforcer would start the deployment
of base-vmB after
vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer
would start base-vmA
and base-vmB concurrently, and then suspend the execution of vmB.b1 until
vmA.a1 is complete and then
let the rest of deployment proceed concurrently, resulting in a faster
completion.

2. More flexible dependencies. For example, mutual dependencies between
resources,
which can be satisfied when orchestrated at a finer granularity. Using the
example described
above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also
vmA.a3 depends_on vmB.b2,
but coarse-granularity model would flag this as a cyclic dependence.

There are two aspects that needs support:

1. Heat/HOT template level constructs to support declarative expression of
such fine-granularity
dependencies and the values communicated / passed for the dependence.
2. Support from Heat engine / analyzer in supporting the runtime ordering,
coordination between
resources, and also the communication of the values.

What are your thoughts?___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Steven Hardy]

On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote:
> Hi everyone,
> 
> I've read this thread and I'd like to share some thoughts. In my opinion,
> workflows (which run on VMs) can be integrated with heat templates as
> follows:
> 
>1. workflow definitions should be defined separately and processed by
>stand-alone workflow engines (chef, puppet etc).

I agree, and I think this is the direction we're headed with the
software-config blueprints - essentially we should end up with some new
Heat *resources* which encapsulate software configuration.

IMO there is some confusion around the scope of HOT, we should not be
adding functionality to it which already exists in established config
management tools IMO, instead we should focus on better integration with
exisitng tools at the resource level, and identifying template interfaces
which require more flexibility (for example serialization primitives)

>2. the HOT resources should reference workflows which they require,
>specifying a type of workflow and the way to access a workflow definition.
>The workflow definition might be provided along with HOT.

So again, I think this acatually has very little to do with HOT.  The
*Heat* resources may define software configuration, or possibly some sort
of workflow, which is acted upon by $thing which is not Heat.

So in the example provided by the OP, maybe you'd have a Murano resource,
which knows how to define the input to the Murano API, which might trigger
workflow type actions to happen in the Murano service.

>3. Heat should treat the orchestration templates as transactions (i.e.
>Heat should be able to rollback in two cases: 1) if something goes wrong
>during processing of an orchestration workflow 2) when a stand-alone
>workflow engine reports an error during processing of a workflow associated
>with a resource)

So we already have the capability for resources to recieve signals, which
would allow (2) in the asynchronous case.  But it seems to me that this is
still a serialization problem, ie a synchronous case, therefore (2) is just
part of (1).

E.g

- Heat stack create starts
- Murano resource created (CREATE IN_PROGRESS state)
- Murano workdlow stuff happens, signals Heat with success/failure
- Murano resource transitions to either COMPLETE or FAILED state
- If a FAILED state happened, e.g on update, we can roll back to the
  previous stack definition (this is already possible in Heat)

>4. Heat should expose an API which enables basic communication between
>running workflows. Additionally, Heat should provide an API to workflows
>that allows workflows to specify whether they completed successfully or
>not. The reference to these APIs should be passed to the workflow engine
>that is responsible for executing workflows on VMs.

I personally don't think this is in scope for Heat.  We already have an API
which exposes the status of stacks and resources.  Exposing some different
API which describes a workflow implemented by a specific subset of resource
types makes no sense to me.

> 
> Pros of each point:
> 1 & 2 - keeps Heat simple and gives a possibility to choose the best
> workflows and engines among available ones.
> 3 - adds some kind of all-or-nothing semantics improving the control and
> awareness of what's going on inside VMs.
> 4 - allows workflow synchronization and communication through Heat API.
> Provides the error reporting mechanism for workflows. If a workflow does
> not need this functionality, it can ignore it.

IMHO (4) is very much a step too far, and is not well aligned with the
current interfaces provided by Heat.

I'm really keen to further discuss the use-cases here, but if possible, it
would be helpful if folks can describe their requirements in less abstract
terms, and with reference to our existing interfaces and template model.

> These thoughts might show some gaps in my understanding of how Heat works,
> but I would like to share them anyway.

You've raised some good points, thanks.

I'm really keen to further discuss the use-cases here, but if possible, it
would be helpful if folks can describe their requirements in less abstract
terms, and with reference to our existing interfaces and template model.
That way we can start defining what is actually missing to support specific
use-cases.

So far I see the following emerging as definite requirements:
- Better/more flexible native serialization interfaces (possbly HOT
  additions)
- Better/more flexible *resources* which encapsulate software configuration
  on instances, probably with the flexibility of applying more than one
  config to an instance (not necessarily related to any HOT changes at all)

Thanks,

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Georgy Okrokvertskhov]

Hi Lakshminaraya,

Thank you for bringing your use case and your thought here. That is exactly
tried to achieve in Murano project.
There are important  aspects you highlighted. Sometime resource model is
two high level to describe deployment process. If you start to use more
granular approach to have defined steps of deployment you will finish with
workflow approach where you have fine control of deployment process but
description will be quite complex.

I think the HOT approach is to provide a simple way do describe you
deployment which consists of solid bricks (resources). If you are using
standard resources you can easily create a simple HOT template for your
deployment. If you need some custom resource you basically have two options
- create new resource class and hide all complexity inside the code or use
some workflows language to describe all steps required. The first approach
is currently supported by Heat. We have an experience of creating new
custom resources for orchestration deployment to specific IT infrastructure
with specific hardware and software.

Right now we are trying to figure out the possibility of adding workflows
to HOT. It looks like adding workflows language directly might harm HOT
simplicity by overloaded DSL syntax and structures.

I actually see the value in Steve's idea to have specific resource or
resource set to call workflows execution on external engine. In this case
HOT template will be still pretty simple as all workflow details will be
hidden, but still manageable without code writing.

Thanks
Gosha


On Wed, Oct 9, 2013 at 11:31 AM, Lakshminaraya Renganarayana <
lren...@us.ibm.com> wrote:

> Steven Hardy  wrote on 10/09/2013 05:24:38 AM:
>
>
> >
> > So as has already been mentioned, Heat defines an internal workflow,
> based
> > on the declarative model defined in the template.
> >
> > The model should define dependencies, and Heat should convert those
> > dependencies into a workflow internally.  IMO if the user also needs to
> > describe a workflow explicitly in the template, then we've probably
> failed
> > to provide the right template interfaces for describing depenendencies.
>
> I agree with Steven here, models should define the dependencies and Heat
> should realize/enforce them. An important design issue is granularity at
> which dependencies are defined and enforced. I am aware of the
> wait-condition
> and signal constructs in Heat, but I find them a bit low-level as they are
> prone
> to the classic dead-lock and race condition problems.  I would like to
> have
> higher level constructs that support finer-granularity dependences which
> are needed for software orchestration. Reading through the various
> disucssion
> on this topic in this mailing list, I see that many would like to have such
> higher level constructs for coordination.
>
> In our experience with software orchestration using our own DSL and also
> with
> some extensions to Heat, we found that the granularity of VMs or Resources
> to be
> too coarse for defining dependencies for software orchestration. For
> example, consider
> a two VM app, with VMs vmA, vmB, and a set of software components (ai's
> and bi's)
> to be installed on them:
>
> vmA = base-vmA + a1 + a2 + a3
> vmB = base-vmB + b1 + b2 + b3
>
> let us say that software component b1 of vmB, requires a config value
> produced by
> software component a1 of vmA. How to declaratively model this dependence?
> Clearly,
> modeling a dependence between just base-vmA and base-vmB is not enough.
> However,
> defining a dependence between the whole of vmA and vmB is too coarse. It
> would be ideal
> to be able to define a dependence at the granularity of software
> components, i.e.,
> vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what
> value
> is passed between vmB.b1 and vmA.a1, so that the communication can be
> facilitated
> by the orchestration engine.
>
> We found that such finer granular modeling of the dependencies provides
> two valuable benefits:
>
> 1. Faster total (resources + software setup) deployment time. For the
> example described
> above, a coarse-granularity dependence enforcer would start the deployment
> of base-vmB after
> vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence
> enforcer would start base-vmA
> and base-vmB concurrently, and then suspend the execution of vmB.b1 until
> vmA.a1 is complete and then
> let the rest of deployment proceed concurrently, resulting in a faster
> completion.
>
> 2. More flexible dependencies. For example, mutual dependencies between
> resources,
> which can be satisfied when orchestrated at a finer granularity. Using the
> example described
> above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also
> vmA.a3 depends_on vmB.b2,
> but coarse-granularity model would flag this as a cyclic dependence.
>
> There are two aspects that needs support:
>
> 1. Heat/HOT template level constructs to support declarative expression of
> such fine-granu

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Joshua Harlow]

Your example sounds a lot like what taskflow is build for doing.

https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py
 is a decent example.

In that one, tasks are created and input/output dependencies are specified 
(provides, rebind, and the execute function arguments itself).

This is combined into the taskflow concept of a flow, one of those flows types 
is a dependency graph.

Using a parallel engine (similar in concept to a heat engine) we can run all 
non-dependent tasks in parallel.

An example that I just created that shows this (and shows it running) that 
closer matches your example.

Program (this will work against the current taskflow codebase): 
http://paste.openstack.org/show/48156/

Output @ http://paste.openstack.org/show/48157/

-Josh

From: Lakshminaraya Renganarayana 
mailto:lren...@us.ibm.com>>
Reply-To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Date: Wednesday, October 9, 2013 11:31 AM
To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for 
workflows


Steven Hardy mailto:sha...@redhat.com>> wrote on 10/09/2013 
05:24:38 AM:

>
> So as has already been mentioned, Heat defines an internal workflow, based
> on the declarative model defined in the template.
>
> The model should define dependencies, and Heat should convert those
> dependencies into a workflow internally.  IMO if the user also needs to
> describe a workflow explicitly in the template, then we've probably failed
> to provide the right template interfaces for describing depenendencies.

I agree with Steven here, models should define the dependencies and Heat
should realize/enforce them. An important design issue is granularity at
which dependencies are defined and enforced. I am aware of the wait-condition
and signal constructs in Heat, but I find them a bit low-level as they are prone
to the classic dead-lock and race condition problems.  I would like to have
higher level constructs that support finer-granularity dependences which
are needed for software orchestration. Reading through the various disucssion
on this topic in this mailing list, I see that many would like to have such
higher level constructs for coordination.

In our experience with software orchestration using our own DSL and also with
some extensions to Heat, we found that the granularity of VMs or Resources to be
too coarse for defining dependencies for software orchestration. For example, 
consider
a two VM app, with VMs vmA, vmB, and a set of software components (ai's and 
bi's)
to be installed on them:

vmA = base-vmA + a1 + a2 + a3
vmB = base-vmB + b1 + b2 + b3

let us say that software component b1 of vmB, requires a config value produced 
by
software component a1 of vmA. How to declaratively model this dependence? 
Clearly,
modeling a dependence between just base-vmA and base-vmB is not enough. However,
defining a dependence between the whole of vmA and vmB is too coarse. It would 
be ideal
to be able to define a dependence at the granularity of software components, 
i.e.,
vmB.b1 depends on vmA.a1. Of course, it would also be good to capture what value
is passed between vmB.b1 and vmA.a1, so that the communication can be 
facilitated
by the orchestration engine.

We found that such finer granular modeling of the dependencies provides two 
valuable benefits:

1. Faster total (resources + software setup) deployment time. For the example 
described
above, a coarse-granularity dependence enforcer would start the deployment of 
base-vmB after
vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence enforcer 
would start base-vmA
and base-vmB concurrently, and then suspend the execution of vmB.b1 until 
vmA.a1 is complete and then
let the rest of deployment proceed concurrently, resulting in a faster 
completion.

2. More flexible dependencies. For example, mutual dependencies between 
resources,
which can be satisfied when orchestrated at a finer granularity. Using the 
example described
above, fine-granularity would allow vmB.b1 depends_on vmA.a1 and also vmA.a3 
depends_on vmB.b2,
but coarse-granularity model would flag this as a cyclic dependence.

There are two aspects that needs support:

1. Heat/HOT template level constructs to support declarative expression of such 
fine-granularity
dependencies and the values communicated / passed for the dependence.
2. Support from Heat engine / analyzer in supporting the runtime ordering, 
coordination between
resources, and also the communication of the values.

What are your thoughts?
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Stan Lagun]

It seems to me that something is missing in our discussion.

If something depends on something else there must be a definition of that
something. It is clear that it is not the case that one instance depends on
another but one application depends on another application. But there is no
such thing as application (service, whatever) in HOT templates. Only
low-level resources. And even resources cannot be grouped to some
application scope because typical HOT template has resources that are
shared between several applications (network, security groups etc.). It
also possible to have several applications sharing a single VM instance.
That brings us to a conclusion that applications and resources cannot be
mixed in the same template on the same level of abstraction.

Now suppose we did somehow established the dependency between two
applications. But this dependency is out of scope of particular HOT
template. Thats because HOT template says what user whishes to install. But
a dependency between applications is an attribute of applications
themselves, not the particular deployment. For example WordPress requires
database. It always does. Not that it requires it within this particular
template but a a universal rule. In Murano we call it data vs. metadata
separation. If there is a metadata that says "WordPress requires DB" then
you not just only don't have to repeat it in each template but you cannot
even ask a system to deploy WordPress without DB.

So the question is maybe we need to think about applications/services and
their metadata before going into workflow orchestration? Otherwise the
whole orchestration would be reinvented time and time again with each new
HOT template.

What are your thoughts on this?


On Wed, Oct 9, 2013 at 11:37 PM, Georgy Okrokvertskhov <
gokrokvertsk...@mirantis.com> wrote:

> Hi Lakshminaraya,
>
> Thank you for bringing your use case and your thought here. That is
> exactly tried to achieve in Murano project.
> There are important  aspects you highlighted. Sometime resource model is
> two high level to describe deployment process. If you start to use more
> granular approach to have defined steps of deployment you will finish with
> workflow approach where you have fine control of deployment process but
> description will be quite complex.
>
> I think the HOT approach is to provide a simple way do describe you
> deployment which consists of solid bricks (resources). If you are using
> standard resources you can easily create a simple HOT template for your
> deployment. If you need some custom resource you basically have two options
> - create new resource class and hide all complexity inside the code or use
> some workflows language to describe all steps required. The first approach
> is currently supported by Heat. We have an experience of creating new
> custom resources for orchestration deployment to specific IT infrastructure
> with specific hardware and software.
>
> Right now we are trying to figure out the possibility of adding workflows
> to HOT. It looks like adding workflows language directly might harm HOT
> simplicity by overloaded DSL syntax and structures.
>
> I actually see the value in Steve's idea to have specific resource or
> resource set to call workflows execution on external engine. In this case
> HOT template will be still pretty simple as all workflow details will be
> hidden, but still manageable without code writing.
>
> Thanks
> Gosha
>
>
> On Wed, Oct 9, 2013 at 11:31 AM, Lakshminaraya Renganarayana <
> lren...@us.ibm.com> wrote:
>
>> Steven Hardy  wrote on 10/09/2013 05:24:38 AM:
>>
>>
>> >
>> > So as has already been mentioned, Heat defines an internal workflow,
>> based
>> > on the declarative model defined in the template.
>> >
>> > The model should define dependencies, and Heat should convert those
>> > dependencies into a workflow internally.  IMO if the user also needs to
>> > describe a workflow explicitly in the template, then we've probably
>> failed
>> > to provide the right template interfaces for describing depenendencies.
>>
>> I agree with Steven here, models should define the dependencies and Heat
>> should realize/enforce them. An important design issue is granularity at
>> which dependencies are defined and enforced. I am aware of the
>> wait-condition
>> and signal constructs in Heat, but I find them a bit low-level as they
>> are prone
>> to the classic dead-lock and race condition problems.  I would like to
>> have
>> higher level constructs that support finer-granularity dependences which
>> are needed for software orchestration. Reading through the various
>> disucssion
>> on this topic in this mailing list, I see that many would like to have
>> such
>> higher level constructs for coordination.
>>
>> In our experience with software orchestration using our own DSL and also
>> with
>> some extensions to Heat, we found that the granularity of VMs or
>> Resources to be
>> too coarse for defining dependencies for software orchestration. 

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Caitlin Bestler]

On 10/9/2013 12:55 PM, Joshua Harlow wrote:

Your example sounds a lot like what taskflow is build for doing.



I'm not that familiar with Heat, so I wanted to bounce this off of
you before doing a public foot-in-mouth on the mailing list.

Is the real issue here the difference between *building* a set of
servers (Heat) versus performing a specific *task* on a set of servers
that was already built (taskflow)?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Hi Joshua,

I agree that there is an element of taskflow in what I described. But, I am
aiming for something much more lightweight which can be naturally blended
with HOT constructs and Heat engine. To be a bit more specific, Heat
already has dependencies and coordination mechanisms. So, I am aiming for
may be just one additional construct in Heat/HOT and some logic in Heat
that would support coordination.

Thanks,
LN

_
Lakshminarayanan Renganarayana
Research Staff Member
IBM T.J. Watson Research Center
http://researcher.ibm.com/person/us-lrengan


Joshua Harlow  wrote on 10/09/2013 03:55:00 PM:

> From: Joshua Harlow 
> To: OpenStack Development Mailing List  d...@lists.openstack.org>, Lakshminaraya Renganarayana/Watson/IBM@IBMUS
> Date: 10/09/2013 03:55 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Your example sounds a lot like what taskflow is build for doing.
>
> https://github.com/stackforge/taskflow/blob/master/taskflow/
> examples/calculate_in_parallel.py is a decent example.
>
> In that one, tasks are created and input/output dependencies are
> specified (provides, rebind, and the execute function arguments itself).
>
> This is combined into the taskflow concept of a flow, one of those
> flows types is a dependency graph.
>
> Using a parallel engine (similar in concept to a heat engine) we can
> run all non-dependent tasks in parallel.
>
> An example that I just created that shows this (and shows it
> running) that closer matches your example.
>
> Program (this will work against the current taskflow codebase):
> http://paste.openstack.org/show/48156/
>
> Output @ http://paste.openstack.org/show/48157/
>
> -Josh
>
> From: Lakshminaraya Renganarayana 
> Reply-To: OpenStack Development Mailing List  d...@lists.openstack.org>
> Date: Wednesday, October 9, 2013 11:31 AM
> To: OpenStack Development Mailing List

> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Steven Hardy  wrote on 10/09/2013 05:24:38 AM:
>
> >
> > So as has already been mentioned, Heat defines an internal workflow,
based
> > on the declarative model defined in the template.
> >
> > The model should define dependencies, and Heat should convert those
> > dependencies into a workflow internally.  IMO if the user also needs to
> > describe a workflow explicitly in the template, then we've probably
failed
> > to provide the right template interfaces for describing depenendencies.
>
> I agree with Steven here, models should define the dependencies and Heat
> should realize/enforce them. An important design issue is granularity at
> which dependencies are defined and enforced. I am aware of the
wait-condition
> and signal constructs in Heat, but I find them a bit low-level as
> they are prone
> to the classic dead-lock and race condition problems.  I would like to
have
> higher level constructs that support finer-granularity dependences which
> are needed for software orchestration. Reading through the various
disucssion
> on this topic in this mailing list, I see that many would like to have
such
> higher level constructs for coordination.
>
> In our experience with software orchestration using our own DSL and also
with
> some extensions to Heat, we found that the granularity of VMs or
> Resources to be
> too coarse for defining dependencies for software orchestration. For
> example, consider
> a two VM app, with VMs vmA, vmB, and a set of software components
> (ai's and bi's)
> to be installed on them:
>
> vmA = base-vmA + a1 + a2 + a3
> vmB = base-vmB + b1 + b2 + b3
>
> let us say that software component b1 of vmB, requires a config
> value produced by
> software component a1 of vmA. How to declaratively model this
> dependence? Clearly,
> modeling a dependence between just base-vmA and base-vmB is not
> enough. However,
> defining a dependence between the whole of vmA and vmB is too
> coarse. It would be ideal
> to be able to define a dependence at the granularity of software
> components, i.e.,
> vmB.b1 depends on vmA.a1. Of course, it would also be good to
> capture what value
> is passed between vmB.b1 and vmA.a1, so that the communication can
> be facilitated
> by the orchestration engine.
>
> We found that such finer granular modeling of the dependencies
> provides two valuable benefits:
>
> 1. Faster total (resources + software setup) deployment time. For
> the example described
> above, a coarse-granularity dependence enforcer would start the
> deployment of base-vmB after
> vmA + a1 + a2 + a3 is completed, but a fine-granularity dependence
> enforcer

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Georgy Okrokvertskhov  wrote on 10/09/2013
03:37:01 PM:

> From: Georgy Okrokvertskhov 
> To: OpenStack Development Mailing List

> Date: 10/09/2013 03:41 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>

> Thank you for bringing your use case and your thought here. That is
> exactly tried to achieve in Murano project.
> There are important  aspects you highlighted. Sometime resource
> model is two high level to describe deployment process. If you start
> to use more granular approach to have defined steps of deployment
> you will finish with workflow approach where you have fine control
> of deployment process but description will be quite complex.

IMHO workflow approaches tend to be heavy-weight. So, I am hoping
for more light-weight data-flow constructs and mechanisms that
can help with the coordination scenarios I have outlined. Data-flow
constructs and mechanisms have had lots of success in other domains,
and I wondering why can't we (the Heat community) leverage the related
theory and tools!

Thanks,
LN

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Joshua Harlow]

Thanks for the clarification.

I'm not sure how much of heat has to change to support what you are aiming for. 
Maybe heat should use taskflow ;)

From: Lakshminaraya Renganarayana 
mailto:lren...@us.ibm.com>>
Date: Wednesday, October 9, 2013 4:34 PM
To: Joshua Harlow mailto:harlo...@yahoo-inc.com>>
Cc: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Heat] HOT Software orchestration proposal for 
workflows


Hi Joshua,

I agree that there is an element of taskflow in what I described. But, I am 
aiming for something much more lightweight which can be naturally blended with 
HOT constructs and Heat engine. To be a bit more specific, Heat already has 
dependencies and coordination mechanisms. So, I am aiming for may be just one 
additional construct in Heat/HOT and some logic in Heat that would support 
coordination.

Thanks,
LN

_
Lakshminarayanan Renganarayana
Research Staff Member
IBM T.J. Watson Research Center
http://researcher.ibm.com/person/us-lrengan


Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote on 
10/09/2013 03:55:00 PM:

> From: Joshua Harlow mailto:harlo...@yahoo-inc.com>>
> To: OpenStack Development Mailing List  d...@lists.openstack.org<mailto:d...@lists.openstack.org>>, Lakshminaraya 
> Renganarayana/Watson/IBM@IBMUS
> Date: 10/09/2013 03:55 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Your example sounds a lot like what taskflow is build for doing.
>
> https://github.com/stackforge/taskflow/blob/master/taskflow/
> examples/calculate_in_parallel.py is a decent example.
>
> In that one, tasks are created and input/output dependencies are
> specified (provides, rebind, and the execute function arguments itself).
>
> This is combined into the taskflow concept of a flow, one of those
> flows types is a dependency graph.
>
> Using a parallel engine (similar in concept to a heat engine) we can
> run all non-dependent tasks in parallel.
>
> An example that I just created that shows this (and shows it
> running) that closer matches your example.
>
> Program (this will work against the current taskflow codebase):
> http://paste.openstack.org/show/48156/
>
> Output @ http://paste.openstack.org/show/48157/
>
> -Josh
>
> From: Lakshminaraya Renganarayana 
> mailto:lren...@us.ibm.com>>
> Reply-To: OpenStack Development Mailing List  d...@lists.openstack.org<mailto:d...@lists.openstack.org>>
> Date: Wednesday, October 9, 2013 11:31 AM
> To: OpenStack Development Mailing List 
> mailto:openstack-dev@lists.openstack.org>>
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Steven Hardy mailto:sha...@redhat.com>> wrote on 
> 10/09/2013 05:24:38 AM:
>
> >
> > So as has already been mentioned, Heat defines an internal workflow, based
> > on the declarative model defined in the template.
> >
> > The model should define dependencies, and Heat should convert those
> > dependencies into a workflow internally.  IMO if the user also needs to
> > describe a workflow explicitly in the template, then we've probably failed
> > to provide the right template interfaces for describing depenendencies.
>
> I agree with Steven here, models should define the dependencies and Heat
> should realize/enforce them. An important design issue is granularity at
> which dependencies are defined and enforced. I am aware of the wait-condition
> and signal constructs in Heat, but I find them a bit low-level as
> they are prone
> to the classic dead-lock and race condition problems.  I would like to have
> higher level constructs that support finer-granularity dependences which
> are needed for software orchestration. Reading through the various disucssion
> on this topic in this mailing list, I see that many would like to have such
> higher level constructs for coordination.
>
> In our experience with software orchestration using our own DSL and also with
> some extensions to Heat, we found that the granularity of VMs or
> Resources to be
> too coarse for defining dependencies for software orchestration. For
> example, consider
> a two VM app, with VMs vmA, vmB, and a set of software components
> (ai's and bi's)
> to be installed on them:
>
> vmA = base-vmA + a1 + a2 + a3
> vmB = base-vmB + b1 + b2 + b3
>
> let us say that software component b1 of vmB, requires a config
> value produced by
> software component a1 of vmA. How to declaratively model this
> dependence? Clearly,
> modeling a dependence between just base-vmA and base-vmB is not
> enough. However,
> defining a dep

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Stan Lagun  wrote on 10/09/2013 04:07:33 PM:

It seems to me that something is missing in our discussion.

> If something depends on something else there must be a definition of
> that something. It is clear that it is not the case that one
> instance depends on another but one application depends on another
> application. But there is no such thing as application (service,
> whatever) in HOT templates. Only low-level resources. And even
> resources cannot be grouped to some application scope because
> typical HOT template has resources that are shared between several
> applications (network, security groups etc.). It also possible to
> have several applications sharing a single VM instance. That brings
> us to a conclusion that applications and resources cannot be mixed
> in the same template on the same level of abstraction.

Good point on the levels of abstraction.

> Now suppose we did somehow established the dependency between two
> applications. But this dependency is out of scope of particular HOT
> template. Thats because HOT template says what user whishes to
> install. But a dependency between applications is an attribute of
> applications themselves, not the particular deployment. For example
> WordPress requires database. It always does. Not that it requires it
> within this particular template but a a universal rule. In Murano we
> call it data vs. metadata separation. If there is a metadata that
> says "WordPress requires DB" then you not just only don't have to
> repeat it in each template but you cannot even ask a system to
> deploy WordPress without DB.

I think the kind of dependency you have outlined above is more of
software component requirements of an application. These kind of
semantic dependencies are important and are probably outside the
scope of Heat.  The kind of dependencies I referred to are of the
nature of data-flow between software components: for example, a
tomcat application server needs (and hence, depends on)  the
DB's username/password to set up its configuration. How do we
model such a data-flow dependence and how to we facilitate the
communication of such values from the DB to the tomcat component?
IMHO, such questions are related to Heat.

> So the question is maybe we need to think about applications/
> services and their metadata before going into workflow
> orchestration? Otherwise the whole orchestration would be reinvented
> time and time again with each new HOT template.

> What are your thoughts on this?

I find your separation of metadata vs. data useful. In my opinion,
the kind of metadata you are trying to capture would be best
modeled by a DSL that sits on top of HOT/Heat.

Thanks,
LN


>

> On Wed, Oct 9, 2013 at 11:37 PM, Georgy Okrokvertskhov <
> gokrokvertsk...@mirantis.com> wrote:
> Hi Lakshminaraya,
>
> Thank you for bringing your use case and your thought here. That is
> exactly tried to achieve in Murano project.
> There are important  aspects you highlighted. Sometime resource
> model is two high level to describe deployment process. If you start
> to use more granular approach to have defined steps of deployment
> you will finish with workflow approach where you have fine control
> of deployment process but description will be quite complex.
>
> I think the HOT approach is to provide a simple way do describe you
> deployment which consists of solid bricks (resources). If you are
> using standard resources you can easily create a simple HOT template
> for your deployment. If you need some custom resource you basically
> have two options - create new resource class and hide all complexity
> inside the code or use some workflows language to describe all steps
> required. The first approach is currently supported by Heat. We have
> an experience of creating new custom resources for orchestration
> deployment to specific IT infrastructure with specific hardware and
software.
>
> Right now we are trying to figure out the possibility of adding
> workflows to HOT. It looks like adding workflows language directly
> might harm HOT simplicity by overloaded DSL syntax and structures.
>
> I actually see the value in Steve's idea to have specific resource
> or resource set to call workflows execution on external engine. In
> this case HOT template will be still pretty simple as all workflow
> details will be hidden, but still manageable without code writing.
>
> Thanks
> Gosha
>

> On Wed, Oct 9, 2013 at 11:31 AM, Lakshminaraya Renganarayana <
> lren...@us.ibm.com> wrote:
> Steven Hardy  wrote on 10/09/2013 05:24:38 AM:
>
>
> >
> > So as has already been mentioned, Heat defines an internal workflow,
based
> > on the declarative model defined in the template.
> >
> > The model should define dependencies, and Heat should convert those
> > dependencies into a workflow internally.  IMO if the user also needs to
> > describe a workflow explicitly in the template, then we've probably
failed
> > to provide the right template interfaces for describing depenendencie

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Angus Salkeld]

On 09/10/13 19:31 +0100, Steven Hardy wrote:

On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote:

Hi everyone,

I've read this thread and I'd like to share some thoughts. In my opinion,
workflows (which run on VMs) can be integrated with heat templates as
follows:

   1. workflow definitions should be defined separately and processed by
   stand-alone workflow engines (chef, puppet etc).


I agree, and I think this is the direction we're headed with the
software-config blueprints - essentially we should end up with some new
Heat *resources* which encapsulate software configuration.


Exactly.

I think we need a software-configuration-aas sub-project that knows
how to take puppet/chef/salt/... config and deploy it. Then Heat just
has Resources for these (OS::SoftwareConfig::Puppet). 

We should even move our WaitConditions and Metadata over to that 
yet-to-be-made service so that Heat is totally clean of software config.


How would this solve ordering issues:

resources:
 config1:
   type: OS::SoftwareConfig::Puppet
   hosted_on: server1
   ...
 config2:
   type: OS::SoftwareConfig::Puppet
   hosted_on: server1
   depends_on: config3
   ...
 config3:
   type: OS::SoftwareConfig::Puppet
   hosted_on: server2
   depends_on: config1
   ...
 server1:
   type: OS::Nova::Server
   ...
 server2:
   type: OS::Nova::Server
   ...


Heat knows all about ordering:
It starts the resources:
server1, server2
config1
config3
config2

There is the normal contract in the client:
we post the config to software-config-service
and we wait for the state == ACTIVE (when the config is applied)
before progressing to a resource that is dependant on it.

-Angus



IMO there is some confusion around the scope of HOT, we should not be
adding functionality to it which already exists in established config
management tools IMO, instead we should focus on better integration with
exisitng tools at the resource level, and identifying template interfaces
which require more flexibility (for example serialization primitives)


   2. the HOT resources should reference workflows which they require,
   specifying a type of workflow and the way to access a workflow definition.
   The workflow definition might be provided along with HOT.


So again, I think this acatually has very little to do with HOT.  The
*Heat* resources may define software configuration, or possibly some sort
of workflow, which is acted upon by $thing which is not Heat.

So in the example provided by the OP, maybe you'd have a Murano resource,
which knows how to define the input to the Murano API, which might trigger
workflow type actions to happen in the Murano service.


   3. Heat should treat the orchestration templates as transactions (i.e.
   Heat should be able to rollback in two cases: 1) if something goes wrong
   during processing of an orchestration workflow 2) when a stand-alone
   workflow engine reports an error during processing of a workflow associated
   with a resource)


So we already have the capability for resources to recieve signals, which
would allow (2) in the asynchronous case.  But it seems to me that this is
still a serialization problem, ie a synchronous case, therefore (2) is just
part of (1).

E.g

- Heat stack create starts
- Murano resource created (CREATE IN_PROGRESS state)
- Murano workdlow stuff happens, signals Heat with success/failure
- Murano resource transitions to either COMPLETE or FAILED state
- If a FAILED state happened, e.g on update, we can roll back to the
 previous stack definition (this is already possible in Heat)


   4. Heat should expose an API which enables basic communication between
   running workflows. Additionally, Heat should provide an API to workflows
   that allows workflows to specify whether they completed successfully or
   not. The reference to these APIs should be passed to the workflow engine
   that is responsible for executing workflows on VMs.


I personally don't think this is in scope for Heat.  We already have an API
which exposes the status of stacks and resources.  Exposing some different
API which describes a workflow implemented by a specific subset of resource
types makes no sense to me.



Pros of each point:
1 & 2 - keeps Heat simple and gives a possibility to choose the best
workflows and engines among available ones.
3 - adds some kind of all-or-nothing semantics improving the control and
awareness of what's going on inside VMs.
4 - allows workflow synchronization and communication through Heat API.
Provides the error reporting mechanism for workflows. If a workflow does
not need this functionality, it can ignore it.


IMHO (4) is very much a step too far, and is not well aligned with the
current interfaces provided by Heat.

I'm really keen to further discuss the use-cases here, but if possible, it
would be helpful if folks can describe their requirements in less abstract
terms, and with reference to our existing interfaces and template model.


These thoughts might show some ga

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Stan Lagun]

This rises number of questions:

1. What about conditional dependencies? Like config3 depends on config1 AND
config2 OR config3.

2. How do I pass values between configs? For example config1 requires value
from user input and config2 needs an output value obtained from applying
config1

3. How would you do error handling? For example config3 on server3 requires
config1 to be applied on server1 and config2 on server2. Suppose that there
was an error while applying config2 (and config1 succeeded). How do I
specify reaction for that? Maybe I need then to try to apply config4 to
server2 and continue or maybe just roll everything back

4. How these config dependencies play with nested stacks and resources like
LoadBalancer that create such stacks? How do I specify that myConfig
depends on HA proxy being configured if that config was declared in nested
stack that is generated by resource's Python code and is not declared in my
HOT template?

5. The solution is not generic. For example I want to write HOT template
for my custom load-balancer and a scalable web-servers group. Load balancer
config depends on all configs of web-servers. But web-servers are created
dynamically (autoscaling). That means dependency graph needs to be also
dynamically modified. But if you explicitly list config names instead of
something like "depends on all configs of web-farm X" you have no way to
describe such rule. In other words we need generic dependency, not just
dependency on particular config

6. What would you do on STACK UPDATE that modifies the dependency graph?

The notation of configs and there


On Thu, Oct 10, 2013 at 4:25 AM, Angus Salkeld  wrote:

> On 09/10/13 19:31 +0100, Steven Hardy wrote:
>
>> On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote:
>>
>>> Hi everyone,
>>>
>>> I've read this thread and I'd like to share some thoughts. In my opinion,
>>> workflows (which run on VMs) can be integrated with heat templates as
>>> follows:
>>>
>>>1. workflow definitions should be defined separately and processed by
>>>stand-alone workflow engines (chef, puppet etc).
>>>
>>
>> I agree, and I think this is the direction we're headed with the
>> software-config blueprints - essentially we should end up with some new
>> Heat *resources* which encapsulate software configuration.
>>
>
> Exactly.
>
> I think we need a software-configuration-aas sub-project that knows
> how to take puppet/chef/salt/... config and deploy it. Then Heat just
> has Resources for these (OS::SoftwareConfig::Puppet).
> We should even move our WaitConditions and Metadata over to that
> yet-to-be-made service so that Heat is totally clean of software config.
>
> How would this solve ordering issues:
>
> resources:
>  config1:
>type: OS::SoftwareConfig::Puppet
>hosted_on: server1
>...
>  config2:
>type: OS::SoftwareConfig::Puppet
>hosted_on: server1
>depends_on: config3
>...
>  config3:
>type: OS::SoftwareConfig::Puppet
>hosted_on: server2
>depends_on: config1
>...
>  server1:
>type: OS::Nova::Server
>...
>  server2:
>type: OS::Nova::Server
>...
>
>
> Heat knows all about ordering:
> It starts the resources:
> server1, server2
> config1
> config3
> config2
>
> There is the normal contract in the client:
> we post the config to software-config-service
> and we wait for the state == ACTIVE (when the config is applied)
> before progressing to a resource that is dependant on it.
>
> -Angus
>
>
>
>> IMO there is some confusion around the scope of HOT, we should not be
>> adding functionality to it which already exists in established config
>> management tools IMO, instead we should focus on better integration with
>> exisitng tools at the resource level, and identifying template interfaces
>> which require more flexibility (for example serialization primitives)
>>
>> 2. the HOT resources should reference workflows which they require,
>>>specifying a type of workflow and the way to access a workflow
>>> definition.
>>>The workflow definition might be provided along with HOT.
>>>
>>
>> So again, I think this acatually has very little to do with HOT.  The
>> *Heat* resources may define software configuration, or possibly some sort
>> of workflow, which is acted upon by $thing which is not Heat.
>>
>> So in the example provided by the OP, maybe you'd have a Murano resource,
>> which knows how to define the input to the Murano API, which might trigger
>> workflow type actions to happen in the Murano service.
>>
>> 3. Heat should treat the orchestration templates as transactions (i.e.
>>>Heat should be able to rollback in two cases: 1) if something goes
>>> wrong
>>>during processing of an orchestration workflow 2) when a stand-alone
>>>workflow engine reports an error during processing of a workflow
>>> associated
>>>with a resource)
>>>
>>
>> So we already have the capability for resources to recieve signals, which
>> would allow (2) in the asynchronous case. 

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Thomas Spatzier]

Hi all,

Lakshminaraya Renganarayana  wrote on 10.10.2013
01:34:41:
> From: Lakshminaraya Renganarayana 
> To: Joshua Harlow ,
> Cc: OpenStack Development Mailing List

> Date: 10.10.2013 01:37
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Hi Joshua,
>
> I agree that there is an element of taskflow in what I described.
> But, I am aiming for something much more lightweight which can be
> naturally blended with HOT constructs and Heat engine. To be a bit
> more specific, Heat already has dependencies and coordination
> mechanisms. So, I am aiming for may be just one additional construct
> in Heat/HOT and some logic in Heat that would support coordination.

First of all, the use case you presented in your earlier mail is really
good and illustrative. And I agree that there should be constructs in HOT
to declare those kinds of dependencies. So how to define this in HOT is one
work item.
How this gets implemented is another item, and yes, maybe this is something
that Heat can delegate to taskflow. Because if taskflow has those
capabilities, why re-implement it.

>
> Thanks,
> LN
>
> _
> Lakshminarayanan Renganarayana
> Research Staff Member
> IBM T.J. Watson Research Center
> http://researcher.ibm.com/person/us-lrengan
>
>
> Joshua Harlow  wrote on 10/09/2013 03:55:00 PM:
>
> > From: Joshua Harlow 
> > To: OpenStack Development Mailing List  > d...@lists.openstack.org>, Lakshminaraya Renganarayana/Watson/IBM@IBMUS
> > Date: 10/09/2013 03:55 PM
> > Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> > proposal for workflows
> >
> > Your example sounds a lot like what taskflow is build for doing.
> >
> > https://github.com/stackforge/taskflow/blob/master/taskflow/
> > examples/calculate_in_parallel.py is a decent example.
> >
> > In that one, tasks are created and input/output dependencies are
> > specified (provides, rebind, and the execute function arguments
itself).
> >
> > This is combined into the taskflow concept of a flow, one of those
> > flows types is a dependency graph.
> >
> > Using a parallel engine (similar in concept to a heat engine) we can
> > run all non-dependent tasks in parallel.
> >
> > An example that I just created that shows this (and shows it
> > running) that closer matches your example.
> >
> > Program (this will work against the current taskflow codebase):
> > http://paste.openstack.org/show/48156/
> >
> > Output @ http://paste.openstack.org/show/48157/
> >
> > -Josh
> >
> > From: Lakshminaraya Renganarayana 
> > Reply-To: OpenStack Development Mailing List  > d...@lists.openstack.org>
> > Date: Wednesday, October 9, 2013 11:31 AM
> > To: OpenStack Development Mailing List

> > Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> > proposal for workflows
> >
> > Steven Hardy  wrote on 10/09/2013 05:24:38 AM:
> >
> > >
> > > So as has already been mentioned, Heat defines an internal workflow,
based
> > > on the declarative model defined in the template.
> > >
> > > The model should define dependencies, and Heat should convert those
> > > dependencies into a workflow internally.  IMO if the user also needs
to
> > > describe a workflow explicitly in the template, then we've probably
failed
> > > to provide the right template interfaces for describing
depenendencies.
> >
> > I agree with Steven here, models should define the dependencies and
Heat
> > should realize/enforce them. An important design issue is granularity
at
> > which dependencies are defined and enforced. I am aware of the
> wait-condition
> > and signal constructs in Heat, but I find them a bit low-level as
> > they are prone
> > to the classic dead-lock and race condition problems.  I would like to
have
> > higher level constructs that support finer-granularity dependences
which
> > are needed for software orchestration. Reading through the
variousdisucssion
> > on this topic in this mailing list, I see that many would like to have
such
> > higher level constructs for coordination.
> >
> > In our experience with software orchestration using our own DSL
> and also with
> > some extensions to Heat, we found that the granularity of VMs or
> > Resources to be
> > too coarse for defining dependencies for software orchestration. For
> > example, consider
> > a two VM app, with VMs vmA, vmB, and a set of software components
> > (ai's and bi's)
> > to be instal

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Angus Salkeld]

On 10/10/13 11:59 +0400, Stan Lagun wrote:

This rises number of questions:

1. What about conditional dependencies? Like config3 depends on config1 AND
config2 OR config3.


We have the AND, but not an OR. To depend on two resources you just
have 2 references to the 2 resources.



2. How do I pass values between configs? For example config1 requires value
from user input and config2 needs an output value obtained from applying
config1


{Fn::GetAtt: [config2, the_name_of_the_attribute]}



3. How would you do error handling? For example config3 on server3 requires
config1 to be applied on server1 and config2 on server2. Suppose that there
was an error while applying config2 (and config1 succeeded). How do I
specify reaction for that? Maybe I need then to try to apply config4 to
server2 and continue or maybe just roll everything back


We currently have no "on_error" but it is not out of scope. The
current action is either to rollback the stack or leave it in the
failed state (depending on what you choose).



4. How these config dependencies play with nested stacks and resources like
LoadBalancer that create such stacks? How do I specify that myConfig
depends on HA proxy being configured if that config was declared in nested
stack that is generated by resource's Python code and is not declared in my
HOT template?


It is normally based on the actual data/variable that you are
dependant on.
loadbalancer: depends on autoscaling instance_list
(actually in the loadbalancer config would be a "GetAtt: [scalegroup, 
InstanceList]")

Then if you want to depend on that config you could depend on an
attribute of that resource that changes on reconfigure.

config1:
  type: OS::SoftwareConfig::Ssh
  properties:
script: {GetAtt: [scalegroup, InstanceList]}
hosted_on: loadbalancer
...

config2:
  type: OS::SoftwareConfig::Ssh
  properties:
script: {GetAtt: [config1, ConfigAppliedCount]}
hosted_on: somewhere_else
...

I am sure we could come up with some better syntax for this. But
the logic seems easily possible to me.

As far as nested stacks go: you just need an output to be useable
externally - basically design your API.




5. The solution is not generic. For example I want to write HOT template
for my custom load-balancer and a scalable web-servers group. Load balancer
config depends on all configs of web-servers. But web-servers are created
dynamically (autoscaling). That means dependency graph needs to be also
dynamically modified. But if you explicitly list config names instead of
something like "depends on all configs of web-farm X" you have no way to
describe such rule. In other words we need generic dependency, not just
dependency on particular config


Why won't just depending on the scaling group be enough? if it needs
to be updated it will update all within the group before progressing
to the dependants.



6. What would you do on STACK UPDATE that modifies the dependency graph?

The notation of configs and there


What we normally do go through the resources, see what can be updated:
- without replacement
- needs deleting
- is new
- requires updating

Each resource type can define what will require replacing or not.

I think we can achieve what you want with some small improvements to
the HOT format and with some new resource types - IMHO.

-Angus



On Thu, Oct 10, 2013 at 4:25 AM, Angus Salkeld  wrote:


On 09/10/13 19:31 +0100, Steven Hardy wrote:


On Wed, Oct 09, 2013 at 06:59:22PM +0200, Alex Rudenko wrote:


Hi everyone,

I've read this thread and I'd like to share some thoughts. In my opinion,
workflows (which run on VMs) can be integrated with heat templates as
follows:

   1. workflow definitions should be defined separately and processed by
   stand-alone workflow engines (chef, puppet etc).



I agree, and I think this is the direction we're headed with the
software-config blueprints - essentially we should end up with some new
Heat *resources* which encapsulate software configuration.



Exactly.

I think we need a software-configuration-aas sub-project that knows
how to take puppet/chef/salt/... config and deploy it. Then Heat just
has Resources for these (OS::SoftwareConfig::Puppet).
We should even move our WaitConditions and Metadata over to that
yet-to-be-made service so that Heat is totally clean of software config.

How would this solve ordering issues:

resources:
 config1:
   type: OS::SoftwareConfig::Puppet
   hosted_on: server1
   ...
 config2:
   type: OS::SoftwareConfig::Puppet
   hosted_on: server1
   depends_on: config3
   ...
 config3:
   type: OS::SoftwareConfig::Puppet
   hosted_on: server2
   depends_on: config1
   ...
 server1:
   type: OS::Nova::Server
   ...
 server2:
   type: OS::Nova::Server
   ...


Heat knows all about ordering:
It starts the resources:
server1, server2
config1
config3
config2

There is the normal contract in the client:
we post the config to software-config-service
and we wait for the state == ACTIVE (when the config is app

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Clint Byrum]

Excerpts from Angus Salkeld's message of 2013-10-10 15:27:48 -0700:
> On 10/10/13 11:59 +0400, Stan Lagun wrote:
> >This rises number of questions:
> >
> >1. What about conditional dependencies? Like config3 depends on config1 AND
> >config2 OR config3.
> 
> We have the AND, but not an OR. To depend on two resources you just
> have 2 references to the 2 resources.
> 

AND is concrete. OR is not. I don't actually think it is useful for what
Heat is intended to do. This is not not packaging, this is deploying.
For deploying, Heat needs to know _what to do_, not "what is possible".

> >
> >2. How do I pass values between configs? For example config1 requires value
> >from user input and config2 needs an output value obtained from applying
> >config1
> 
> {Fn::GetAtt: [config2, the_name_of_the_attribute]}
> 

This is a little bit misleading. Heat does not have any good ways to
get a "value obtained from applying config1". The "data" attribute of
the WaitCondition is the only way I know, and it is really unwieldy,
as it can basically only dump a json string of all of the things each
signaler has fed back in.

That said, I wonder how many of the "value obtained from applying config1"
would be satisfied by the recently proposed "random string generation"
resource. Most of the time what people want to communicate back is just
auth details. If we push auth details in from Heat to both sides, that
alleviates all of my current use cases for this type of feature.

> >
> >3. How would you do error handling? For example config3 on server3 requires
> >config1 to be applied on server1 and config2 on server2. Suppose that there
> >was an error while applying config2 (and config1 succeeded). How do I
> >specify reaction for that? Maybe I need then to try to apply config4 to
> >server2 and continue or maybe just roll everything back
> 
> We currently have no "on_error" but it is not out of scope. The
> current action is either to rollback the stack or leave it in the
> failed state (depending on what you choose).
> 

Right, I can definitely see more actions being added as we identify the
commonly desired options.

> >
> >4. How these config dependencies play with nested stacks and resources like
> >LoadBalancer that create such stacks? How do I specify that myConfig
> >depends on HA proxy being configured if that config was declared in nested
> >stack that is generated by resource's Python code and is not declared in my
> >HOT template?
> 
> It is normally based on the actual data/variable that you are
> dependant on.
> loadbalancer: depends on autoscaling instance_list
> (actually in the loadbalancer config would be a "GetAtt: [scalegroup, 
> InstanceList]")
> 
> Then if you want to depend on that config you could depend on an
> attribute of that resource that changes on reconfigure.
> 
> config1:
>type: OS::SoftwareConfig::Ssh
>properties:
>  script: {GetAtt: [scalegroup, InstanceList]}
>  hosted_on: loadbalancer
>  ...
> 
> config2:
>type: OS::SoftwareConfig::Ssh
>properties:
>  script: {GetAtt: [config1, ConfigAppliedCount]}
>  hosted_on: somewhere_else
>  ...
> 
> I am sure we could come up with some better syntax for this. But
> the logic seems easily possible to me.
> 
> As far as nested stacks go: you just need an output to be useable
> externally - basically design your API.
> 
> >
> >5. The solution is not generic. For example I want to write HOT template
> >for my custom load-balancer and a scalable web-servers group. Load balancer
> >config depends on all configs of web-servers. But web-servers are created
> >dynamically (autoscaling). That means dependency graph needs to be also
> >dynamically modified. But if you explicitly list config names instead of
> >something like "depends on all configs of web-farm X" you have no way to
> >describe such rule. In other words we need generic dependency, not just
> >dependency on particular config
> 
> Why won't just depending on the scaling group be enough? if it needs
> to be updated it will update all within the group before progressing
> to the dependants.
> 

In the example, loadbalancer doesn't have to depend on all of the nodes
being configured.  Why would it? It gets a signal when the list changes,
but it can be created as soon as the _group_ is created.

Anyway, no dependency is needed. Your LB has health checks, you feed
them in, and when the webservers are configured, they pass, and it sends
traffic there.

> >
> >6. What would you do on STACK UPDATE that modifies the dependency graph?
> >
> >The notation of configs and there
> 
> What we normally do go through the resources, see what can be updated:
> - without replacement
> - needs deleting
> - is new
> - requires updating
> 
> Each resource type can define what will require replacing or not.
> 
> I think we can achieve what you want with some small improvements to
> the HOT format and with some new resource types - IMHO.

Agree with Angus here. I think we're closer to yo

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Stan Lagun]

Hello,

Thanks Angus, Clint, I've got your design.

It seems that Murano can built on top of that. With service metadata
knowledge Murano can generate HOT templates with set of interdependent
configs.
Here is what will be needed:

1. Ability to implement support for custom software configuration tool
(type: OS::SoftwareConfig::MuranoAgent)
2. Ability to provide arbitrary input values for the config
3. Ability to return arbitrary (JSON-compatible) data structure from config
application and use attributes of that structure as an input for other
configs
4. Ability to provide config body that is an input to Murano Agent of
arbitrary size
5. Work well with large graph of configs with a lot of dependencies.
Independent configs on different VMs should be applied in parallel.

Does it confirm to your plans?




On Fri, Oct 11, 2013 at 3:47 AM, Clint Byrum  wrote:

> Excerpts from Angus Salkeld's message of 2013-10-10 15:27:48 -0700:
> > On 10/10/13 11:59 +0400, Stan Lagun wrote:
> > >This rises number of questions:
> > >
> > >1. What about conditional dependencies? Like config3 depends on config1
> AND
> > >config2 OR config3.
> >
> > We have the AND, but not an OR. To depend on two resources you just
> > have 2 references to the 2 resources.
> >
>
> AND is concrete. OR is not. I don't actually think it is useful for what
> Heat is intended to do. This is not not packaging, this is deploying.
> For deploying, Heat needs to know _what to do_, not "what is possible".
>
> > >
> > >2. How do I pass values between configs? For example config1 requires
> value
> > >from user input and config2 needs an output value obtained from applying
> > >config1
> >
> > {Fn::GetAtt: [config2, the_name_of_the_attribute]}
> >
>
> This is a little bit misleading. Heat does not have any good ways to
> get a "value obtained from applying config1". The "data" attribute of
> the WaitCondition is the only way I know, and it is really unwieldy,
> as it can basically only dump a json string of all of the things each
> signaler has fed back in.
>
> That said, I wonder how many of the "value obtained from applying config1"
> would be satisfied by the recently proposed "random string generation"
> resource. Most of the time what people want to communicate back is just
> auth details. If we push auth details in from Heat to both sides, that
> alleviates all of my current use cases for this type of feature.
>
> > >
> > >3. How would you do error handling? For example config3 on server3
> requires
> > >config1 to be applied on server1 and config2 on server2. Suppose that
> there
> > >was an error while applying config2 (and config1 succeeded). How do I
> > >specify reaction for that? Maybe I need then to try to apply config4 to
> > >server2 and continue or maybe just roll everything back
> >
> > We currently have no "on_error" but it is not out of scope. The
> > current action is either to rollback the stack or leave it in the
> > failed state (depending on what you choose).
> >
>
> Right, I can definitely see more actions being added as we identify the
> commonly desired options.
>
> > >
> > >4. How these config dependencies play with nested stacks and resources
> like
> > >LoadBalancer that create such stacks? How do I specify that myConfig
> > >depends on HA proxy being configured if that config was declared in
> nested
> > >stack that is generated by resource's Python code and is not declared
> in my
> > >HOT template?
> >
> > It is normally based on the actual data/variable that you are
> > dependant on.
> > loadbalancer: depends on autoscaling instance_list
> > (actually in the loadbalancer config would be a "GetAtt: [scalegroup,
> InstanceList]")
> >
> > Then if you want to depend on that config you could depend on an
> > attribute of that resource that changes on reconfigure.
> >
> > config1:
> >type: OS::SoftwareConfig::Ssh
> >properties:
> >  script: {GetAtt: [scalegroup, InstanceList]}
> >  hosted_on: loadbalancer
> >  ...
> >
> > config2:
> >type: OS::SoftwareConfig::Ssh
> >properties:
> >  script: {GetAtt: [config1, ConfigAppliedCount]}
> >  hosted_on: somewhere_else
> >  ...
> >
> > I am sure we could come up with some better syntax for this. But
> > the logic seems easily possible to me.
> >
> > As far as nested stacks go: you just need an output to be useable
> > externally - basically design your API.
> >
> > >
> > >5. The solution is not generic. For example I want to write HOT template
> > >for my custom load-balancer and a scalable web-servers group. Load
> balancer
> > >config depends on all configs of web-servers. But web-servers are
> created
> > >dynamically (autoscaling). That means dependency graph needs to be also
> > >dynamically modified. But if you explicitly list config names instead of
> > >something like "depends on all configs of web-farm X" you have no way to
> > >describe such rule. In other words we need generic dependency, not just
> > >dependency on particular config
> >

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Clint Byrum]

Excerpts from Stan Lagun's message of 2013-10-11 07:22:37 -0700:
> Hello,
> 
> Thanks Angus, Clint, I've got your design.
> 
> It seems that Murano can built on top of that. With service metadata
> knowledge Murano can generate HOT templates with set of interdependent
> configs.
> Here is what will be needed:
> 
> 1. Ability to implement support for custom software configuration tool
> (type: OS::SoftwareConfig::MuranoAgent)

These are really syntactic sugar, but you can implement them as resource
plugins for users who want Murano resources. In the absence of this,
just putting things in the free form metadata area of the resource that
the MuranoAgent can interpret would suffice.

> 2. Ability to provide arbitrary input values for the config

We already have that, there's a free-form json document called "metadata"
attached to every resource. Or maybe I missed what you mean here. The
new capability that is in the works that will make that better is to
have multiple reusable metadata blocks referenced on one instance.

> 3. Ability to return arbitrary (JSON-compatible) data structure from config
> application and use attributes of that structure as an input for other
> configs

Note that I'd like to see more use cases specified for this ability. The
random string generator that Steve Baker has put up should handle most
cases where you just need passwords. Generated key sharing might best
be deferred to something like Barbican which does a lot more than Heat
to try and keep your secrets safe.

> 4. Ability to provide config body that is an input to Murano Agent of
> arbitrary size

Isn't this the same as 2?

> 5. Work well with large graph of configs with a lot of dependencies.
> Independent configs on different VMs should be applied in parallel.
>

Yes this does look good. For dependent configs, the exsiting wait
condition can be used.

> Does it confirm to your plans?
> 

I think it confirms that we're heading toward consensus on where to draw
the software config vs. infrastructure orchestration line. That is very
exciting. :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Caitlin Bestler]

On 10/9/2013 12:55 PM, Joshua Harlow wrote:

Your example sounds a lot like what taskflow is build for doing.

https://github.com/stackforge/taskflow/blob/master/taskflow/examples/calculate_in_parallel.py
 is
a decent example.

In that one, tasks are created and input/output dependencies are
specified (provides, rebind, and the execute function arguments itself).

This is combined into the taskflow concept of a flow, one of those flows
types is a dependency graph.

Using a parallel engine (similar in concept to a heat engine) we can run
all non-dependent tasks in parallel.

An example that I just created that shows this (and shows it running)
that closer matches your example.

Program (this will work against the current taskflow
codebase): http://paste.openstack.org/show/48156/



I think that there is a major difference between building a set of 
virtual servers (what Heat does) and performing a specific task on

a set of servers (what taskflow is designed for).

Taskflow is currently planned to have a more complex and robust
state machine that what Heat plans. This is natural given that
simplicity has a relatively higher value for deployment, while
efficiency of execution matters more for a task that will be
performed repeatedly.

However, if a simpler state model is needed, keep in mind that
a simpler interface can always be translated into the more complete
interface with a shim layer. You cannot build a more flexible solution
on top of a simple solution.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Excellent discussion on various issues around orchestration and
coordination -- thanks to you all, in particular to Clint, Angus, Stan,
Thomas, Joshua, Zane, Steve ...

After reading the discussions, I am finding the following themes emerging
(please feel free to correct/add):

1. Most of the building blocks needed for effective coordination and
orchestration are already in Heat/HOT.
2. Heat would like to view software configuration as a resource (type) with
related providers + plugins
3. There is scope for communication/synchronization mechanisms that would
complement the wait-conditions and signals

I would like to propose a simple abstraction that would complement the
current wait-conditions and signals. My proposal is based our experience
with supporting such an abstraction on our DSL and also on an extension of
Heat.  In a nut-shell, this abstraction is a global data space (visible
across resources, stacks) from which resources can read and write their
inputs / outputs PLUS the semantics that reads will block until the read
values are available and writes are non-blocking. We used ZooKeeper to
implement this global data space and the blocking-read/non-blocking-writes
semantics. But, these could be implemented using several other mechanisms
and I believe the techniques currently used by Heat for meta-data service
can be used here.

I would like to make clear that I am not proposing a replacement for
wait-conditions and signals. I am hoping that wait-conditions and signals
would be used by power-users (concurrent/distributed programming experts)
and the proposed abstraction would be used by folks (like me) who do not
want to reason about concurrency and related problems. Also, the proposed
global data-space with blocking reads and non-blocking writes is not a new
idea (google tuple-spaces, linda) and it has been proven in other domains
such as coordination languages to improve the level of abstraction and
productivity.

The benefits of the proposed abstraction are:
G1. Support finer granularity of dependences
G2. Allow Heat to reason/analyze about these dependences so that it can
order resource creations/management
G3. Avoid classic synchronization problems such as dead-locks and race
conditions
G4 *Conjecture* : Capture most of the coordination use cases (including
those required for software configuration / orchestration).

Here is more detailed description: Let us say that we can use either
pre-defined or custom resource types to define resources at arbitrary
levels of granularity. This can be easily supported and I guess is already
possible in current version of Heat/HOT. Given this, the proposed
abstraction has two parts: (1) an interface style specification a
resource's inputs and outputs and (2) a global name/data space. The
interface specification which would capture

- INPUTS: all the attributes that are consumed/used/read by that resource
(currently, we have Ref, GetAttrs that can give this implicitly)

- OUTPUTS: all the attributes that are produced/written by that resource (I
do not know if this write-set is currently well-defined for a resource. I
think some of them are implicitly defined by Heat on particular resource
types.)

- Global name-space and data-space : all the values produced and consumed
(INPUTS/OUTPUTS) are described using a names that are fully qualified
(XXX.stack_name.resource_name.property_name). The data values associated
with these names are stored in a global data-space.  Reads are blocking,
i.e., reading a value will block the execution resource/thread until the
value is available. Writes are non-blocking, i.e., any thread can write a
value and the write will succeed immediately.

The ability to define resources at arbitrary levels of granularity together
with the explicit specification of INPUTS/OUTPUTS allows us to reap the
benefits G1 and G2 outlined above. Note that the ability to reason about
the inputs/outputs of each resource and the induced dependencies will also
allow Heat to detect dead-locks via dependence cycles (benefit G3). This is
already done today in Heat for Refs, GetAttr on base-resources, but the
proposal is to extend the same to arbitrary attributes for any resource.
The blocking-read and non-blocking writes further structures the
specification to avoid deadlocks and race conditions (benefit G3).

As for G4, the conjecture, I can only give as evidence our experience with
using our DSL with the proposed abstraction to deploy a few reasonably
large applications :-)

I would like to know your comments and suggestions. Also, if there is
interest I can write a Blueprint / proposal with more details and
use-cases.

Thanks,
LN



Clint Byrum  wrote on 10/11/2013 12:40:19 PM:

> From: Clint Byrum 
> To: openstack-dev 
> Date: 10/11/2013 12:43 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> Excerpts from Stan Lagun's message of 2013-10-11 07:22:37 -0700:
> > Hello,
> 

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Clint Byrum  wrote on 10/11/2013 12:40:19 PM:

> From: Clint Byrum 
> To: openstack-dev 
> Date: 10/11/2013 12:43 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> > 3. Ability to return arbitrary (JSON-compatible) data structure from
config
> > application and use attributes of that structure as an input for other
> > configs
>
> Note that I'd like to see more use cases specified for this ability. The
> random string generator that Steve Baker has put up should handle most
> cases where you just need passwords. Generated key sharing might best
> be deferred to something like Barbican which does a lot more than Heat
> to try and keep your secrets safe.

I had seen a deployment scenario that needed more than random string
generator. It was during the deployment of a system that has clustered
application servers, i.e., a cluster of application server nodes + a
cluster manager node. The deployment progresses by all the VMs
(cluster-manager and cluster-nodes) starting concurrently. Then the
cluster-nodes wait for the cluster-manager to send them data (xml) to
configure themselves. The cluster-manager after reading its own config
file, generates config-data for each cluster-node and sends it to them.

Thanks,
LN___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Steven Dake]

On 10/11/2013 11:55 AM, Lakshminaraya Renganarayana wrote:


Clint Byrum  wrote on 10/11/2013 12:40:19 PM:

> From: Clint Byrum 
> To: openstack-dev 
> Date: 10/11/2013 12:43 PM
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> > 3. Ability to return arbitrary (JSON-compatible) data structure 
from config

> > application and use attributes of that structure as an input for other
> > configs
>
> Note that I'd like to see more use cases specified for this ability. The
> random string generator that Steve Baker has put up should handle most
> cases where you just need passwords. Generated key sharing might best
> be deferred to something like Barbican which does a lot more than Heat
> to try and keep your secrets safe.

I had seen a deployment scenario that needed more than random string 
generator. It was during the deployment of a system that has clustered 
application servers, i.e., a cluster of application server nodes + a 
cluster manager node. The deployment progresses by all the VMs 
(cluster-manager and cluster-nodes) starting concurrently. Then the 
cluster-nodes wait for the cluster-manager to send them data (xml) to 
configure themselves. The cluster-manager after reading its own config 
file, generates config-data for each cluster-node and sends it to them.



Is the config data per cluster node unique to each node?  If not:

Change deployment to following model:
1. deploy cluster-manager as a resource with a waitcondition - passing 
the data using the cfn-signal  -d to send the xml blob
2. have cluster nodes wait on wait condition in #1, using data from the 
cfn-signal


If so, join the config data sent in cfn-signal and break it apart by the 
various cluster nodes in #2


Thanks,
LN


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Stan Lagun]

On Fri, Oct 11, 2013 at 8:40 PM, Clint Byrum  wrote:


> > 2. Ability to provide arbitrary input values for the config
> We already have that, there's a free-form json document called "metadata"
> attached to every resource. Or maybe I missed what you mean here. The
> new capability that is in the works that will make that better is to
> have multiple reusable metadata blocks referenced on one instance.
>
>
Yes, that's what I meant - metadata attached to configs rather than
instances.



>  > 3. Ability to return arbitrary (JSON-compatible) data structure from
> config
> > application and use attributes of that structure as an input for other
> > configs
>
> Note that I'd like to see more use cases specified for this ability. The
> random string generator that Steve Baker has put up should handle most
> cases where you just need passwords. Generated key sharing might best
> be deferred to something like Barbican which does a lot more than Heat
> to try and keep your secrets safe.
>
>
Murano's execution plans that are sent to Murano Agent are similar to
Python functions in that they have input and output. The output may be a
script exit code, captured stdout/stderr, value returned from
PowerShell/Python function etc. Although it is rare case where output from
one execution plan is required as an input for another plan but it happens.
For example execution plan 1 did created network interface on VM with DHCP
enabled and execution plan (that may be executed on another machine)
requires IP address obtained on that interface. In this case IP address
would be the returned value


> > 4. Ability to provide config body that is an input to Murano Agent of
> > arbitrary size
>
> Isn't this the same as 2?
>

Not exactly the same, but config body can be an attribute of config's
metadata with a special reserved name



> I think it confirms that we're heading toward consensus on where to draw
> the software config vs. infrastructure orchestration line. That is very
> exciting. :)
>

This is indeed very promising. If Murano can do all the orchestration via
HOT templates and Heat doesn't deal with service metadata that is out of
scope of HOT we can eliminate most of the overlapping between projects and
actually complement each other. The only thing I concerned about in this
context is
https://wiki.openstack.org/wiki/Heat/DSL2 and
https://wiki.openstack.org/wiki/Heat/Open_API as this is very similar to
what Murano does


-- 
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
sla...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Angus Salkeld]

rest I can write a Blueprint / proposal with more details and
use-cases.


We already have this functionality. Whenever you use a Ref or Attr
from another resource we become dependant on it. This feels like
replacing an existing system with a completely different one with
very little benefit.

To me what is missing to better support complex software configuration
is :
- better integrating with existing configuration tools (puppet, chef,
  salt, ansible, etc). (resource types)
- integration with a ssh key store (Barbican) so the tools above can
  do non-userdata related configuration

-Angus



Thanks,
LN



Clint Byrum  wrote on 10/11/2013 12:40:19 PM:


From: Clint Byrum 
To: openstack-dev 
Date: 10/11/2013 12:43 PM
Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
proposal for workflows

Excerpts from Stan Lagun's message of 2013-10-11 07:22:37 -0700:
> Hello,
>
> Thanks Angus, Clint, I've got your design.
>
> It seems that Murano can built on top of that. With service metadata
> knowledge Murano can generate HOT templates with set of interdependent
> configs.
> Here is what will be needed:
>
> 1. Ability to implement support for custom software configuration tool
> (type: OS::SoftwareConfig::MuranoAgent)

These are really syntactic sugar, but you can implement them as resource
plugins for users who want Murano resources. In the absence of this,
just putting things in the free form metadata area of the resource that
the MuranoAgent can interpret would suffice.

> 2. Ability to provide arbitrary input values for the config

We already have that, there's a free-form json document called "metadata"
attached to every resource. Or maybe I missed what you mean here. The
new capability that is in the works that will make that better is to
have multiple reusable metadata blocks referenced on one instance.

> 3. Ability to return arbitrary (JSON-compatible) data structure from

config

> application and use attributes of that structure as an input for other
> configs

Note that I'd like to see more use cases specified for this ability. The
random string generator that Steve Baker has put up should handle most
cases where you just need passwords. Generated key sharing might best
be deferred to something like Barbican which does a lot more than Heat
to try and keep your secrets safe.

> 4. Ability to provide config body that is an input to Murano Agent of
> arbitrary size

Isn't this the same as 2?

> 5. Work well with large graph of configs with a lot of dependencies.
> Independent configs on different VMs should be applied in parallel.
>

Yes this does look good. For dependent configs, the exsiting wait
condition can be used.

> Does it confirm to your plans?
>

I think it confirms that we're heading toward consensus on where to draw
the software config vs. infrastructure orchestration line. That is very
exciting. :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Thomas Spatzier]

Steven Dake  wrote on 11.10.2013 21:02:38:
> From: Steven Dake 
> To: OpenStack Development Mailing List
,
> Date: 11.10.2013 21:04
> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> proposal for workflows
>
> On 10/11/2013 11:55 AM, Lakshminaraya Renganarayana wrote:
> Clint Byrum  wrote on 10/11/2013 12:40:19 PM:
>
> > From: Clint Byrum 
> > To: openstack-dev 
> > Date: 10/11/2013 12:43 PM
> > Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
> > proposal for workflows
> >
> > > 3. Ability to return arbitrary (JSON-compatible) data structure
> from config
> > > application and use attributes of that structure as an input for
other
> > > configs
> >
> > Note that I'd like to see more use cases specified for this ability.
The
> > random string generator that Steve Baker has put up should handle most
> > cases where you just need passwords. Generated key sharing might best
> > be deferred to something like Barbican which does a lot more than Heat
> > to try and keep your secrets safe.
>
> I had seen a deployment scenario that needed more than random string
> generator. It was during the deployment of a system that has
> clustered application servers, i.e., a cluster of application server
> nodes + a cluster manager node. The deployment progresses by all the
> VMs (cluster-manager and cluster-nodes) starting concurrently. Then
> the cluster-nodes wait for the cluster-manager to send them data
> (xml) to configure themselves. The cluster-manager after reading its
> own config file, generates config-data for each cluster-node and
> sends it to them.

> Is the config data per cluster node unique to each node?  If not:

I think Lakshmi's example (IBM WebSphere, right?) talks about a case where
the per cluster member info is unique per member, so the one fits all
approach does not work. In addition, I think there is a constraint that
members must join one by one and cannot join concurrently.

>
> Change deployment to following model:
> 1. deploy cluster-manager as a resource with a waitcondition -
> passing the data using the cfn-signal  -d to send the xml blob
> 2. have cluster nodes wait on wait condition in #1, using data from
> the cfn-signal
>
> If so, join the config data sent in cfn-signal and break it apart by
> the various cluster nodes in #2
> Thanks,
> LN

>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Lakshminaraya Renganarayana]

Hi Angus,

Thanks for detailed reply. I have a few comments that I have written below
in the context.


Angus Salkeld  wrote on 10/13/2013 06:40:01 PM:

> >
> >- INPUTS: all the attributes that are consumed/used/read by that
resource
> >(currently, we have Ref, GetAttrs that can give this implicitly)
> >
> >- OUTPUTS: all the attributes that are produced/written by that resource
(I
> >do not know if this write-set is currently well-defined for a resource.
I
> >think some of them are implicitly defined by Heat on particular resource
> >types.)
> >
> >- Global name-space and data-space : all the values produced and
consumed
> >(INPUTS/OUTPUTS) are described using a names that are fully qualified
> >(XXX.stack_name.resource_name.property_name). The data values associated
> >with these names are stored in a global data-space.  Reads are blocking,
> >i.e., reading a value will block the execution resource/thread until the
> >value is available. Writes are non-blocking, i.e., any thread can write
a
> >value and the write will succeed immediately.
>
> I don't believe this would give us any new behaviour.

I believe that in today's Heat, wait-conditions and signals are the only
mechanism for synchronization during software configuration. The proposed
mechanism would provide a higher level synchronization based on
blocking-reads.
For example, if one is using Chef for software configuration, then the
recipes can use the proposed mechanism to wait for the all the node[][]
attributes
they require before starting the recipe execution. And, Heat can actually
analyze and reason about deadlock properties of such a synchronization. On
the other hand, if the recipe were using wait-conditions how would Heat
reason about deadlock properties of it?

>
> >
> >The ability to define resources at arbitrary levels of granularity
together
> >with the explicit specification of INPUTS/OUTPUTS allows us to reap the
> >benefits G1 and G2 outlined above. Note that the ability to reason about
> >the inputs/outputs of each resource and the induced dependencies will
also
> >allow Heat to detect dead-locks via dependence cycles (benefit G3). This
is
> >already done today in Heat for Refs, GetAttr on base-resources, but the
> >proposal is to extend the same to arbitrary attributes for any resource.
>
> How are TemplateResources and NestedStacks any different? To my
> knowledge this is aleady the case.
>
> >The blocking-read and non-blocking writes further structures the
> >specification to avoid deadlocks and race conditions (benefit G3).
>
> Have you experienced deadlocks with heat? I have never seen this...

Heat as it is today does not tackle the problem of synchronization during
software configuration and hence the problems I see cannot be attributed to
Heat and can only be attributed to the scripts / recipes that do the
software configuration. However, if we envision Heat to provide some
support for software configuration I can easily imagine cases where
it is impossible for Heat to analyze/reason with wait-conditions and hence
leading to deadlocks. Wait-conditions and signals are equal to
Timed-Semaphores in their power and expressivity and these are
known for their problems with deadlocks.

>
> To me what is missing to better support complex software configuration
> is :
> - better integrating with existing configuration tools (puppet, chef,
>salt, ansible, etc). (resource types)

One question is, whether in this integration the synchronization is
completely
left to the configuration tools or Heat would be involved in it. If it is
left to configuration tools, say chef, then the question is how does the
iterative convergence style execution of chef interfere with the schedule
order that Heat determines for a template. On the other hand, if Heat
provides
the mechanism for synchronization, then the question is whether
wait-conditions
and signals are the right abstractions for them. What are your thoughts on
this?

Thanks,
LN
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[John Davidge -X (jodavidg - AAP3 INC at Cisco)]

It looks like this discussion involves many of the issues faced when
developing the Curvature & Donabe frameworks, which were presented at the
Portland Summit - slides and video here:

http://www.openstack.org/summit/portland-2013/session-videos/presentation/i
nteractive-visual-orchestration-with-curvature-and-donabe

Much of the work on the Donabe side revolved around defining a simple
JSON-based API for describing the sorts of virtual application templates
being discussed. All of the code for both Curvature and Donabe has
recently been made open source and is available here:

http://ciscosystems.github.io/curvature/

http://ciscosystems.github.io/donabe/

It looks like some of the ground covered by these projects can be helpful
to this discussion.

John Davidge
jodav...@cisco.com



>-- Forwarded message --
>From: Thomas Spatzier 
>Date: Wed, Oct 9, 2013 at 12:40 AM
>Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
>proposal for workflows
>To: OpenStack Development Mailing List 
>
>
>Excerpts from Clint Byrum's message
>
>> From: Clint Byrum 
>> To: openstack-dev ,
>> Date: 09.10.2013 03:54
>> Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
>> proposal for workflows
>>
>> Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700:
>> > Hello,
>> >
>> >
>> > That is why it is necessary to have some central coordination service
>which
>> > would handle deployment workflow and perform specific actions (create
>VMs
>> > and other OpenStack resources, do something on that VM) on each stage
>> > according to that workflow. We think that Heat is the best place for
>such
>> > service.
>> >
>>
>> I'm not so sure. Heat is part of the Orchestration program, not
>>workflow.
>>
>
>I agree. HOT so far was thought to be a format for describing templates in
>a structural, declaritive way. Adding workflows would stretch it quite a
>bit. Maybe we should see what aspects make sense to be added to HOT, and
>then how to do workflow like orchestration in a layer on top.
>
>> > Our idea is to extend HOT DSL by adding  workflow definition
>capabilities
>> > as an explicit list of resources, components¹ states and actions.
>States
>> > may depend on each other so that you can reach state X only after
>you¹ve
>> > reached states Y and Z that the X depends on. The goal is from initial
>> > state to reach some final state ³Deployed².
>> >
>
>We also would like to add some mechanisms to HOT for declaratively doing
>software component orchestration in Heat, e.g. saying that one component
>depends on another one, or needs input from another one once it has been
>deployed etc. (I BTW started to write a wiki page, which is admittedly far
>from complete, but I would be happy to work on it with interested folks -
>https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider).
>However, we must be careful not to make such features too complicated so
>nobody will be able to use it any more. That said, I believe we could make
>HOT cover some levels of complexity, but not all. And then maybe workflow
>based orchestration on top is needed.
>
>>
>> Orchestration is not workflow, and HOT is an orchestration templating
>> language, not a workflow language. Extending it would just complect two
>> very different (though certainly related) tasks.
>>
>> I think the appropriate thing to do is actually to join up with the
>> TaskFlow project and consider building it into a workflow service or
>tools
>> (it is just a library right now).
>>
>> > There is such state graph for each of our deployment entities
>>(service,
>> > VMs, other things). There is also an action that must be performed on
>each
>> > state.
>>
>> Heat does its own translation of the orchestration template into a
>> workflow right now, but we have already discussed using TaskFlow to
>> break up the orchestration graph into distributable jobs. As we get more
>> sophisticated on updates (rolling/canary for instance) we'll need to
>> be able to reason about the process without having to glue all the
>> pieces together.
>>
>> > We propose to extend HOT DSL with workflow definition capabilities
>where
>> > you can describe step by step instruction to install service and
>properly
>> > handle errors on each step.
>> >
>> > We already have an experience in implementation of the DSL, workflow
>> > description and processing mechanism for complex deployments and
>believe
>> > we¹ll all benefit by re-using th

Re: [openstack-dev] [Heat] HOT Software orchestration proposal for workflows

[Zane Bitter]

On 18/10/13 20:24, John Davidge -X (jodavidg - AAP3 INC at Cisco) wrote:

It looks like this discussion involves many of the issues faced when
developing the Curvature & Donabe frameworks, which were presented at the
Portland Summit - slides and video here:

http://www.openstack.org/summit/portland-2013/session-videos/presentation/i
nteractive-visual-orchestration-with-curvature-and-donabe

Much of the work on the Donabe side revolved around defining a simple
JSON-based API for describing the sorts of virtual application templates
being discussed. All of the code for both Curvature and Donabe has
recently been made open source and is available here:

http://ciscosystems.github.io/curvature/

http://ciscosystems.github.io/donabe/


Hey John,
Congrats on getting this stuff Open-Sourced BTW (I know it's been out 
for a while now).


Can you be more specific about the parts that are relevant to this 
discussion? I'd be interested to know how Donabe handles configuring the 
software on Nova servers for a start.



It looks like some of the ground covered by these projects can be helpful
to this discussion.


Yep, it would be great to get input from any folks in the community who 
have experience with this problem.


cheers,
Zane.



John Davidge
jodav...@cisco.com




-- Forwarded message --
From: Thomas Spatzier 
Date: Wed, Oct 9, 2013 at 12:40 AM
Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
proposal for workflows
To: OpenStack Development Mailing List 


Excerpts from Clint Byrum's message


From: Clint Byrum 
To: openstack-dev ,
Date: 09.10.2013 03:54
Subject: Re: [openstack-dev] [Heat] HOT Software orchestration
proposal for workflows

Excerpts from Stan Lagun's message of 2013-10-08 13:53:45 -0700:

Hello,


That is why it is necessary to have some central coordination service

which

would handle deployment workflow and perform specific actions (create

VMs

and other OpenStack resources, do something on that VM) on each stage
according to that workflow. We think that Heat is the best place for

such

service.



I'm not so sure. Heat is part of the Orchestration program, not
workflow.



I agree. HOT so far was thought to be a format for describing templates in
a structural, declaritive way. Adding workflows would stretch it quite a
bit. Maybe we should see what aspects make sense to be added to HOT, and
then how to do workflow like orchestration in a layer on top.


Our idea is to extend HOT DSL by adding  workflow definition

capabilities

as an explicit list of resources, components¹ states and actions.

States

may depend on each other so that you can reach state X only after

you¹ve

reached states Y and Z that the X depends on. The goal is from initial
state to reach some final state ³Deployed².



We also would like to add some mechanisms to HOT for declaratively doing
software component orchestration in Heat, e.g. saying that one component
depends on another one, or needs input from another one once it has been
deployed etc. (I BTW started to write a wiki page, which is admittedly far
from complete, but I would be happy to work on it with interested folks -
https://wiki.openstack.org/wiki/Heat/Software-Configuration-Provider).
However, we must be careful not to make such features too complicated so
nobody will be able to use it any more. That said, I believe we could make
HOT cover some levels of complexity, but not all. And then maybe workflow
based orchestration on top is needed.



Orchestration is not workflow, and HOT is an orchestration templating
language, not a workflow language. Extending it would just complect two
very different (though certainly related) tasks.

I think the appropriate thing to do is actually to join up with the
TaskFlow project and consider building it into a workflow service or

tools

(it is just a library right now).


There is such state graph for each of our deployment entities

(service,

VMs, other things). There is also an action that must be performed on

each

state.


Heat does its own translation of the orchestration template into a
workflow right now, but we have already discussed using TaskFlow to
break up the orchestration graph into distributable jobs. As we get more
sophisticated on updates (rolling/canary for instance) we'll need to
be able to reason about the process without having to glue all the
pieces together.


We propose to extend HOT DSL with workflow definition capabilities

where

you can describe step by step instruction to install service and

properly

handle errors on each step.

We already have an experience in implementation of the DSL, workflow
description and processing mechanism for complex deployments and

believe

we¹ll all benefit by re-using this experience and existing code,

having

properly discussed and agreed on abstraction layers and distribution

of

responsibilities between OS components. There is an idea of

implementing

part of workflow processing