Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Steve Baker  wrote on 21.11.2013 21:19:07:
> From: Steve Baker 
> To: openstack-dev@lists.openstack.org,
> Date: 21.11.2013 21:25
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> On 11/21/2013 08:48 PM, Thomas Spatzier wrote:
> > Excerpts from Steve Baker's message on 21.11.2013 00:00:47:
> >> From: Steve Baker 
> >> To: openstack-dev@lists.openstack.org,
> >> Date: 21.11.2013 00:04
> >> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> >> refined after design summit discussions

> > I thought about the name "SoftwareApplier" some more and while it is
clear
> > what it does (it applies a software config to a server), the naming is
not
> > really consistent with all the other resources in Heat. Every other
> > resource type is called after the thing that you get when the template
gets
> > instantiated (a "Server", a "FloatingIP", a "VolumeAttachment" etc). In
> > case of SoftwareApplier what you actually get from a user perspective
is a
> > deployed instance of the piece of software described be a
SoftwareConfig.
> > Therefore, I was calling it SoftwareDeployment orignally, because you
get a
> > software deployment (according to a config). Any comments on that name?
> SoftwareDeployment is a better name, apart from those 3 extra letters.
> I'll rename my POC.  Sorry nannj, you'll need to rename them back ;)

Ok, I'll change the name back in the wiki :-)

>
> > If we think this thru with respect to "remove-config" (even though this
> > needs more thought), a SoftwareApplier (that thing itself) would not
really
> > go to state DELETE_IN_PROGRESS during an update. It is always there on
the
> > VM but the software it deploys gets deleted and then reapplied or
> > whatever ...
> >
> > Now thinking more about update scenarios (which we can leave for an
> > iteration after the initial deployment is working), in my mental model
it
> > would be more consistent to have information for "handle_create",
> > "handle_delete", "handle_update" kinds of events all defined in the
> > SoftwareConfig resource. SoftwareConfig for represent configuration
> > information for one specific piece of software, e.g. a web server. So
it
> > could provide all the information you need to install it, to uninstall
it,
> > or to update its config. By updating the SoftwareApplier's (or
> > SoftwareDeployment's - my preferred name) state at runtime, the
in-instance
> > tools would grab the respective script of whatever an run it.
> >
> > So SoftwareConfig could look like:
> >
> > resources:
> >   my_webserver_config:
> > type: OS::Heat::SoftwareConfig
> > properties:
> >   http_port:
> > type: number
> >   # some more config props
> >
> >   config_create:
http://www.example.com/my_scripts/webserver/install.sh
> >   config_delete:
> > http://www.example.com/my_scripts/webserver/uninstall.sh
> >   config_update:
> > http://www.example.com/my_scripts/webserver/applyupdate.sh
> >
> >
> > At runtime, when a SoftwareApplier gets created, it looks for the
> > 'config_create' hook and triggers that automation. When it gets
deleted, it
> > looks for the 'config_delete' hook and so on. Only config_create is
> > mandatory.
> > I think that would also give us nice extensibility for future use
cases.
> > For example, Heat today does not support something like stop-stack or
> > start-stack which would be pretty useful though. If we have it one day,
we
> > would just add a 'config_start' hook to the SoftwareConfig.
> >
> >
> > [1]
> >
https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-spec
> > [2] https://blueprints.launchpad.net/heat/+spec/hot-software-config
> >
> With the caveat that what we're discussing here is a future
enhancement...
>
> The problem I see with config_create/config_update/config_delete in a
> single SoftwareConfig is that we probably can't assume these 3 scripts
> consume the same inputs and produce the same outputs.

We could make it a convention that creators of software configs have to use
the same signature for the automation of create, delete etc. Or at least
input param names must be the same, while some pieces might take a subset
only. E.g. delete will probably take less inputs. This way we could have a
self-contained config.
As you said above, implementation-wise this is probably a future
enhancement, so once we have he 

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Thomas Spatzier  wrote on 11/21/2013 02:48:14 
AM:
> ...
> Now thinking more about update scenarios (which we can leave for an
> iteration after the initial deployment is working),

I recommend thinking about UPDATE from the start.  We should have an 
implementation in which CREATE and UPDATE share as much mechanism as is 
reasonable, which requires thinking about UPDATE while designing CREATE.

> in my mental model it
> would be more consistent to have information for "handle_create",
> "handle_delete", "handle_update" kinds of events all defined in the
> SoftwareConfig resource.

+1 for putting these on the definition instead of the use; I also noted 
this earlier.

-1 for having an update method.  The orientation to idempotent 
forward-progress operations means that we need only one, which handles 
both CREATE and UPDATE.

Regards,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/21/2013 08:48 PM, Thomas Spatzier wrote:
> Excerpts from Steve Baker's message on 21.11.2013 00:00:47:
>> From: Steve Baker 
>> To: openstack-dev@lists.openstack.org,
>> Date: 21.11.2013 00:04
>> Subject: Re: [openstack-dev] [Heat] HOT software configuration
>> refined after design summit discussions
>>
>> On 11/21/2013 11:41 AM, Clint Byrum wrote:
>>> Excerpts from Mike Spreitzer's message of 2013-11-20 13:46:25 -0800:
>>>> Clint Byrum  wrote on 11/19/2013 04:28:31 PM:
> 
>
>>> I am worried about the explosion of possibilities that comes from
> trying
>>> to deal with all of the diff's possible inside an instance. If there is
> an
>>> actual REST interface for a thing, then yes, let's use that. For
> instance,
>>> if we are using docker, there is in fact a very straight forward way to
>>> say "remove entity X". If we are using packages we have the same thing.
>>> However, if we are just trying to write chef configurations, we have to
>>> write reverse chef configurations.
>>>
>>> What I meant to convey is "let's give this piece of the interface a lot
> of
>>> thought". Not "this is wrong to even have." Given a couple of days now,
>>> I think we do need "apply" and "remove". We should also provide really
>>> solid example templates for this concept.
>> You're right, I'm already starting to see issues with my current
> approach.
>> This smells like a new blueprint. I'll remove it from the scope of the
>> current software config work and raise a blueprint to track
> remove-config.
>
> So I read thru those recent discussions and in parallel also started to
> update the design wiki. BTW, nanjj renamed the wiki to [1] (but also made a
> redirect from the previous ...-WIP page) and linked it as spec to BP [2].
>
> I'll leave out the remove-config thing for now. While thinking about the
> overall picture, I came up with some other comments:
>
> I thought about the name "SoftwareApplier" some more and while it is clear
> what it does (it applies a software config to a server), the naming is not
> really consistent with all the other resources in Heat. Every other
> resource type is called after the thing that you get when the template gets
> instantiated (a "Server", a "FloatingIP", a "VolumeAttachment" etc). In
> case of SoftwareApplier what you actually get from a user perspective is a
> deployed instance of the piece of software described be a SoftwareConfig.
> Therefore, I was calling it SoftwareDeployment orignally, because you get a
> software deployment (according to a config). Any comments on that name?
SoftwareDeployment is a better name, apart from those 3 extra letters.
I'll rename my POC.  Sorry nannj, you'll need to rename them back ;)

> If we think this thru with respect to "remove-config" (even though this
> needs more thought), a SoftwareApplier (that thing itself) would not really
> go to state DELETE_IN_PROGRESS during an update. It is always there on the
> VM but the software it deploys gets deleted and then reapplied or
> whatever ...
>
> Now thinking more about update scenarios (which we can leave for an
> iteration after the initial deployment is working), in my mental model it
> would be more consistent to have information for "handle_create",
> "handle_delete", "handle_update" kinds of events all defined in the
> SoftwareConfig resource. SoftwareConfig for represent configuration
> information for one specific piece of software, e.g. a web server. So it
> could provide all the information you need to install it, to uninstall it,
> or to update its config. By updating the SoftwareApplier's (or
> SoftwareDeployment's - my preferred name) state at runtime, the in-instance
> tools would grab the respective script of whatever an run it.
>
> So SoftwareConfig could look like:
>
> resources:
>   my_webserver_config:
> type: OS::Heat::SoftwareConfig
> properties:
>   http_port:
> type: number
>   # some more config props
>
>   config_create: http://www.example.com/my_scripts/webserver/install.sh
>   config_delete:
> http://www.example.com/my_scripts/webserver/uninstall.sh
>   config_update:
> http://www.example.com/my_scripts/webserver/applyupdate.sh
>
>
> At runtime, when a SoftwareApplier gets created, it looks for the
> 'config_create' hook and triggers that automation. When it gets deleted, it
> looks for the 'config_delete' hook and so on. Only config_create is
> m

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Excerpts from Steve Baker's message on 21.11.2013 00:00:47:
> From: Steve Baker 
> To: openstack-dev@lists.openstack.org,
> Date: 21.11.2013 00:04
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> On 11/21/2013 11:41 AM, Clint Byrum wrote:
> > Excerpts from Mike Spreitzer's message of 2013-11-20 13:46:25 -0800:
> >> Clint Byrum  wrote on 11/19/2013 04:28:31 PM:



> >>
> > I am worried about the explosion of possibilities that comes from
trying
> > to deal with all of the diff's possible inside an instance. If there is
an
> > actual REST interface for a thing, then yes, let's use that. For
instance,
> > if we are using docker, there is in fact a very straight forward way to
> > say "remove entity X". If we are using packages we have the same thing.
> > However, if we are just trying to write chef configurations, we have to
> > write reverse chef configurations.
> >
> > What I meant to convey is "let's give this piece of the interface a lot
of
> > thought". Not "this is wrong to even have." Given a couple of days now,
> > I think we do need "apply" and "remove". We should also provide really
> > solid example templates for this concept.
> You're right, I'm already starting to see issues with my current
approach.
>
> This smells like a new blueprint. I'll remove it from the scope of the
> current software config work and raise a blueprint to track
remove-config.

So I read thru those recent discussions and in parallel also started to
update the design wiki. BTW, nanjj renamed the wiki to [1] (but also made a
redirect from the previous ...-WIP page) and linked it as spec to BP [2].

I'll leave out the remove-config thing for now. While thinking about the
overall picture, I came up with some other comments:

I thought about the name "SoftwareApplier" some more and while it is clear
what it does (it applies a software config to a server), the naming is not
really consistent with all the other resources in Heat. Every other
resource type is called after the thing that you get when the template gets
instantiated (a "Server", a "FloatingIP", a "VolumeAttachment" etc). In
case of SoftwareApplier what you actually get from a user perspective is a
deployed instance of the piece of software described be a SoftwareConfig.
Therefore, I was calling it SoftwareDeployment orignally, because you get a
software deployment (according to a config). Any comments on that name?

If we think this thru with respect to "remove-config" (even though this
needs more thought), a SoftwareApplier (that thing itself) would not really
go to state DELETE_IN_PROGRESS during an update. It is always there on the
VM but the software it deploys gets deleted and then reapplied or
whatever ...

Now thinking more about update scenarios (which we can leave for an
iteration after the initial deployment is working), in my mental model it
would be more consistent to have information for "handle_create",
"handle_delete", "handle_update" kinds of events all defined in the
SoftwareConfig resource. SoftwareConfig for represent configuration
information for one specific piece of software, e.g. a web server. So it
could provide all the information you need to install it, to uninstall it,
or to update its config. By updating the SoftwareApplier's (or
SoftwareDeployment's - my preferred name) state at runtime, the in-instance
tools would grab the respective script of whatever an run it.

So SoftwareConfig could look like:

resources:
  my_webserver_config:
type: OS::Heat::SoftwareConfig
properties:
  http_port:
type: number
  # some more config props

  config_create: http://www.example.com/my_scripts/webserver/install.sh
  config_delete:
http://www.example.com/my_scripts/webserver/uninstall.sh
  config_update:
http://www.example.com/my_scripts/webserver/applyupdate.sh


At runtime, when a SoftwareApplier gets created, it looks for the
'config_create' hook and triggers that automation. When it gets deleted, it
looks for the 'config_delete' hook and so on. Only config_create is
mandatory.
I think that would also give us nice extensibility for future use cases.
For example, Heat today does not support something like stop-stack or
start-stack which would be pretty useful though. If we have it one day, we
would just add a 'config_start' hook to the SoftwareConfig.


[1]
https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-spec
[2] https://blueprints.launchpad.net/heat/+spec/hot-software-config

> >>>>> ...
> >>>> A specific use-case I'm trying to address here is trip

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Mike Spreitzer's message of 2013-11-20 15:16:45 -0800:
> Clint Byrum  wrote on 11/20/2013 05:41:16 PM:
> 
> > Autoscaling doesn't really fly with stateful services IMO.
> 
> I presume you're concerned about the "auto" part, not the "scaling".  Even 
> a stateful group is something you may want to scale; it just takes a more 
> complex set of operations to accomplish that.  If we can make a Heat 
> autoscaling group invoke the right set of operations, why not?
> 

It is most definitely possible and necessary. We _must_ do this.

It does not fly with today's limited auto-scaling.

If we can get the link between orchestration and work-flow working well,
the end result should be harmonious automation and then automatic scaling
will indeed be possible for stateful services.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Clint Byrum  wrote on 11/20/2013 05:41:16 PM:

> Autoscaling doesn't really fly with stateful services IMO.

I presume you're concerned about the "auto" part, not the "scaling".  Even 
a stateful group is something you may want to scale; it just takes a more 
complex set of operations to accomplish that.  If we can make a Heat 
autoscaling group invoke the right set of operations, why not?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Steve Baker  wrote on 11/20/2013 06:00:47 PM:
...
> > What I meant to convey is "let's give this piece of the interface a 
lot of
> > thought". Not "this is wrong to even have." Given a couple of days 
now,
> > I think we do need "apply" and "remove". We should also provide really
> > solid example templates for this concept.
> You're right, I'm already starting to see issues with my current 
approach.

Can you elaborate?  After a little thought, I am starting to warm to the 
idea of a 'remove' operation.  It fits the desired-state-based management 
paradigm like this: it says how to move from state PRESENT to state 
ABSENT.

BTW, the pair of add and remove operations belong on the config definition 
not the config use.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Regarding my previous email:

> Steve Baker  wrote on 11/19/2013 03:40:54 PM:
> ...
> > How to define and deliver this agent is the challenge. Some options 
are:
> > 1) install it as part of the image customization/bootstrapping (golden
> > images or cloud-init) 
> 
> > 2) define a (mustache?) template in the SoftwareConfig which
> > os-collect-config transforms into the agent script, which
> > os-collect-config then executes 
> 
> I do not follow what you mean here.  Can you please elaborate a little? 
> 
> > 3) a CM tool specific implementation of SoftwareApplier builds and
> > delivers a complete agent to os-collect-config which executes it
> 
> Could instead the agent be a package that the applier templates asks
> to be installed? 

We may not need to active agent to be CM tool specific.  I think the 
remarks about os-collect-config show this.  Maybe all we need is a CM tool 
specific package to be installed to supply the (I use the singular because 
of our earlier conversation that affirmed the expectation of only one CM 
tool per VM) hook that os-collect-config calls.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/21/2013 11:41 AM, Clint Byrum wrote:
> Excerpts from Mike Spreitzer's message of 2013-11-20 13:46:25 -0800:
>> Clint Byrum  wrote on 11/19/2013 04:28:31 PM:
>>> From: Clint Byrum 
>>> To: openstack-dev , 
>>> Date: 11/19/2013 04:30 PM
>>> Subject: Re: [openstack-dev] [Heat] HOT software configuration 
>>> refined after design summit discussions
>>>
>>> Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
>>>> On 11/20/2013 09:50 AM, Clint Byrum wrote:
>>>>> Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
>>>>>> Regarding apply_config/remove_config, if a SoftwareApplier resource 
>> is
>>>>>> deleted it should trigger any remove_config and wait for the server 
>> to
>>>>>> acknowledge when that is complete. This allows for any
>>>>>> evacuation/deregistering workloads to be executed.
>>>>>>
>>>>> I'm a little worried about the road that leads us down. Most 
>> configuration
>>>>> software defines forward progress only. Meaning, if you want 
>> something
>>>>> not there, you don't remove it from your assertions, you assert that 
>> it
>>>>> is not there.
>> I am worried too.  But I do not entirely follow your reasoning.  When I 
>> UPDATE a stack with a new template, am I supposed to write in that 
>> template not just what I want the stack to be but also how that differs 
>> from what it currently is?  That is not REST.  Not that I am a total REST 
>> zealot, but I am a fan of managing in terms of desired state.  But I agree 
>> there is a conflict between defining a 'remove' operation and the "forward 
>> progress only" mindset of most config tooling.
>>
> I am worried about the explosion of possibilities that comes from trying
> to deal with all of the diff's possible inside an instance. If there is an
> actual REST interface for a thing, then yes, let's use that. For instance,
> if we are using docker, there is in fact a very straight forward way to
> say "remove entity X". If we are using packages we have the same thing.
> However, if we are just trying to write chef configurations, we have to
> write reverse chef configurations.
>
> What I meant to convey is "let's give this piece of the interface a lot of
> thought". Not "this is wrong to even have." Given a couple of days now,
> I think we do need "apply" and "remove". We should also provide really
> solid example templates for this concept.
You're right, I'm already starting to see issues with my current approach.

This smells like a new blueprint. I'll remove it from the scope of the
current software config work and raise a blueprint to track remove-config.
>>>>> ...
>>>> A specific use-case I'm trying to address here is tripleo doing an
>>>> update-replace on a nova compute node. The remove_config contains the
>>>> workload to evacuate VMs and signal heat when the node is ready to be
>>>> shut down. This is more involved than just "uninstall the things".
>>>>
>>>> Could you outline in some more detail how you think this could be 
>> done?
>>> So for that we would not remove the software configuration for the
>>> nova-compute, we would assert that the machine needs vms evacuated.
>>> We want evacuation to be something we explicitly do, not a side effect
>>> of deleting things.
>> Really?  You want to force the user to explicitly say "evacuate the VMs" 
>> in all the various ways a host deletion can happen?  E.g., when an 
>> autoscaling group of hosts shrinks?
>>
> Autoscaling doesn't really fly with stateful services IMO. Also for
> TripleO's use case, auto-scaling is not really a high priority. Hardware
> isn't nearly as easily allocatable as VM's.
>
> Anyway, there is a really complicated work-flow for decomissioning
> any stateful service, and it differs wildly between them. I do want to
> have a place to define that work-flow and reliably trigger it when it
> needs to be triggered. I do not want it to _only_ be available in the
> "delete this resource" case, and I also do not want it to _always_
> be run in that case, as I may legitimately be destroying the data too.
> I need a way to express that intention, and in my mind, the way to do
> that is to first complete an evacuation and then delete the thing.
>
> Better ideas are _most_ welcome.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Mike Spreitzer's message of 2013-11-20 13:46:25 -0800:
> Clint Byrum  wrote on 11/19/2013 04:28:31 PM:
> > From: Clint Byrum 
> > To: openstack-dev , 
> > Date: 11/19/2013 04:30 PM
> > Subject: Re: [openstack-dev] [Heat] HOT software configuration 
> > refined after design summit discussions
> > 
> > Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
> > > On 11/20/2013 09:50 AM, Clint Byrum wrote:
> > > > Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> > > >> Regarding apply_config/remove_config, if a SoftwareApplier resource 
> is
> > > >> deleted it should trigger any remove_config and wait for the server 
> to
> > > >> acknowledge when that is complete. This allows for any
> > > >> evacuation/deregistering workloads to be executed.
> > > >>
> > > > I'm a little worried about the road that leads us down. Most 
> configuration
> > > > software defines forward progress only. Meaning, if you want 
> something
> > > > not there, you don't remove it from your assertions, you assert that 
> it
> > > > is not there.
> 
> I am worried too.  But I do not entirely follow your reasoning.  When I 
> UPDATE a stack with a new template, am I supposed to write in that 
> template not just what I want the stack to be but also how that differs 
> from what it currently is?  That is not REST.  Not that I am a total REST 
> zealot, but I am a fan of managing in terms of desired state.  But I agree 
> there is a conflict between defining a 'remove' operation and the "forward 
> progress only" mindset of most config tooling.
> 

I am worried about the explosion of possibilities that comes from trying
to deal with all of the diff's possible inside an instance. If there is an
actual REST interface for a thing, then yes, let's use that. For instance,
if we are using docker, there is in fact a very straight forward way to
say "remove entity X". If we are using packages we have the same thing.
However, if we are just trying to write chef configurations, we have to
write reverse chef configurations.

What I meant to convey is "let's give this piece of the interface a lot of
thought". Not "this is wrong to even have." Given a couple of days now,
I think we do need "apply" and "remove". We should also provide really
solid example templates for this concept.

> > > > ...
> > > A specific use-case I'm trying to address here is tripleo doing an
> > > update-replace on a nova compute node. The remove_config contains the
> > > workload to evacuate VMs and signal heat when the node is ready to be
> > > shut down. This is more involved than just "uninstall the things".
> > > 
> > > Could you outline in some more detail how you think this could be 
> done?
> > > 
> > 
> > So for that we would not remove the software configuration for the
> > nova-compute, we would assert that the machine needs vms evacuated.
> > We want evacuation to be something we explicitly do, not a side effect
> > of deleting things.
> 
> Really?  You want to force the user to explicitly say "evacuate the VMs" 
> in all the various ways a host deletion can happen?  E.g., when an 
> autoscaling group of hosts shrinks?
> 

Autoscaling doesn't really fly with stateful services IMO. Also for
TripleO's use case, auto-scaling is not really a high priority. Hardware
isn't nearly as easily allocatable as VM's.

Anyway, there is a really complicated work-flow for decomissioning
any stateful service, and it differs wildly between them. I do want to
have a place to define that work-flow and reliably trigger it when it
needs to be triggered. I do not want it to _only_ be available in the
"delete this resource" case, and I also do not want it to _always_
be run in that case, as I may legitimately be destroying the data too.
I need a way to express that intention, and in my mind, the way to do
that is to first complete an evacuation and then delete the thing.

Better ideas are _most_ welcome.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/21/2013 10:46 AM, Mike Spreitzer wrote:
> Clint Byrum  wrote on 11/19/2013 04:28:31 PM:
> > From: Clint Byrum 
> > To: openstack-dev ,
> > Date: 11/19/2013 04:30 PM
> > Subject: Re: [openstack-dev] [Heat] HOT software configuration
> > refined after design summit discussions
> >
> > Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
> > > On 11/20/2013 09:50 AM, Clint Byrum wrote:
> > > > Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> > > >> Regarding apply_config/remove_config, if a SoftwareApplier
> resource is
> > > >> deleted it should trigger any remove_config and wait for the
> server to
> > > >> acknowledge when that is complete. This allows for any
> > > >> evacuation/deregistering workloads to be executed.
> > > >>
> > > > I'm a little worried about the road that leads us down. Most
> configuration
> > > > software defines forward progress only. Meaning, if you want
> something
> > > > not there, you don't remove it from your assertions, you assert
> that it
> > > > is not there.
>
> I am worried too.  But I do not entirely follow your reasoning.  When
> I UPDATE a stack with a new template, am I supposed to write in that
> template not just what I want the stack to be but also how that
> differs from what it currently is?  That is not REST.  Not that I am a
> total REST zealot, but I am a fan of managing in terms of desired
> state.  But I agree there is a conflict between defining a 'remove'
> operation and the "forward progress only" mindset of most config tooling.
>
As I'm currently proposing, here are some stack update scenarios:
* update results in modified software config, apply_config will be
executed again on the affected server
* update results in a server that requires replacement. This results in:
  * execute the remove_config workload on that server. The
SoftwareApplier resource remains in DELETE_IN_PROGRESS until signalled
that remove_config is complete
  * delete the server
  * create the replacement server
  * execute apply_config on that server...
> > > > ...
> > > A specific use-case I'm trying to address here is tripleo doing an
> > > update-replace on a nova compute node. The remove_config contains the
> > > workload to evacuate VMs and signal heat when the node is ready to be
> > > shut down. This is more involved than just "uninstall the things".
> > >
> > > Could you outline in some more detail how you think this could be
> done?
> > >
> >
> > So for that we would not remove the software configuration for the
> > nova-compute, we would assert that the machine needs vms evacuated.
> > We want evacuation to be something we explicitly do, not a side effect
> > of deleting things.
>
> Really?  You want to force the user to explicitly say "evacuate the
> VMs" in all the various ways a host deletion can happen?  E.g., when
> an autoscaling group of hosts shrinks?
>
Nobody is being forced. remove_config is entirely optional and only
exists for the more complex scenarios requiring evacuation/deregistering.

If remove_config is not specified, the SoftwareApplier should probably
go straight to DELETE_COMPLETE without waiting for any signal.

> > Perhaps having delete hooks for starting delete
> > work-flows is right, but it set off a red flag for me so I want to make
> > sure we think it through.
> >
> > Also IIRC, evacuation is not necessarily an in-instance thing. It looks
> > more like the weird thing we've been talking about lately which is
> > "how do we orchestrate tenant API's":
> >
> > https://etherpad.openstack.org/p/orchestrate-tenant-apis
>
> This looks promising to me.
It looks like these might be represented as SoftwareConfigs
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Clint Byrum  wrote on 11/19/2013 04:28:31 PM:
> From: Clint Byrum 
> To: openstack-dev , 
> Date: 11/19/2013 04:30 PM
> Subject: Re: [openstack-dev] [Heat] HOT software configuration 
> refined after design summit discussions
> 
> Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
> > On 11/20/2013 09:50 AM, Clint Byrum wrote:
> > > Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> > >> Regarding apply_config/remove_config, if a SoftwareApplier resource 
is
> > >> deleted it should trigger any remove_config and wait for the server 
to
> > >> acknowledge when that is complete. This allows for any
> > >> evacuation/deregistering workloads to be executed.
> > >>
> > > I'm a little worried about the road that leads us down. Most 
configuration
> > > software defines forward progress only. Meaning, if you want 
something
> > > not there, you don't remove it from your assertions, you assert that 
it
> > > is not there.

I am worried too.  But I do not entirely follow your reasoning.  When I 
UPDATE a stack with a new template, am I supposed to write in that 
template not just what I want the stack to be but also how that differs 
from what it currently is?  That is not REST.  Not that I am a total REST 
zealot, but I am a fan of managing in terms of desired state.  But I agree 
there is a conflict between defining a 'remove' operation and the "forward 
progress only" mindset of most config tooling.

> > > ...
> > A specific use-case I'm trying to address here is tripleo doing an
> > update-replace on a nova compute node. The remove_config contains the
> > workload to evacuate VMs and signal heat when the node is ready to be
> > shut down. This is more involved than just "uninstall the things".
> > 
> > Could you outline in some more detail how you think this could be 
done?
> > 
> 
> So for that we would not remove the software configuration for the
> nova-compute, we would assert that the machine needs vms evacuated.
> We want evacuation to be something we explicitly do, not a side effect
> of deleting things.

Really?  You want to force the user to explicitly say "evacuate the VMs" 
in all the various ways a host deletion can happen?  E.g., when an 
autoscaling group of hosts shrinks?

> Perhaps having delete hooks for starting delete
> work-flows is right, but it set off a red flag for me so I want to make
> sure we think it through.
> 
> Also IIRC, evacuation is not necessarily an in-instance thing. It looks
> more like the weird thing we've been talking about lately which is
> "how do we orchestrate tenant API's":
> 
> https://etherpad.openstack.org/p/orchestrate-tenant-apis

This looks promising to me.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Steve Baker  wrote on 11/19/2013 03:40:54 PM:
...
> How to define and deliver this agent is the challenge. Some options are:
> 1) install it as part of the image customization/bootstrapping (golden
> images or cloud-init)

> 2) define a (mustache?) template in the SoftwareConfig which
> os-collect-config transforms into the agent script, which
> os-collect-config then executes

I do not follow what you mean here.  Can you please elaborate a little?

> 3) a CM tool specific implementation of SoftwareApplier builds and
> delivers a complete agent to os-collect-config which executes it

Could instead the agent be a package that the applier templates asks to be 
installed?

Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Steve Baker  wrote on 20.11.2013 09:51:34:
> From: Steve Baker 
> To: openstack-dev@lists.openstack.org,
> Date: 20.11.2013 09:55
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> On 11/20/2013 09:29 PM, Clint Byrum wrote:
> > Excerpts from Thomas Spatzier's message of 2013-11-19 23:35:40 -0800:
> >> Excerpts from Steve Baker's message on 19.11.2013 21:40:54:
> >>> From: Steve Baker 
> >>> To: openstack-dev@lists.openstack.org,
> >>> Date: 19.11.2013 21:43
> >>> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> >>> refined after design summit discussions
> >>>
> >> 
> >>> I think there needs to a CM tool specific agent delivered to the
server
> >>> which os-collect-config invokes. This agent will transform the config
> >>> data (input values, CM script, CM specific specialness) to a CM tool
> >>> invocation.
> >>>
> >>> How to define and deliver this agent is the challenge. Some options
are:
> >>> 1) install it as part of the image customization/bootstrapping
(golden
> >>> images or cloud-init)
> >>> 2) define a (mustache?) template in the SoftwareConfig which
> >>> os-collect-config transforms into the agent script, which
> >>> os-collect-config then executes
> >>> 3) a CM tool specific implementation of SoftwareApplier builds and
> >>> delivers a complete agent to os-collect-config which executes it
> >>>
> >>> I may be leaning towards 3) at the moment. Hopefully any agent can be
> >>> generated with a sufficiently sophisticated base SoftwareApplier
type,
> >>> plus maybe some richer intrinsic functions.
> >> This is good summary of options; about the same we had in mind. And we
were
> >> also leaning towards 3. Probably the approach we would take is to get
a
> >> SoftwareApplier running for one CM tool (e.g. Chef), then look at
another
> >> tool (base shell scripts), and then see what the generic parts art
that can
> >> be factored into a base class.
> >>
> >>>>> The POC I'm working on is actually backed by a REST API which does
> >> dumb
> >>>>> (but structured) storage of SoftwareConfig and SoftwareApplier
> >> entities.
> >>>>> This has some interesting implications for managing SoftwareConfig
> >>>>> resources outside the context of the stack which uses them, but
lets
> >> not
> >>>>> worry too much about that *yet*.
> >>>> Sounds good. We are also defining some blueprints to break down the
> >> overall
> >>>> software config topic. We plan to share them later this week, and
then
> >> we
> >>>> can consolidate with your plans and see how we can best join forces.
> >>>>
> >>>>
> >>> At this point it would be very helpful to spec out how specific CM
tools
> >>> are invoked with given inputs, script, and CM tool specific options.
> >> That's our plan; and we would probably start with scripts and chef.
> >>
> >>> Maybe if you start with shell scripts, cfn-init and chef then we can
all
> >>> contribute other CM tools like os-config-applier, puppet, ansible,
> >>> saltstack.
> >>>
> >>> Hopefully by then my POC will at least be able to create resources,
if
> >>> not deliver some data to servers.
> >> We've been thinking about getting metadata to the in-instance parts on
the
> >> server and whether the resources you are building can serve the
purpose.
> >> I.e. pass and endpoint to the SoftwareConfig resources to the instance
and
> >> let the instance query the metadata from the resource. Sounds like
this is
> >> what you had in mind, so that would be a good point for integrating
the
> >> work. In the meantime, we can think of some shortcuts.
> >>
> > Note that os-collect-config is intended to be a light-weight generic
> > in-instance agent to do exactly this. Watch for Metadata changes, and
> > feed them to an underlying tool in a predictable interface. I'd hope
> > that any of the appliers would mostly just configure os-collect-config
> > to run a wrapper that speaks os-collect-config's interface.
> >
> > The interface is defined in the README:
> >
> > https://pypi.python.org/pypi/os-collect-config
> >
> > It is inevitable that we will extend os-collect-config to be able to
> >

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/20/2013 09:29 PM, Clint Byrum wrote:
> Excerpts from Thomas Spatzier's message of 2013-11-19 23:35:40 -0800:
>> Excerpts from Steve Baker's message on 19.11.2013 21:40:54:
>>> From: Steve Baker 
>>> To: openstack-dev@lists.openstack.org,
>>> Date: 19.11.2013 21:43
>>> Subject: Re: [openstack-dev] [Heat] HOT software configuration
>>> refined after design summit discussions
>>>
>> 
>>> I think there needs to a CM tool specific agent delivered to the server
>>> which os-collect-config invokes. This agent will transform the config
>>> data (input values, CM script, CM specific specialness) to a CM tool
>>> invocation.
>>>
>>> How to define and deliver this agent is the challenge. Some options are:
>>> 1) install it as part of the image customization/bootstrapping (golden
>>> images or cloud-init)
>>> 2) define a (mustache?) template in the SoftwareConfig which
>>> os-collect-config transforms into the agent script, which
>>> os-collect-config then executes
>>> 3) a CM tool specific implementation of SoftwareApplier builds and
>>> delivers a complete agent to os-collect-config which executes it
>>>
>>> I may be leaning towards 3) at the moment. Hopefully any agent can be
>>> generated with a sufficiently sophisticated base SoftwareApplier type,
>>> plus maybe some richer intrinsic functions.
>> This is good summary of options; about the same we had in mind. And we were
>> also leaning towards 3. Probably the approach we would take is to get a
>> SoftwareApplier running for one CM tool (e.g. Chef), then look at another
>> tool (base shell scripts), and then see what the generic parts art that can
>> be factored into a base class.
>>
>>>>> The POC I'm working on is actually backed by a REST API which does
>> dumb
>>>>> (but structured) storage of SoftwareConfig and SoftwareApplier
>> entities.
>>>>> This has some interesting implications for managing SoftwareConfig
>>>>> resources outside the context of the stack which uses them, but lets
>> not
>>>>> worry too much about that *yet*.
>>>> Sounds good. We are also defining some blueprints to break down the
>> overall
>>>> software config topic. We plan to share them later this week, and then
>> we
>>>> can consolidate with your plans and see how we can best join forces.
>>>>
>>>>
>>> At this point it would be very helpful to spec out how specific CM tools
>>> are invoked with given inputs, script, and CM tool specific options.
>> That's our plan; and we would probably start with scripts and chef.
>>
>>> Maybe if you start with shell scripts, cfn-init and chef then we can all
>>> contribute other CM tools like os-config-applier, puppet, ansible,
>>> saltstack.
>>>
>>> Hopefully by then my POC will at least be able to create resources, if
>>> not deliver some data to servers.
>> We've been thinking about getting metadata to the in-instance parts on the
>> server and whether the resources you are building can serve the purpose.
>> I.e. pass and endpoint to the SoftwareConfig resources to the instance and
>> let the instance query the metadata from the resource. Sounds like this is
>> what you had in mind, so that would be a good point for integrating the
>> work. In the meantime, we can think of some shortcuts.
>>
> Note that os-collect-config is intended to be a light-weight generic
> in-instance agent to do exactly this. Watch for Metadata changes, and
> feed them to an underlying tool in a predictable interface. I'd hope
> that any of the appliers would mostly just configure os-collect-config
> to run a wrapper that speaks os-collect-config's interface.
>
> The interface is defined in the README:
>
> https://pypi.python.org/pypi/os-collect-config
>
> It is inevitable that we will extend os-collect-config to be able to
> collect config data from whatever API these config applier resources
> make available. I would suggest then that we not all go off and reinvent
> os-collect-config for each applier, but rather enhance os-collect-config
> as needed and write wrappers for the other config tools which implement
> its interface.
>
> os-apply-config already understands this interface for obvious reasons.
>
> Bash scripts can use os-apply-config to extract individual values, as
> you might see in some of the os-refresh-config scripts that are run as
> part of tripleo. I don't think anything further is really needed there.
>
> For chef, some kind of ohai plugin to read os-collect-config's collected
> data would make sense.
>
I'd definitely start with occ as Clint outlines. It would be nice if occ
only had to be configured to poll metadata for the OS::Nova::Server to
fetch the aggregated data for the currently available SoftwareAppliers.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Thomas Spatzier's message of 2013-11-19 23:35:40 -0800:
> Excerpts from Steve Baker's message on 19.11.2013 21:40:54:
> > From: Steve Baker 
> > To: openstack-dev@lists.openstack.org,
> > Date: 19.11.2013 21:43
> > Subject: Re: [openstack-dev] [Heat] HOT software configuration
> > refined after design summit discussions
> >
> 
> > I think there needs to a CM tool specific agent delivered to the server
> > which os-collect-config invokes. This agent will transform the config
> > data (input values, CM script, CM specific specialness) to a CM tool
> > invocation.
> >
> > How to define and deliver this agent is the challenge. Some options are:
> > 1) install it as part of the image customization/bootstrapping (golden
> > images or cloud-init)
> > 2) define a (mustache?) template in the SoftwareConfig which
> > os-collect-config transforms into the agent script, which
> > os-collect-config then executes
> > 3) a CM tool specific implementation of SoftwareApplier builds and
> > delivers a complete agent to os-collect-config which executes it
> >
> > I may be leaning towards 3) at the moment. Hopefully any agent can be
> > generated with a sufficiently sophisticated base SoftwareApplier type,
> > plus maybe some richer intrinsic functions.
> 
> This is good summary of options; about the same we had in mind. And we were
> also leaning towards 3. Probably the approach we would take is to get a
> SoftwareApplier running for one CM tool (e.g. Chef), then look at another
> tool (base shell scripts), and then see what the generic parts art that can
> be factored into a base class.
> 
> > >> The POC I'm working on is actually backed by a REST API which does
> dumb
> > >> (but structured) storage of SoftwareConfig and SoftwareApplier
> entities.
> > >> This has some interesting implications for managing SoftwareConfig
> > >> resources outside the context of the stack which uses them, but lets
> not
> > >> worry too much about that *yet*.
> > > Sounds good. We are also defining some blueprints to break down the
> overall
> > > software config topic. We plan to share them later this week, and then
> we
> > > can consolidate with your plans and see how we can best join forces.
> > >
> > >
> > At this point it would be very helpful to spec out how specific CM tools
> > are invoked with given inputs, script, and CM tool specific options.
> 
> That's our plan; and we would probably start with scripts and chef.
> 
> >
> > Maybe if you start with shell scripts, cfn-init and chef then we can all
> > contribute other CM tools like os-config-applier, puppet, ansible,
> > saltstack.
> >
> > Hopefully by then my POC will at least be able to create resources, if
> > not deliver some data to servers.
> 
> We've been thinking about getting metadata to the in-instance parts on the
> server and whether the resources you are building can serve the purpose.
> I.e. pass and endpoint to the SoftwareConfig resources to the instance and
> let the instance query the metadata from the resource. Sounds like this is
> what you had in mind, so that would be a good point for integrating the
> work. In the meantime, we can think of some shortcuts.
> 

Note that os-collect-config is intended to be a light-weight generic
in-instance agent to do exactly this. Watch for Metadata changes, and
feed them to an underlying tool in a predictable interface. I'd hope
that any of the appliers would mostly just configure os-collect-config
to run a wrapper that speaks os-collect-config's interface.

The interface is defined in the README:

https://pypi.python.org/pypi/os-collect-config

It is inevitable that we will extend os-collect-config to be able to
collect config data from whatever API these config applier resources
make available. I would suggest then that we not all go off and reinvent
os-collect-config for each applier, but rather enhance os-collect-config
as needed and write wrappers for the other config tools which implement
its interface.

os-apply-config already understands this interface for obvious reasons.

Bash scripts can use os-apply-config to extract individual values, as
you might see in some of the os-refresh-config scripts that are run as
part of tripleo. I don't think anything further is really needed there.

For chef, some kind of ohai plugin to read os-collect-config's collected
data would make sense.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Excerpts from Steve Baker's message on 19.11.2013 21:40:54:
> From: Steve Baker 
> To: openstack-dev@lists.openstack.org,
> Date: 19.11.2013 21:43
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>

> I think there needs to a CM tool specific agent delivered to the server
> which os-collect-config invokes. This agent will transform the config
> data (input values, CM script, CM specific specialness) to a CM tool
> invocation.
>
> How to define and deliver this agent is the challenge. Some options are:
> 1) install it as part of the image customization/bootstrapping (golden
> images or cloud-init)
> 2) define a (mustache?) template in the SoftwareConfig which
> os-collect-config transforms into the agent script, which
> os-collect-config then executes
> 3) a CM tool specific implementation of SoftwareApplier builds and
> delivers a complete agent to os-collect-config which executes it
>
> I may be leaning towards 3) at the moment. Hopefully any agent can be
> generated with a sufficiently sophisticated base SoftwareApplier type,
> plus maybe some richer intrinsic functions.

This is good summary of options; about the same we had in mind. And we were
also leaning towards 3. Probably the approach we would take is to get a
SoftwareApplier running for one CM tool (e.g. Chef), then look at another
tool (base shell scripts), and then see what the generic parts art that can
be factored into a base class.

> >> The POC I'm working on is actually backed by a REST API which does
dumb
> >> (but structured) storage of SoftwareConfig and SoftwareApplier
entities.
> >> This has some interesting implications for managing SoftwareConfig
> >> resources outside the context of the stack which uses them, but lets
not
> >> worry too much about that *yet*.
> > Sounds good. We are also defining some blueprints to break down the
overall
> > software config topic. We plan to share them later this week, and then
we
> > can consolidate with your plans and see how we can best join forces.
> >
> >
> At this point it would be very helpful to spec out how specific CM tools
> are invoked with given inputs, script, and CM tool specific options.

That's our plan; and we would probably start with scripts and chef.

>
> Maybe if you start with shell scripts, cfn-init and chef then we can all
> contribute other CM tools like os-config-applier, puppet, ansible,
> saltstack.
>
> Hopefully by then my POC will at least be able to create resources, if
> not deliver some data to servers.

We've been thinking about getting metadata to the in-instance parts on the
server and whether the resources you are building can serve the purpose.
I.e. pass and endpoint to the SoftwareConfig resources to the instance and
let the instance query the metadata from the resource. Sounds like this is
what you had in mind, so that would be a good point for integrating the
work. In the meantime, we can think of some shortcuts.

>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Steve Baker's message of 2013-11-19 13:06:21 -0800:
> On 11/20/2013 09:50 AM, Clint Byrum wrote:
> > Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> >> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> >> deleted it should trigger any remove_config and wait for the server to
> >> acknowledge when that is complete. This allows for any
> >> evacuation/deregistering workloads to be executed.
> >>
> > I'm a little worried about the road that leads us down. Most configuration
> > software defines forward progress only. Meaning, if you want something
> > not there, you don't remove it from your assertions, you assert that it
> > is not there.
> >
> > The reason this is different than the way we operate with resources is
> > that resources are all under Heat's direct control via well defined
> > APIs. In-instance things, however, will be indirectly controlled. So I
> > feel like focusing on a "diff" mechanism for user-deployed tools may be
> > unnecessary and might confuse. I'd much rather have a "converge"
> > mechanism for the users to focus on.
> >
> >
> A specific use-case I'm trying to address here is tripleo doing an
> update-replace on a nova compute node. The remove_config contains the
> workload to evacuate VMs and signal heat when the node is ready to be
> shut down. This is more involved than just "uninstall the things".
> 
> Could you outline in some more detail how you think this could be done?
> 

So for that we would not remove the software configuration for the
nova-compute, we would assert that the machine needs vms evacuated.
We want evacuation to be something we explicitly do, not a side effect
of deleting things. Perhaps having delete hooks for starting delete
work-flows is right, but it set off a red flag for me so I want to make
sure we think it through.

Also IIRC, evacuation is not necessarily an in-instance thing. It looks
more like the weird thing we've been talking about lately which is
"how do we orchestrate tenant API's":

https://etherpad.openstack.org/p/orchestrate-tenant-apis

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/20/2013 09:50 AM, Clint Byrum wrote:
> Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
>> Regarding apply_config/remove_config, if a SoftwareApplier resource is
>> deleted it should trigger any remove_config and wait for the server to
>> acknowledge when that is complete. This allows for any
>> evacuation/deregistering workloads to be executed.
>>
> I'm a little worried about the road that leads us down. Most configuration
> software defines forward progress only. Meaning, if you want something
> not there, you don't remove it from your assertions, you assert that it
> is not there.
>
> The reason this is different than the way we operate with resources is
> that resources are all under Heat's direct control via well defined
> APIs. In-instance things, however, will be indirectly controlled. So I
> feel like focusing on a "diff" mechanism for user-deployed tools may be
> unnecessary and might confuse. I'd much rather have a "converge"
> mechanism for the users to focus on.
>
>
A specific use-case I'm trying to address here is tripleo doing an
update-replace on a nova compute node. The remove_config contains the
workload to evacuate VMs and signal heat when the node is ready to be
shut down. This is more involved than just "uninstall the things".

Could you outline in some more detail how you think this could be done?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Steve Baker's message of 2013-11-18 12:52:04 -0800:
> 
> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> deleted it should trigger any remove_config and wait for the server to
> acknowledge when that is complete. This allows for any
> evacuation/deregistering workloads to be executed.
> 

I'm a little worried about the road that leads us down. Most configuration
software defines forward progress only. Meaning, if you want something
not there, you don't remove it from your assertions, you assert that it
is not there.

The reason this is different than the way we operate with resources is
that resources are all under Heat's direct control via well defined
APIs. In-instance things, however, will be indirectly controlled. So I
feel like focusing on a "diff" mechanism for user-deployed tools may be
unnecessary and might confuse. I'd much rather have a "converge"
mechanism for the users to focus on.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/19/2013 08:37 PM, Thomas Spatzier wrote:
> Steve Baker  wrote on 18.11.2013 21:52:04:
>> From: Steve Baker 
>> To: openstack-dev@lists.openstack.org,
>> Date: 18.11.2013 21:54
>> Subject: Re: [openstack-dev] [Heat] HOT software configuration
>> refined after design summit discussions
>>
>> On 11/19/2013 02:22 AM, Thomas Spatzier wrote:
>>> Hi all,
>>>
>>> I have reworked the wiki page [1] I created last week to reflect
>>> discussions we had on the mail list and in IRC. From ML discussions
> last
>>> week it looked like we were all basically on the same page (with some
>>> details to be worked out), and I hope the new draft eliminates some
>>> confusion that the original draft had.
>>>
>>> [1]
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
>> Thanks Thomas, this looks really good. I've actually started on a POC
>> which maps to this model.
> Good to hear that, Steve :-)
> Now that we are converging, should we consolidate the various wiki pages
> and just have one? E.g. copy the complete contents of
> hot-software-config-WIP to your original hot-software-config, or deprecate
> all others and make hot-software-config-WIP the master?
Lets just bless hot-software-config-WIP and add to it as we flesh out
the implementation.

>> I've used different semantics which you may actually prefer some of,
>> please comment below.
>>
>> Resource types:
>> SoftwareConfig -> SoftwareConfig (yay!)
>> SoftwareDeployment -> SoftwareApplier - less typing, less mouth-fatigue
> I'm ok with SoftwareApplier. If we don't hear objections, I can change it
> in the wiki.
>
>> SoftwareConfig properties:
>> parameters -> inputs - just because parameters is overloaded already.
> Makes sense.
>
>> Although if the CM tool has their own semantics for inputs then that
>> should be used in that SoftwareConfig resource implementation instead.
>> outputs -> outputs
>>
>> SoftwareApplier properties:
>> software_config -> apply_config - because there will sometimes be a
>> corresponding remove_config
> Makes sense, and the remove_config thought is a very good point!
>
>> server -> server
>> parameters -> input_values - to match the 'inputs' schema property in
>> SoftwareConfig
> Agree on input_values.
>
>> Other comments on hot-software-config-WIP:
>>
>> Regarding apply_config/remove_config, if a SoftwareApplier resource is
>> deleted it should trigger any remove_config and wait for the server to
>> acknowledge when that is complete. This allows for any
>> evacuation/deregistering workloads to be executed.
>>
>> I'm unclear yet what the SoftwareConfig 'role' is for, unless the role
>> specifies the contract for a given inputs and outputs schema? How would
>> this be documented or enforced? I'm inclined to leave it out for now.
> So about 'role', as I stated in the wiki, my thinking was that there will
> be different SoftwareConfig and SoftwareApplier implementations per CM tool
> (more on that below), since all CM tools will probably have their specific
> metadata and runtime implementation. So in my example I was using Chef, and
> 'role' is just a Chef concept, i.e. you take a cookbook and configure a
> specific Chef role on a server.
OK, its Chef specific; I'm fine with that.
>> It should be possible to write a SoftwareConfig type for a new CM tool
>> as a provider template. This has some nice implications for deployers
>> and users.
> I think provider templates are a good thing to have clean componentization
> for re-use. However, I think it still would be good to allow users to
> define their SoftwareConfigs inline in a template for simple use cases. I
> heard that requirement in several posts on the ML last week.
> The question is whether we can live with a single implementation of
> SoftwareConfig and SoftwareApplier then (see also below).
Yes, a provider template would encapsulate some base SoftwareConfig
resource type, but users would be free to use this type inline in their
template too.
>> My hope is that there will not need to be a different SoftwareApplier
>> type written for each CM tool. But maybe there will be one for each
>> delivery mechanism. The first implementation will use metadata polling
>> and signals, another might use Marconi. Bootstrapping an image to
>> consume a given CM tool and applied configuration data is something that
>> we need to do, but we can make it beyond the scope of this particular
>> proposal.
> I was thinking about a single im

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Steve Baker  wrote on 18.11.2013 21:52:04:
> From: Steve Baker 
> To: openstack-dev@lists.openstack.org,
> Date: 18.11.2013 21:54
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> On 11/19/2013 02:22 AM, Thomas Spatzier wrote:
> > Hi all,
> >
> > I have reworked the wiki page [1] I created last week to reflect
> > discussions we had on the mail list and in IRC. From ML discussions
last
> > week it looked like we were all basically on the same page (with some
> > details to be worked out), and I hope the new draft eliminates some
> > confusion that the original draft had.
> >
> > [1]
https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >
> Thanks Thomas, this looks really good. I've actually started on a POC
> which maps to this model.

Good to hear that, Steve :-)
Now that we are converging, should we consolidate the various wiki pages
and just have one? E.g. copy the complete contents of
hot-software-config-WIP to your original hot-software-config, or deprecate
all others and make hot-software-config-WIP the master?

>
> I've used different semantics which you may actually prefer some of,
> please comment below.
>
> Resource types:
> SoftwareConfig -> SoftwareConfig (yay!)
> SoftwareDeployment -> SoftwareApplier - less typing, less mouth-fatigue

I'm ok with SoftwareApplier. If we don't hear objections, I can change it
in the wiki.

>
> SoftwareConfig properties:
> parameters -> inputs - just because parameters is overloaded already.

Makes sense.

> Although if the CM tool has their own semantics for inputs then that
> should be used in that SoftwareConfig resource implementation instead.
> outputs -> outputs
>
> SoftwareApplier properties:
> software_config -> apply_config - because there will sometimes be a
> corresponding remove_config

Makes sense, and the remove_config thought is a very good point!

> server -> server
> parameters -> input_values - to match the 'inputs' schema property in
> SoftwareConfig

Agree on input_values.

>
> Other comments on hot-software-config-WIP:
>
> Regarding apply_config/remove_config, if a SoftwareApplier resource is
> deleted it should trigger any remove_config and wait for the server to
> acknowledge when that is complete. This allows for any
> evacuation/deregistering workloads to be executed.
>
> I'm unclear yet what the SoftwareConfig 'role' is for, unless the role
> specifies the contract for a given inputs and outputs schema? How would
> this be documented or enforced? I'm inclined to leave it out for now.

So about 'role', as I stated in the wiki, my thinking was that there will
be different SoftwareConfig and SoftwareApplier implementations per CM tool
(more on that below), since all CM tools will probably have their specific
metadata and runtime implementation. So in my example I was using Chef, and
'role' is just a Chef concept, i.e. you take a cookbook and configure a
specific Chef role on a server.

>
> It should be possible to write a SoftwareConfig type for a new CM tool
> as a provider template. This has some nice implications for deployers
> and users.

I think provider templates are a good thing to have clean componentization
for re-use. However, I think it still would be good to allow users to
define their SoftwareConfigs inline in a template for simple use cases. I
heard that requirement in several posts on the ML last week.
The question is whether we can live with a single implementation of
SoftwareConfig and SoftwareApplier then (see also below).

>
> My hope is that there will not need to be a different SoftwareApplier
> type written for each CM tool. But maybe there will be one for each
> delivery mechanism. The first implementation will use metadata polling
> and signals, another might use Marconi. Bootstrapping an image to
> consume a given CM tool and applied configuration data is something that
> we need to do, but we can make it beyond the scope of this particular
> proposal.

I was thinking about a single implementation, too. However, I cannot really
imagine how one single implementation could handle both the different
metadata of different CM tools, and different runtime implementation. I
think we would want to support at least a handful of the most favorite
tools, but cannot see at the moment how to cover them all in one
implementation. My thought was that there could be a super-class for common
behavior, and then plugins with specific behavior for each tool.

Anyway, all of that needs to be verified, so working on PoC patches is
definitely the right thing to do. For example, if we work on implementation
for two CM tools (e.g. Chef and simple scripts)

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/19/2013 02:22 AM, Thomas Spatzier wrote:
> Hi all,
>
> I have reworked the wiki page [1] I created last week to reflect
> discussions we had on the mail list and in IRC. From ML discussions last
> week it looked like we were all basically on the same page (with some
> details to be worked out), and I hope the new draft eliminates some
> confusion that the original draft had.
>
> [1] https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
>
Thanks Thomas, this looks really good. I've actually started on a POC
which maps to this model.

I've used different semantics which you may actually prefer some of,
please comment below.

Resource types:
SoftwareConfig -> SoftwareConfig (yay!)
SoftwareDeployment -> SoftwareApplier - less typing, less mouth-fatigue

SoftwareConfig properties:
parameters -> inputs - just because parameters is overloaded already.
Although if the CM tool has their own semantics for inputs then that
should be used in that SoftwareConfig resource implementation instead.
outputs -> outputs

SoftwareApplier properties:
software_config -> apply_config - because there will sometimes be a
corresponding remove_config
server -> server
parameters -> input_values - to match the 'inputs' schema property in
SoftwareConfig

Other comments on hot-software-config-WIP:

Regarding apply_config/remove_config, if a SoftwareApplier resource is
deleted it should trigger any remove_config and wait for the server to
acknowledge when that is complete. This allows for any
evacuation/deregistering workloads to be executed.

I'm unclear yet what the SoftwareConfig 'role' is for, unless the role
specifies the contract for a given inputs and outputs schema? How would
this be documented or enforced? I'm inclined to leave it out for now.

It should be possible to write a SoftwareConfig type for a new CM tool
as a provider template. This has some nice implications for deployers
and users.

My hope is that there will not need to be a different SoftwareApplier
type written for each CM tool. But maybe there will be one for each
delivery mechanism. The first implementation will use metadata polling
and signals, another might use Marconi. Bootstrapping an image to
consume a given CM tool and applied configuration data is something that
we need to do, but we can make it beyond the scope of this particular
proposal.

The POC I'm working on is actually backed by a REST API which does dumb
(but structured) storage of SoftwareConfig and SoftwareApplier entities.
This has some interesting implications for managing SoftwareConfig
resources outside the context of the stack which uses them, but lets not
worry too much about that *yet*.

cheers

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Hi all,

I have reworked the wiki page [1] I created last week to reflect
discussions we had on the mail list and in IRC. From ML discussions last
week it looked like we were all basically on the same page (with some
details to be worked out), and I hope the new draft eliminates some
confusion that the original draft had.

[1] https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP

Regards,
Thomas


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

It seems to me we have been discussing a proposal whose write-up 
intertwines two ideas: (1) making software components look like resources, 
and (2) using nested stacks and environments to achieve the pattern of 
definitions and uses.  The ideas are separable, and I think the discussion 
has sort of revealed that.  There are issues with both.  Regarding the 
first, I think Clint has been pursuing the critical issue.

Clint Byrum  wrote on 11/12/2013 07:34:34 PM:
> ...
> > >I have implementation questions about both of these approaches 
though,
> > >as it appears they'd have to reach backward in the graph to insert
> > >their configuration, or have a generic bucket for all configuration
> > 
> > Yeah, it does depend on the implementation. If we use Mistral the
> > agent will need to ask Mistral for the tasks that apply to the server.
> > 
> > $ mistral task-consume \
> >   --tags=instance_id=$(my_instance_id);stack_id=$(stack_id)
> > 
> 
> Actually that makes perfect sense. Thanks. If we have a hidden work-flow
> handle for any resources that have need of it passed in much the same
> way we pass in the cfn-metadata-server now, that would allow us to write
> our work-flow afterward. The image can decide when it wants to start
> doing work-flows and can decide to just spin until the work-flow exists
> and is enabled. For the tiny-work-flow case of one task it works the
> same as the complicated work-flow, so I think this sounds like a
> workable plan.. assuming a work-flow service. :)

If I understand correctly, Clint is dismayed at the idea of a heat engine 
plugin "reaching backward" to find the software configs needed to apply to 
a server, but is OK with a task in a taskflow doing that.  It seems like a 
double standard to me.  And it makes the software config stuff depend on 
an essentially independent big change, which is dismaying to those of us 
eager to make progress on software config.  How about allowing a Heat 
plugin to query for dependents in the graph?

Regarding issue (2), look at where we are going.  If I write a template 
that I want to share with you, and that template exercises the 
definition/use distinction, I have to give you: (i) the templates that 
comprise my definitions, (ii) my template that has my uses, and (iii) a 
*fragment* of environment that binds the templates in (i) to the names 
used for them in (ii).  This is a pretty ragged assembly of stuff.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/14/2013 06:02 AM, Zane Bitter wrote:
> On 13/11/13 01:34, Clint Byrum wrote:
>> Excerpts from Angus Salkeld's message of 2013-11-12 15:22:44 -0800:
>>> >IMO is should just be a template/formatted file.
>>> >
>> I'd prefer that we have the ability to pull in a chunk of in-line
>> template
>> as well. Perhaps that is the template resource, I have not thought that
>> through. It is not o-k, IMO, to push things off entirely to external
>> files/urls/"providers", etc. That is just cumbersome and unnecessary for
>> a common case which is to deploy two things using the same base config
>> with parameters having different values.
>>
>> Of course, for my use case of having different topologies reusing bits
>> of config, it is perfect to have the reusable bits split into different
>> files.
>
> So, if I understand Angus's get_file suggestion correctly, it parses
> out to the equivalent of inlining the file's contents. So if you
> implement the resource as accepting inline data and add in get_file,
> then you get:
>   a) Composability, OR
>   b) Everything in one file
>
> but not both. I think that is probably sufficient, but I would be
> interested in your opinion: is it essential that you be able to
> compose software components defined in the same file?
>
> Note that the implementation of get_file would also involve
> python-heatclient automagically detecting it and making sure the
> relevant file is uploaded in the files section. So this shouldn't
> create a lot of mental overhead for the user.
>
> (BTW I think I like this plan.)
>
Yes, and get_file is a HOT function which is only evaluated where other
functions are evaluated, which is probably a good thing.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steve Baker]

On 11/14/2013 06:11 AM, Zane Bitter wrote:
> On 11/11/13 17:57, Thomas Spatzier wrote:
>>
>> Hi all,
>>
>> I have just posted the following wiki page to reflect a refined proposal
>> for HOT software configuration based on discussions at the design summit
>> last week. Angus also put a sample up in an etherpad last week, but
>> we did
>> not have enough time to go thru it in the design session. My write-up is
>> based on Angus' sample, actually a refinement, and on discussions we
>> had in
>> breaks, plus it is trying to reflect all the good input from ML
>> discussions
>> and Steve Baker's initial proposal.
>>
>> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
>>
>> Please review and provide feedback.
>
> I believe there's an error in the Explicit dependency section, where
> it says that depends_on is a property. In cfn DependsOn actually
> exists at the same level as Type, Properties, &c.
>
> resources:
>   client:
> type: My::Software::SomeClient
> properties:
>   server: { get_resource: my_server }
>   params:
> # params ...
> depends_on:
>   - get_resource: server_process1
>   - get_resource: server_process2
>
> And conceptually this seems correct, because it applies to any kind of
> resource, whereas properties are defined per-resource-type.
>
> Don't be fooled by our implementation:
> https://review.openstack.org/#/c/44733/
>
> It also doesn't support a list, but I think we can and should fix that
> in HOT.
This has already been fixed
https://review.openstack.org/#/c/51507/1
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Zane Bitter]

On 13/11/13 18:29, Thomas Spatzier wrote:

It also doesn't support a list, but I think we can and should fix that
>in HOT.

Doesn't DependsOn already support lists? I quickly checked the code and it
seems it does:
https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L288


Oh, cool. Looks like Angus added that last month. Thanks, I missed that 
one :)


- ZB

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Zane Bitter  wrote on 13.11.2013 18:11:18:
> From: Zane Bitter 
> To: openstack-dev@lists.openstack.org,
> Date: 13.11.2013 18:14
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> On 11/11/13 17:57, Thomas Spatzier wrote:

> >
> > https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >
> > Please review and provide feedback.
>
> I believe there's an error in the Explicit dependency section, where it
> says that depends_on is a property. In cfn DependsOn actually exists at
> the same level as Type, Properties, &c.
>
> resources:
>client:
>  type: My::Software::SomeClient
>  properties:
>server: { get_resource: my_server }
>params:
>  # params ...
>  depends_on:
>- get_resource: server_process1
>- get_resource: server_process2

Good point. I think the reason was tied too much to the provider template
concept where all properties get passed automatically to the provider
template and in there you can basically do anything that is necessary,
including hanlding dependencies. But I was missing the fact that this is a
generic concept for all resources.
I'll fix it in the wiki.

>
> And conceptually this seems correct, because it applies to any kind of
> resource, whereas properties are defined per-resource-type.
>
> Don't be fooled by our implementation:
> https://review.openstack.org/#/c/44733/
>
> It also doesn't support a list, but I think we can and should fix that
> in HOT.

Doesn't DependsOn already support lists? I quickly checked the code and it
seems it does:
https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L288

>
> cheers,
> Zane.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Zane Bitter]

On 11/11/13 17:57, Thomas Spatzier wrote:


Hi all,

I have just posted the following wiki page to reflect a refined proposal
for HOT software configuration based on discussions at the design summit
last week. Angus also put a sample up in an etherpad last week, but we did
not have enough time to go thru it in the design session. My write-up is
based on Angus' sample, actually a refinement, and on discussions we had in
breaks, plus it is trying to reflect all the good input from ML discussions
and Steve Baker's initial proposal.

https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP

Please review and provide feedback.


I believe there's an error in the Explicit dependency section, where it 
says that depends_on is a property. In cfn DependsOn actually exists at 
the same level as Type, Properties, &c.


resources:
  client:
type: My::Software::SomeClient
properties:
  server: { get_resource: my_server }
  params:
# params ...
depends_on:
  - get_resource: server_process1
  - get_resource: server_process2

And conceptually this seems correct, because it applies to any kind of 
resource, whereas properties are defined per-resource-type.


Don't be fooled by our implementation: 
https://review.openstack.org/#/c/44733/


It also doesn't support a list, but I think we can and should fix that 
in HOT.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Zane Bitter]

On 13/11/13 01:34, Clint Byrum wrote:

Excerpts from Angus Salkeld's message of 2013-11-12 15:22:44 -0800:

>IMO is should just be a template/formatted file.
>

I'd prefer that we have the ability to pull in a chunk of in-line template
as well. Perhaps that is the template resource, I have not thought that
through. It is not o-k, IMO, to push things off entirely to external
files/urls/"providers", etc. That is just cumbersome and unnecessary for
a common case which is to deploy two things using the same base config
with parameters having different values.

Of course, for my use case of having different topologies reusing bits
of config, it is perfect to have the reusable bits split into different
files.


So, if I understand Angus's get_file suggestion correctly, it parses out 
to the equivalent of inlining the file's contents. So if you implement 
the resource as accepting inline data and add in get_file, then you get:

  a) Composability, OR
  b) Everything in one file

but not both. I think that is probably sufficient, but I would be 
interested in your opinion: is it essential that you be able to compose 
software components defined in the same file?


Note that the implementation of get_file would also involve 
python-heatclient automagically detecting it and making sure the 
relevant file is uploaded in the files section. So this shouldn't create 
a lot of mental overhead for the user.


(BTW I think I like this plan.)

cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Excerpts from Clint Byrum's message on 13.11.2013 09:59:56:
> From: Clint Byrum 
> To: openstack-dev ,
> Date: 13.11.2013 10:03
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> Excerpts from Thomas Spatzier's message of 2013-11-13 00:28:59 -0800:
> > Angus Salkeld  wrote on 13.11.2013 00:22:44:

> > >
> > >
> > > This is what Thomas defined, with one optimisation.
> > > - The webconfig is a yaml template.
> > >
> > > As you say the component is static - if so why even put it inline in
> > > the template (well that was my thinking, it seems like a template not
> > > really a resource).
> >
> > Yes, exactly. Our idea was to put it in its own file since it is really
> > static and having it in its own file makes it much more reusable.
> > With 'WebConfig' defined inline in the template as in the snippet
above,
> > you will have to update many template files where you use the
component,
> > whereas you will only have to touch one place when it is in its own
file.
> > Ok, the example above looks simple, but in reality we will see more
complex
> > sets of parameters etc.
> > Maybe for very simple use cases, we can allow a shortcut of inlining it
in
> > the template (I mentioned this in the wiki) and avoid the need for a
> > separate file.
> >
>
> I think I understand now, and we're all basically on the same page. As
> usual, I was confused by the subtleties.

Yeah, I think we are all on the same page, just sorting out the details on
how to express the same thing.
Maybe we can still work a bit on syntax details to make it more intuitive.

>
> I think the in-line capability is critical to have in the near-term
> plan, but would +2 an implementation that left it out at the beginning.

I agree to have the inline version as a shortcut, and I think we can have
if for free with what is on the wiki page.
You could basically just use the OS::Heat::ChefSoftwareComponent resource
in the separate template file directly in the template that contains the
server. Maybe we will need this additional association class then if people
have problems with having a pointer to the server from the component.

>
> Before we "ratify" this and people run off and write code, I'd like to
> present my "problems" in TripleO and try to see if I can express them
> using the spec you've laid out. Will try and do that in the next couple
> of days.

Yes, that would be great. If you can share your thought experiment on an
etherpad or something, I could also try to refine the proposal if you hit
cases where it does not work.

Maybe we can also get together after the IRC meeting today and have some
discussion with Angus, Steven, Steve etc.

>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Thomas Spatzier's message of 2013-11-13 00:28:59 -0800:
> Angus Salkeld  wrote on 13.11.2013 00:22:44:
> > From: Angus Salkeld 
> > To: openstack-dev@lists.openstack.org,
> > Date: 13.11.2013 00:25
> > Subject: Re: [openstack-dev] [Heat] HOT software configuration
> > refined after design summit discussions
> >
> > On 12/11/13 10:32 -0800, Clint Byrum wrote:
> > >Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> > >>
> > >> Hi all,
> > >>
> > >> I have just posted the following wiki page to reflect a refined
> proposal
> > >> for HOT software configuration based on discussions at the design
> summit
> > >> last week. Angus also put a sample up in an etherpad last week, but we
> did
> > >> not have enough time to go thru it in the design session. My write-up
> is
> > >> based on Angus' sample, actually a refinement, and on discussionswe
> had in
> > >> breaks, plus it is trying to reflect all the good input from ML
> discussions
> > >> and Steve Baker's initial proposal.
> > >>
> > >>
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> > >>
> > >> Please review and provide feedback.
> > >
> > >Hi Thomas, thanks for spelling this out clearly.
> > >
> > >I am still -1 on anything that specifies the place a configuration is
> > >hosted inside the configuration definition itself. Because
> configurations
> > >are encapsulated by servers, it makes more sense to me that the servers
> > >(or server groups) would specify their configurations. If changing to a
> > >more logical model is just too hard for TOSCA to adapt to, then I
> suggest
> > >this be an area that TOSCA differs from Heat. We don't need two models
> > >for communicating configurations to servers, and I'd prefer Heat stay
> > >focused on making HOT template authors' and users' lives better.
> > >
> > >I have seen an alternative approach which separates a configuration
> > >definition from a configuration deployer. This at least makes it clear
> > >that the configuration is a part of a server. In pseudo-HOT:
> > >
> > >resources:
> > >  WebConfig:
> > >type: OS::Heat::ChefCookbook
> > >properties:
> > >  cookbook_url: https://some.test/foo
> > >  parameters:
> > >endpoint_host:
> > >  type: string
> > >  WebServer:
> > >type: OS::Nova::Server
> > >properties:
> > >  image: webserver
> > >  flavor: 100
> > >  DeployWebConfig:
> > >type: OS::Heat::ConfigDeployer
> > >properties:
> > >  configuration: {get_resource: WebConfig}
> > >  on_server: {get_resource: WebServer}
> > >  parameters:
> > >endpoint_host: {get_attribute: [ WebServer, first_ip]}
> >
> >
> > This is what Thomas defined, with one optimisation.
> > - The webconfig is a yaml template.
> >
> > As you say the component is static - if so why even put it inline in
> > the template (well that was my thinking, it seems like a template not
> > really a resource).
> 
> Yes, exactly. Our idea was to put it in its own file since it is really
> static and having it in its own file makes it much more reusable.
> With 'WebConfig' defined inline in the template as in the snippet above,
> you will have to update many template files where you use the component,
> whereas you will only have to touch one place when it is in its own file.
> Ok, the example above looks simple, but in reality we will see more complex
> sets of parameters etc.
> Maybe for very simple use cases, we can allow a shortcut of inlining it in
> the template (I mentioned this in the wiki) and avoid the need for a
> separate file.
> 

I think I understand now, and we're all basically on the same page. As
usual, I was confused by the subtleties.

I think the in-line capability is critical to have in the near-term
plan, but would +2 an implementation that left it out at the beginning.

Before we "ratify" this and people run off and write code, I'd like to
present my "problems" in TripleO and try to see if I can express them
using the spec you've laid out. Will try and do that in the next couple
of days.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Angus Salkeld  wrote on 13.11.2013 00:22:44:
> From: Angus Salkeld 
> To: openstack-dev@lists.openstack.org,
> Date: 13.11.2013 00:25
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> On 12/11/13 10:32 -0800, Clint Byrum wrote:
> >Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> >>
> >> Hi all,
> >>
> >> I have just posted the following wiki page to reflect a refined
proposal
> >> for HOT software configuration based on discussions at the design
summit
> >> last week. Angus also put a sample up in an etherpad last week, but we
did
> >> not have enough time to go thru it in the design session. My write-up
is
> >> based on Angus' sample, actually a refinement, and on discussionswe
had in
> >> breaks, plus it is trying to reflect all the good input from ML
discussions
> >> and Steve Baker's initial proposal.
> >>
> >>
https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >>
> >> Please review and provide feedback.
> >
> >Hi Thomas, thanks for spelling this out clearly.
> >
> >I am still -1 on anything that specifies the place a configuration is
> >hosted inside the configuration definition itself. Because
configurations
> >are encapsulated by servers, it makes more sense to me that the servers
> >(or server groups) would specify their configurations. If changing to a
> >more logical model is just too hard for TOSCA to adapt to, then I
suggest
> >this be an area that TOSCA differs from Heat. We don't need two models
> >for communicating configurations to servers, and I'd prefer Heat stay
> >focused on making HOT template authors' and users' lives better.
> >
> >I have seen an alternative approach which separates a configuration
> >definition from a configuration deployer. This at least makes it clear
> >that the configuration is a part of a server. In pseudo-HOT:
> >
> >resources:
> >  WebConfig:
> >type: OS::Heat::ChefCookbook
> >properties:
> >  cookbook_url: https://some.test/foo
> >  parameters:
> >endpoint_host:
> >  type: string
> >  WebServer:
> >type: OS::Nova::Server
> >properties:
> >  image: webserver
> >  flavor: 100
> >  DeployWebConfig:
> >type: OS::Heat::ConfigDeployer
> >properties:
> >  configuration: {get_resource: WebConfig}
> >  on_server: {get_resource: WebServer}
> >  parameters:
> >endpoint_host: {get_attribute: [ WebServer, first_ip]}
>
>
> This is what Thomas defined, with one optimisation.
> - The webconfig is a yaml template.
>
> As you say the component is static - if so why even put it inline in
> the template (well that was my thinking, it seems like a template not
> really a resource).

Yes, exactly. Our idea was to put it in its own file since it is really
static and having it in its own file makes it much more reusable.
With 'WebConfig' defined inline in the template as in the snippet above,
you will have to update many template files where you use the component,
whereas you will only have to touch one place when it is in its own file.
Ok, the example above looks simple, but in reality we will see more complex
sets of parameters etc.
Maybe for very simple use cases, we can allow a shortcut of inlining it in
the template (I mentioned this in the wiki) and avoid the need for a
separate file.

>
> >
> >I have implementation questions about both of these approaches though,
> >as it appears they'd have to reach backward in the graph to insert
> >their configuration, or have a generic bucket for all configuration
>
> Yeah, it does depend on the implementation. If we use Mistral the
> agent will need to ask Mistral for the tasks that apply to the server.
>
> $ mistral task-consume \
>   --tags=instance_id=$(my_instance_id);stack_id=$(stack_id)
>
>
> >to be inserted. IMO that would look a lot like the method I proposed,
> >which was to just have a list of components attached directly to the
> >server like this:
> >
> >components:
> >  WebConfig:
> >type: Chef::Cookbook
> >properties:
> >  cookbook_url: https://some.test/foo
> >  parameters:
> >endpoing_host:
> >  type: string
> >resources:
> >  WebServer:
> >type: OS::Nova::Server
> >properties:
> >  image: webserver
> >  flavor: 100
> >components:
> >  - webconfig:
> >component: {get

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Georgy Okrokvertskhov  wrote on 12.11.2013
21:27:13:
> From: Georgy Okrokvertskhov 
> To: "OpenStack Development Mailing List (not for usage questions)"
> ,
> Date: 12.11.2013 21:29
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> Hi,
>
> I agree with Clint that component placement specified inside
> component configuration is not a right thing. I remember that mostly
> everyone agreed that "hosted_on" should not be in HOT templates.
> When one specify placement explicitly inside  a component definition
> it prevents the following:
> 1. Reusability - you can't reuse component without creating its
> definition copy with another placement parameter.

See my reply to Clint's mail. The deployment location in form of the
"server" reference is _not_ hardcoded in the component definition. All we
do is to provide a pointer to the server where a software shall be deployed
at deploy time. You can use a component definition in many place, and in
each place where you use it you provide it a pointer to the target server.

> 2. Composability - it will be no clear way to express composable
> configurations. There was a clear way in a template showed during
> design session where server had a list of components to be placed.

I think we have full composability with the "deployment" resources that
mark uses of software component definitions.

> 3. Deployment order - some components should be placed in strict
> order and it will be much easier just make an ordered list of
> components then express artificial dependencies between them just
> for ordering.

With the "deployment" resources and Heat's normal way of handling
dependencies between any resources, we should be able have proper ordering.
I agree that strict ordering is probably the most easy way of doing it, but
we have implementations that do deployment in a more flexible manner
without any problems.

>
> Thanks
> Georgy
>

> On Tue, Nov 12, 2013 at 10:32 AM, Clint Byrum  wrote:
> Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> >
> > Hi all,
> >
> > I have just posted the following wiki page to reflect a refined
proposal
> > for HOT software configuration based on discussions at the design
summit
> > last week. Angus also put a sample up in an etherpad last week, but we
did
> > not have enough time to go thru it in the design session. My write-up
is
> > based on Angus' sample, actually a refinement, and on discussions we
had in
> > breaks, plus it is trying to reflect all the good input from ML
discussions
> > and Steve Baker's initial proposal.
> >
> > https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >
> > Please review and provide feedback.

> Hi Thomas, thanks for spelling this out clearly.
>
> I am still -1 on anything that specifies the place a configuration is
> hosted inside the configuration definition itself. Because configurations
> are encapsulated by servers, it makes more sense to me that the servers
> (or server groups) would specify their configurations. If changing to a
> more logical model is just too hard for TOSCA to adapt to, then I suggest
> this be an area that TOSCA differs from Heat. We don't need two models
> for communicating configurations to servers, and I'd prefer Heat stay
> focused on making HOT template authors' and users' lives better.
>
> I have seen an alternative approach which separates a configuration
> definition from a configuration deployer. This at least makes it clear
> that the configuration is a part of a server. In pseudo-HOT:
>
> resources:
>   WebConfig:
>     type: OS::Heat::ChefCookbook
>     properties:
>       cookbook_url: https://some.test/foo
>       parameters:
>         endpoint_host:
>           type: string
>   WebServer:
>     type: OS::Nova::Server
>     properties:
>       image: webserver
>       flavor: 100
>   DeployWebConfig:
>     type: OS::Heat::ConfigDeployer
>     properties:
>       configuration: {get_resource: WebConfig}
>       on_server: {get_resource: WebServer}
>       parameters:
>         endpoint_host: {get_attribute: [ WebServer, first_ip]}
>
> I have implementation questions about both of these approaches though,
> as it appears they'd have to reach backward in the graph to insert
> their configuration, or have a generic bucket for all configuration
> to be inserted. IMO that would look a lot like the method I proposed,
> which was to just have a list of components attached directly to the
> server like this:
>
> components:
>   WebConfig:
>     type: Chef::Cookbook
&

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Thomas Spatzier]

Excerpts form Clint Byrum's message on 12.11.2013 19:32:50:
> From: Clint Byrum 
> To: openstack-dev ,
> Date: 12.11.2013 19:35
> Subject: Re: [openstack-dev] [Heat] HOT software configuration
> refined after design summit discussions
>
> Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> >
> > Hi all,
> >
> > I have just posted the following wiki page to reflect a refined
proposal

> Hi Thomas, thanks for spelling this out clearly.
>
> I am still -1 on anything that specifies the place a configuration is
> hosted inside the configuration definition itself. Because configurations
> are encapsulated by servers, it makes more sense to me that the servers
> (or server groups) would specify their configurations. If changing to a

IMO the current proposal does _not_ the concrete hosting inside component
definition. The component definition is in this external template file and
all we do is give it a pointer to the server at deploy time so that the
implementation can perform whatever is needed to perform at that time.
The resource in the actual template file is like the intermediate
association resource you are suggesting below (similar to what
VolumeAttachment does), so this is the place where you say which component
gets deployed where. This represents a concrete use of a software
component. Again, all we do is pass in a pointer to the server where _this
use_ of the software component shall be installed.

> more logical model is just too hard for TOSCA to adapt to, then I suggest
> this be an area that TOSCA differs from Heat. We don't need two models

The current proposal was done completely unrelated to TOSCA, but really
just a try to have a pragmatic approach for solving the use cases we talked
about. I don't really care in which directions the relations point. Both
ways can be easily mapped to TOSCA. I just think the current proposal is
intuitive, at least to me. And you could see it as kind of a short notation
that avoids another association class.

> for communicating configurations to servers, and I'd prefer Heat stay
> focused on making HOT template authors' and users' lives better.
>
> I have seen an alternative approach which separates a configuration
> definition from a configuration deployer. This at least makes it clear
> that the configuration is a part of a server. In pseudo-HOT:
>
> resources:
>   WebConfig:
> type: OS::Heat::ChefCookbook
> properties:
>   cookbook_url: https://some.test/foo
>   parameters:
> endpoint_host:
>   type: string
>   WebServer:
> type: OS::Nova::Server
> properties:
>   image: webserver
>   flavor: 100
>   DeployWebConfig:
> type: OS::Heat::ConfigDeployer
> properties:
>   configuration: {get_resource: WebConfig}
>   on_server: {get_resource: WebServer}
>   parameters:
> endpoint_host: {get_attribute: [ WebServer, first_ip]}

The DeployWebConfig association class actually is the 'mysql' resource in
the template on the wiki page. See the Design alternative section I put it.
That would be fine with me as well.

>



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Angus Salkeld's message of 2013-11-12 15:22:44 -0800:
> On 12/11/13 10:32 -0800, Clint Byrum wrote:
> >Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> >>
> >> Hi all,
> >>
> >> I have just posted the following wiki page to reflect a refined proposal
> >> for HOT software configuration based on discussions at the design summit
> >> last week. Angus also put a sample up in an etherpad last week, but we did
> >> not have enough time to go thru it in the design session. My write-up is
> >> based on Angus' sample, actually a refinement, and on discussions we had in
> >> breaks, plus it is trying to reflect all the good input from ML discussions
> >> and Steve Baker's initial proposal.
> >>
> >> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >>
> >> Please review and provide feedback.
> >
> >Hi Thomas, thanks for spelling this out clearly.
> >
> >I am still -1 on anything that specifies the place a configuration is
> >hosted inside the configuration definition itself. Because configurations
> >are encapsulated by servers, it makes more sense to me that the servers
> >(or server groups) would specify their configurations. If changing to a
> >more logical model is just too hard for TOSCA to adapt to, then I suggest
> >this be an area that TOSCA differs from Heat. We don't need two models
> >for communicating configurations to servers, and I'd prefer Heat stay
> >focused on making HOT template authors' and users' lives better.
> >
> >I have seen an alternative approach which separates a configuration
> >definition from a configuration deployer. This at least makes it clear
> >that the configuration is a part of a server. In pseudo-HOT:
> >
> >resources:
> >  WebConfig:
> >type: OS::Heat::ChefCookbook
> >properties:
> >  cookbook_url: https://some.test/foo
> >  parameters:
> >endpoint_host:
> >  type: string
> >  WebServer:
> >type: OS::Nova::Server
> >properties:
> >  image: webserver
> >  flavor: 100
> >  DeployWebConfig:
> >type: OS::Heat::ConfigDeployer
> >properties:
> >  configuration: {get_resource: WebConfig}
> >  on_server: {get_resource: WebServer}
> >  parameters:
> >endpoint_host: {get_attribute: [ WebServer, first_ip]}
> 
> 
> This is what Thomas defined, with one optimisation.
> - The webconfig is a yaml template.
> 
> As you say the component is static - if so why even put it inline in
> the template (well that was my thinking, it seems like a template not
> really a resource).
> 

Doing it in-line or pulled in via some other method, I think making
things composable calls for both.

> >
> >I have implementation questions about both of these approaches though,
> >as it appears they'd have to reach backward in the graph to insert
> >their configuration, or have a generic bucket for all configuration
> 
> Yeah, it does depend on the implementation. If we use Mistral the
> agent will need to ask Mistral for the tasks that apply to the server.
> 
> $ mistral task-consume \
>   --tags=instance_id=$(my_instance_id);stack_id=$(stack_id)
> 

Actually that makes perfect sense. Thanks. If we have a hidden work-flow
handle for any resources that have need of it passed in much the same
way we pass in the cfn-metadata-server now, that would allow us to write
our work-flow afterward. The image can decide when it wants to start
doing work-flows and can decide to just spin until the work-flow exists
and is enabled. For the tiny-work-flow case of one task it works the
same as the complicated work-flow, so I think this sounds like a
workable plan.. assuming a work-flow service. :)

> >to be inserted. IMO that would look a lot like the method I proposed,
> >which was to just have a list of components attached directly to the
> >server like this:
> >
> >components:
> >  WebConfig:
> >type: Chef::Cookbook
> >properties:
> >  cookbook_url: https://some.test/foo
> >  parameters:
> >endpoing_host:
> >  type: string
> >resources:
> >  WebServer:
> >type: OS::Nova::Server
> >properties:
> >  image: webserver
> >  flavor: 100
> >components:
> >  - webconfig:
> >component: {get_component: WebConfig}
> >parameters:
> >  endpoint_host: {get_attribute: [ WebServer, first_ip ]}
> I'd change this to:
> 
>  components:
>- webconfig:
>  component: {get_file: ./my_configs/webconfig.yaml}
>  parameters:
>endpoint_host: {get_attribute: [ WebServer, first_ip ]}
> 
> This *could* be a short hand notation like the volumes property on
> aws instances.
> 

That makes sense.

> >
> >Of course, the keen eye will see the circular dependency there with the
> >WebServer trying to know its own IP. We've identified quite a few use
> >cases for self-referencing attributes, so that is a separate problem we
> >should solve independent of the template composition problem.
> 
> (aside) I don't like t

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Angus Salkeld]

On 12/11/13 10:32 -0800, Clint Byrum wrote:

Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:


Hi all,

I have just posted the following wiki page to reflect a refined proposal
for HOT software configuration based on discussions at the design summit
last week. Angus also put a sample up in an etherpad last week, but we did
not have enough time to go thru it in the design session. My write-up is
based on Angus' sample, actually a refinement, and on discussions we had in
breaks, plus it is trying to reflect all the good input from ML discussions
and Steve Baker's initial proposal.

https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP

Please review and provide feedback.


Hi Thomas, thanks for spelling this out clearly.

I am still -1 on anything that specifies the place a configuration is
hosted inside the configuration definition itself. Because configurations
are encapsulated by servers, it makes more sense to me that the servers
(or server groups) would specify their configurations. If changing to a
more logical model is just too hard for TOSCA to adapt to, then I suggest
this be an area that TOSCA differs from Heat. We don't need two models
for communicating configurations to servers, and I'd prefer Heat stay
focused on making HOT template authors' and users' lives better.

I have seen an alternative approach which separates a configuration
definition from a configuration deployer. This at least makes it clear
that the configuration is a part of a server. In pseudo-HOT:

resources:
 WebConfig:
   type: OS::Heat::ChefCookbook
   properties:
 cookbook_url: https://some.test/foo
 parameters:
   endpoint_host:
 type: string
 WebServer:
   type: OS::Nova::Server
   properties:
 image: webserver
 flavor: 100
 DeployWebConfig:
   type: OS::Heat::ConfigDeployer
   properties:
 configuration: {get_resource: WebConfig}
 on_server: {get_resource: WebServer}
 parameters:
   endpoint_host: {get_attribute: [ WebServer, first_ip]}



This is what Thomas defined, with one optimisation.
- The webconfig is a yaml template.

As you say the component is static - if so why even put it inline in
the template (well that was my thinking, it seems like a template not
really a resource).



I have implementation questions about both of these approaches though,
as it appears they'd have to reach backward in the graph to insert
their configuration, or have a generic bucket for all configuration


Yeah, it does depend on the implementation. If we use Mistral the
agent will need to ask Mistral for the tasks that apply to the server.

$ mistral task-consume \
 --tags=instance_id=$(my_instance_id);stack_id=$(stack_id)



to be inserted. IMO that would look a lot like the method I proposed,
which was to just have a list of components attached directly to the
server like this:

components:
 WebConfig:
   type: Chef::Cookbook
   properties:
 cookbook_url: https://some.test/foo
 parameters:
   endpoing_host:
 type: string
resources:
 WebServer:
   type: OS::Nova::Server
   properties:
 image: webserver
 flavor: 100
   components:
 - webconfig:
   component: {get_component: WebConfig}
   parameters:
 endpoint_host: {get_attribute: [ WebServer, first_ip ]}

I'd change this to:

components:
  - webconfig:
component: {get_file: ./my_configs/webconfig.yaml}
parameters:
  endpoint_host: {get_attribute: [ WebServer, first_ip ]}

This *could* be a short hand notation like the volumes property on
aws instances.



Of course, the keen eye will see the circular dependency there with the
WebServer trying to know its own IP. We've identified quite a few use
cases for self-referencing attributes, so that is a separate problem we
should solve independent of the template composition problem.


(aside) I don't like the idea of self ref as it breaks the idea that
references are resolved top down. Basically we have to put in
a nasty hack to produce broken behaviour (first resolution is
bogus and only following resoultions are possibly correct).

In this case just use the deployer to break your circular dep?



Anyway, I prefer the idea that parse-time things are called components
and run-time things are resources. I don't need a database entry for
"WebConfig" above. It is in the template and entirely static, just
sitting there as a reusable chunk for servers to pull in as-needed.


IMO is should just be a template/formatted file.



Anyway, I don't feel that we resolved any of these issues in the session
about configuration at the summit. If we did, we did not record them
in the etherpad or the blueprint. We barely got through the prepared
list of requirements and only were able to spell out problems, not
any solutions. So forgive me if I missed something and want to keep on
discussing this.

___
OpenStack-dev mailing list
Open

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Georgy Okrokvertskhov]

Hi,

I agree with Clint that component placement specified inside component
configuration is not a right thing. I remember that mostly everyone agreed
that "hosted_on" should not be in HOT templates. When one specify placement
explicitly inside  a component definition it prevents the following:
1. Reusability - you can't reuse component without creating its definition
copy with another placement parameter.
2. Composability - it will be no clear way to express composable
configurations. There was a clear way in a template showed during design
session where server had a list of components to be placed.
3. Deployment order - some components should be placed in strict order and
it will be much easier just make an ordered list of components then express
artificial dependencies between them just for ordering.

Thanks
Georgy


On Tue, Nov 12, 2013 at 10:32 AM, Clint Byrum  wrote:

> Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> >
> > Hi all,
> >
> > I have just posted the following wiki page to reflect a refined proposal
> > for HOT software configuration based on discussions at the design summit
> > last week. Angus also put a sample up in an etherpad last week, but we
> did
> > not have enough time to go thru it in the design session. My write-up is
> > based on Angus' sample, actually a refinement, and on discussions we had
> in
> > breaks, plus it is trying to reflect all the good input from ML
> discussions
> > and Steve Baker's initial proposal.
> >
> > https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >
> > Please review and provide feedback.
>
> Hi Thomas, thanks for spelling this out clearly.
>
> I am still -1 on anything that specifies the place a configuration is
> hosted inside the configuration definition itself. Because configurations
> are encapsulated by servers, it makes more sense to me that the servers
> (or server groups) would specify their configurations. If changing to a
> more logical model is just too hard for TOSCA to adapt to, then I suggest
> this be an area that TOSCA differs from Heat. We don't need two models
> for communicating configurations to servers, and I'd prefer Heat stay
> focused on making HOT template authors' and users' lives better.
>
> I have seen an alternative approach which separates a configuration
> definition from a configuration deployer. This at least makes it clear
> that the configuration is a part of a server. In pseudo-HOT:
>
> resources:
>   WebConfig:
> type: OS::Heat::ChefCookbook
> properties:
>   cookbook_url: https://some.test/foo
>   parameters:
> endpoint_host:
>   type: string
>   WebServer:
> type: OS::Nova::Server
> properties:
>   image: webserver
>   flavor: 100
>   DeployWebConfig:
> type: OS::Heat::ConfigDeployer
> properties:
>   configuration: {get_resource: WebConfig}
>   on_server: {get_resource: WebServer}
>   parameters:
> endpoint_host: {get_attribute: [ WebServer, first_ip]}
>
> I have implementation questions about both of these approaches though,
> as it appears they'd have to reach backward in the graph to insert
> their configuration, or have a generic bucket for all configuration
> to be inserted. IMO that would look a lot like the method I proposed,
> which was to just have a list of components attached directly to the
> server like this:
>
> components:
>   WebConfig:
> type: Chef::Cookbook
> properties:
>   cookbook_url: https://some.test/foo
>   parameters:
> endpoing_host:
>   type: string
> resources:
>   WebServer:
> type: OS::Nova::Server
> properties:
>   image: webserver
>   flavor: 100
> components:
>   - webconfig:
> component: {get_component: WebConfig}
> parameters:
>   endpoint_host: {get_attribute: [ WebServer, first_ip ]}
>
> Of course, the keen eye will see the circular dependency there with the
> WebServer trying to know its own IP. We've identified quite a few use
> cases for self-referencing attributes, so that is a separate problem we
> should solve independent of the template composition problem.
>
> Anyway, I prefer the idea that parse-time things are called components
> and run-time things are resources. I don't need a database entry for
> "WebConfig" above. It is in the template and entirely static, just
> sitting there as a reusable chunk for servers to pull in as-needed.
>
> Anyway, I don't feel that we resolved any of these issues in the session
> about configuration at the summit. If we did, we did not record them
> in the etherpad or the blueprint. We barely got through the prepared
> list of requirements and only were able to spell out problems, not
> any solutions. So forgive me if I missed something and want to keep on
> discussing this.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailm

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Christopher Armstrong]

On Tue, Nov 12, 2013 at 10:32 AM, Clint Byrum  wrote:

> Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> >
> > Hi all,
> >
> > I have just posted the following wiki page to reflect a refined proposal
> > for HOT software configuration based on discussions at the design summit
> > last week. Angus also put a sample up in an etherpad last week, but we
> did
> > not have enough time to go thru it in the design session. My write-up is
> > based on Angus' sample, actually a refinement, and on discussions we had
> in
> > breaks, plus it is trying to reflect all the good input from ML
> discussions
> > and Steve Baker's initial proposal.
> >
> > https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> >
> > Please review and provide feedback.
>
> Hi Thomas, thanks for spelling this out clearly.
>
> I am still -1 on anything that specifies the place a configuration is
> hosted inside the configuration definition itself. Because configurations
> are encapsulated by servers, it makes more sense to me that the servers
> (or server groups) would specify their configurations. If changing to a
> more logical model is just too hard for TOSCA to adapt to, then I suggest
> this be an area that TOSCA differs from Heat. We don't need two models
> for communicating configurations to servers, and I'd prefer Heat stay
> focused on making HOT template authors' and users' lives better.
>
>

I agree that the specification of which configs go on which servers should
be separated from both. This is how the good configuration tools like
puppet work anyway: you specify your servers, you specify ways to configure
them, and you specify which servers get which configs, all in potentially
separate places for maximum flexibility.

-- 
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Clint Byrum]

Excerpts from Thomas Spatzier's message of 2013-11-11 08:57:58 -0800:
> 
> Hi all,
> 
> I have just posted the following wiki page to reflect a refined proposal
> for HOT software configuration based on discussions at the design summit
> last week. Angus also put a sample up in an etherpad last week, but we did
> not have enough time to go thru it in the design session. My write-up is
> based on Angus' sample, actually a refinement, and on discussions we had in
> breaks, plus it is trying to reflect all the good input from ML discussions
> and Steve Baker's initial proposal.
> 
> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
> 
> Please review and provide feedback.

Hi Thomas, thanks for spelling this out clearly.

I am still -1 on anything that specifies the place a configuration is
hosted inside the configuration definition itself. Because configurations
are encapsulated by servers, it makes more sense to me that the servers
(or server groups) would specify their configurations. If changing to a
more logical model is just too hard for TOSCA to adapt to, then I suggest
this be an area that TOSCA differs from Heat. We don't need two models
for communicating configurations to servers, and I'd prefer Heat stay
focused on making HOT template authors' and users' lives better.

I have seen an alternative approach which separates a configuration
definition from a configuration deployer. This at least makes it clear
that the configuration is a part of a server. In pseudo-HOT:

resources:
  WebConfig:
type: OS::Heat::ChefCookbook
properties:
  cookbook_url: https://some.test/foo
  parameters:
endpoint_host:
  type: string
  WebServer:
type: OS::Nova::Server
properties:
  image: webserver
  flavor: 100
  DeployWebConfig:
type: OS::Heat::ConfigDeployer
properties:
  configuration: {get_resource: WebConfig}
  on_server: {get_resource: WebServer}
  parameters:
endpoint_host: {get_attribute: [ WebServer, first_ip]}

I have implementation questions about both of these approaches though,
as it appears they'd have to reach backward in the graph to insert
their configuration, or have a generic bucket for all configuration
to be inserted. IMO that would look a lot like the method I proposed,
which was to just have a list of components attached directly to the
server like this:

components:
  WebConfig:
type: Chef::Cookbook
properties:
  cookbook_url: https://some.test/foo
  parameters:
endpoing_host:
  type: string
resources:
  WebServer:
type: OS::Nova::Server
properties:
  image: webserver
  flavor: 100
components:
  - webconfig:
component: {get_component: WebConfig}
parameters:
  endpoint_host: {get_attribute: [ WebServer, first_ip ]}

Of course, the keen eye will see the circular dependency there with the
WebServer trying to know its own IP. We've identified quite a few use
cases for self-referencing attributes, so that is a separate problem we
should solve independent of the template composition problem.

Anyway, I prefer the idea that parse-time things are called components
and run-time things are resources. I don't need a database entry for
"WebConfig" above. It is in the template and entirely static, just
sitting there as a reusable chunk for servers to pull in as-needed.

Anyway, I don't feel that we resolved any of these issues in the session
about configuration at the summit. If we did, we did not record them
in the etherpad or the blueprint. We barely got through the prepared
list of requirements and only were able to spell out problems, not
any solutions. So forgive me if I missed something and want to keep on
discussing this.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Alex Heneveld]

Thanks Steve, Zane, Mike.  I withdraw the $idea ... for now (*).

As a user who wants to move away from fiddling with infrastructure I 
like where this is going.  :)


--A


(*)  If there start to be lots of get_param usages and other keywords 
then perhaps $this -- and requiring $$ escapes -- or similar will become 
a good thing but it's hardly needed now, esp if aim is to preserve 
backwards compatibility.



On 12/11/2013 16:43, Steven Hardy wrote:

On Tue, Nov 12, 2013 at 01:59:35PM +, Alex Heneveld wrote:

This new proposal [1] looks like a very positive change on top of
[2].  Simpler both to read and write.

Agreed, thanks Thomas for taking the time to write this up, it looks like a
good reflection of what was discussed last week to me.


One minor suggestion is to consider using a special character (eg $)
rather than reserved keywords.  As I understand it the keywords are
only interpreted when they exactly match the value of a key in a
map, so it is already unlikely to be problematic.  However I think
it would be more familiar and clear if we instead used the rule that
any item (key or value) which _starts_ with a $ is interpreted
specially.  What those rules are is TBD but you could for instance
write functions -- as either `$get_param('xxx')` or `$get_param:
xxx` -- as well as allow accessing a parameter directly `$xxx `.

So, there has been some discussion on this topic related to the str_replace
usage of $ prefixes.  In the end we decided straight string replacement was
better in that case, see:

https://review.openstack.org/#/c/47772/

I'm -1 on any backwards incompatible HOT syntax changes at this point,
unless they can be very strongly justified - we did add a disclaimer to the
Havana release notes saying HOT was an unstable in-progress format, but
people will be using it nevertheless.

If possible, I'd prefer to focus on adding stuff to enable the
software-config functionality to work, in as backwards-compatible way as is
practical.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Mike Spreitzer]

Zane Bitter  wrote on 11/12/2013 12:06:14 PM:

> ... However, if 
> for any reason you have a dict key starting with "$" and we interpret 
> that specially, then you are basically hosed since you almost certainly 
> _needed_ it to actually start with "$" for a reason. So -1.
> 
> cheers,
> Zane.

Yeah, I actually have this case in some private usage.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Zane Bitter]

On 12/11/13 14:59, Alex Heneveld wrote:

One minor suggestion is to consider using a special character (eg $)
rather than reserved keywords.  As I understand it the keywords are only
interpreted when they exactly match the value of a key in a map, so it
is already unlikely to be problematic.  However I think it would be more
familiar and clear if we instead used the rule that any item (key or
value) which _starts_ with a $ is interpreted specially.  What those
rules are is TBD but you could for instance write functions -- as either
`$get_param('xxx')` or `$get_param: xxx` -- as well as allow accessing a
parameter directly `$xxx `.


This sounds like a nice idea on the surface. AWS accomplished the same 
thing by namespacing functions with the Fn:: prefix (except for 'Ref', 
bizarrely), and it works fine because the chances are if you randomly 
(maybe in a Metadata section) have a dict key that happens to start with 
"Fn::" then you can probably just choose a different name. However, if 
for any reason you have a dict key starting with "$" and we interpret 
that specially, then you are basically hosed since you almost certainly 
_needed_ it to actually start with "$" for a reason. So -1.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Steven Hardy]

On Tue, Nov 12, 2013 at 01:59:35PM +, Alex Heneveld wrote:
> This new proposal [1] looks like a very positive change on top of
> [2].  Simpler both to read and write.

Agreed, thanks Thomas for taking the time to write this up, it looks like a
good reflection of what was discussed last week to me.

> One minor suggestion is to consider using a special character (eg $)
> rather than reserved keywords.  As I understand it the keywords are
> only interpreted when they exactly match the value of a key in a
> map, so it is already unlikely to be problematic.  However I think
> it would be more familiar and clear if we instead used the rule that
> any item (key or value) which _starts_ with a $ is interpreted
> specially.  What those rules are is TBD but you could for instance
> write functions -- as either `$get_param('xxx')` or `$get_param:
> xxx` -- as well as allow accessing a parameter directly `$xxx `.

So, there has been some discussion on this topic related to the str_replace
usage of $ prefixes.  In the end we decided straight string replacement was
better in that case, see:

https://review.openstack.org/#/c/47772/

I'm -1 on any backwards incompatible HOT syntax changes at this point,
unless they can be very strongly justified - we did add a disclaimer to the
Havana release notes saying HOT was an unstable in-progress format, but
people will be using it nevertheless.

If possible, I'd prefer to focus on adding stuff to enable the
software-config functionality to work, in as backwards-compatible way as is
practical.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] HOT software configuration refined after design summit discussions

[Alex Heneveld]

+1

This new proposal [1] looks like a very positive change on top of [2].  
Simpler both to read and write.


One minor suggestion is to consider using a special character (eg $) 
rather than reserved keywords.  As I understand it the keywords are only 
interpreted when they exactly match the value of a key in a map, so it 
is already unlikely to be problematic.  However I think it would be more 
familiar and clear if we instead used the rule that any item (key or 
value) which _starts_ with a $ is interpreted specially.  What those 
rules are is TBD but you could for instance write functions -- as either 
`$get_param('xxx')` or `$get_param: xxx` -- as well as allow accessing a 
parameter directly `$xxx `.


Very excited to see where this goes.

Best
Alex


[1] https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP
[2] https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config


On 11/11/2013 16:57, Thomas Spatzier wrote:

Hi all,

I have just posted the following wiki page to reflect a refined proposal
for HOT software configuration based on discussions at the design summit
last week. Angus also put a sample up in an etherpad last week, but we did
not have enough time to go thru it in the design session. My write-up is
based on Angus' sample, actually a refinement, and on discussions we had in
breaks, plus it is trying to reflect all the good input from ML discussions
and Steve Baker's initial proposal.

https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-WIP

Please review and provide feedback.

Regards,
Thomas


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev