Re: [openstack-dev] [Glare][Heat][TripleO] Heat artifact type

[Randall Burt]

FWIW, option 2 is almost required unless we plan to be able to bundle multiple 
environments with a single template. While having a single environment for a 
single template can be useful, the even *more* useful scenario (and the primary 
one driving the development of environments initially) is when you have options 
as to how a template behaves (use Trove for the backend or pop vms and software 
config to install a database). IMO, you'd want to de-couple environments from 
the templates given that multiple environment could work for the same template.
 
On Jul 20, 2016, at 8:58 AM, "Mikhail Fedosin" 
 wrote:

> 
> 
> On Wed, Jul 20, 2016 at 5:12 AM, Qiming Teng  
> wrote:
> On Tue, Jul 19, 2016 at 06:44:06PM +0300, Oleksii Chuprykov wrote:
> > Hello!
> >
> > Today it was announced that Glare is ready for public review
> > http://lists.openstack.org/pipermail/openstack-dev/2016-July/099553.html So
> > we are ready to start working on integration Heat with Glare and
> > implementing a POC. After discussions with Glare team we see two design
> > options:
> >
> > 1) Create one artifact type that will contain template, nested templates
> > and environments.
> > Pros: It is easy to maintain integrity. Since artifact is immutable, we can
> > guarantee the consistency and prevent from accidentally removing of
> > dependent environment.
> > Cons: If we need to add new environments to use them with template, we need
> > to create new artifact.
> >
> > 2) Create 2 artifact types: environment and template.
> > Pros: It is easy to add new environments. You just need to create new
> > dependency from template artifact to environment one.
> > Cons: Some environment can be (mistakenly) removed, and template that have
> > dependencies on it will be in inconsistent state.
> 
> Option 2 looks more flexible to me. I'm not sure we are encouraging
> users to introduce or rely on a hard dependency from a template to an
> environment file. With that, it is still good to know whether glare
> supports the concept of 'reference' where a referenced artifact cannot
> be deleted.
> 
> Hey! 
> 
> Indeed, option 2 is more flexible, but in this case users have to manually 
> control dependencies, which is may be hard sometimes. Also, initially Glare 
> won't support 'hard' dependencies, this feature will be added in next 
> version, because it requires additional discussions. For this reason I 
> recommend option 1 and let Glare control template consistency for you, it 
> won't allow users to break anything. 
> 
> Best,
> Mike
>  
> 
>  - Qiming
> 
> > So we want to hear your opinions and suggestions on the matter. Thanks in
> > advance!
> >
> > Best regards,
> > Oleksii Chuprykov
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] upgrade options for custom heat resource plug-ins

[Randall Burt]

There is a mechanism to mark them as support status "hidden" so that they don't 
show up in resource-type-show and aren't allowed in new templates, but older 
templates should still work. Eventually they may go away altogether but that 
should be far in the future. For your custom resources, you can decide when or 
if to ever remove them.

On Apr 11, 2016, at 3:58 PM, "Praveen Yalagandula" 
 wrote:

> Randall,
> 
> Thanks for your reply.
> I was wondering especially about those "deprecated" properties.. What happens 
> after several releases? Do you just remove them at that point? If the 
> expected maximum lifespan of a stack is shorter than the span for which those 
> "deprecated" properties are maintained, then removing them works. But what 
> happens if it is longer?
> 
> Cheers,
> Praveen
> 
> On Mon, Apr 11, 2016 at 12:02 PM Randall Burt  
> wrote:
> Not really. Ideally, you need to write your resource such that these changes 
> are backwards compatible. We do this for the resources we ship with Heat (add 
> new properties while supporting deprecated properties for several releases).
> 
> On Apr 11, 2016, at 1:06 PM, "Praveen Yalagandula" 
>  wrote:
> 
> > Hi,
> >
> > We are developing a custom heat resource plug-in and wondering about how to 
> > handle plug-in upgrades. As our product's object model changes with new 
> > releases, we will need to release updated resource plug-in code too. 
> > However, the "properties" stored in the heat DB for the existing resources, 
> > whose definitions have been upgraded, need to be updated too. Was there any 
> > discussion on this?
> >
> > Thanks,
> > Praveen Yalagandula
> > Avi Networks
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] upgrade options for custom heat resource plug-ins

[Randall Burt]

Not really. Ideally, you need to write your resource such that these changes 
are backwards compatible. We do this for the resources we ship with Heat (add 
new properties while supporting deprecated properties for several releases).

On Apr 11, 2016, at 1:06 PM, "Praveen Yalagandula" 
 wrote:

> Hi,
> 
> We are developing a custom heat resource plug-in and wondering about how to 
> handle plug-in upgrades. As our product's object model changes with new 
> releases, we will need to release updated resource plug-in code too. However, 
> the "properties" stored in the heat DB for the existing resources, whose 
> definitions have been upgraded, need to be updated too. Was there any 
> discussion on this?
> 
> Thanks,
> Praveen Yalagandula
> Avi Networks
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Rico Lin for heat-core

[Randall Burt]

+1

 Original message 
From: Sergey Kraynev
Date:12/07/2015 6:41 AM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: [openstack-dev] [heat] Rico Lin for heat-core

Hi all.

I'd like to nominate Rico Lin for heat-core. He did awesome job with providing 
useful and valuable reviews. Also his contribution is really high [1] .

[1] http://stackalytics.com/report/contribution/heat-group/60

Heat core-team, please vote with:
 +1 - if you agree
  -1 - if you disagree

--
Regards,
Sergey.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] publicURL vs internalURL for resource validation

[Randall Burt]

This is a coin flip IMO. Heat can be deployed separately from other 
infrastructure.  If this does change, I'd like to see a fail over mechanism in 
the client plugin mechanism that will switch to the public URL if it can't 
access the private one.

 Original message 
From: Matt Fischer
Date:10/24/2015 2:35 AM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: Re: [openstack-dev] [Heat] publicURL vs internalURL for resource 
validation


>From an operations point of view I'd also prefer all service to service calls 
>to go through the internalURL is there a reason it's not default?

On Oct 24, 2015 7:56 AM, "Attila Szlovencsak" 
mailto:att...@componentsoft.eu>> wrote:
Hi!

I am using Openstack Kilo (2015.1.1)
As I learned from the code, heat-engine uses the endpoint-type "publicURL",
when validating templates. I also see that I can override that from
heat.conf via [clients_XXX]/endpoint_type.


heat/engine/clients/os/nova.py

def _create(self):
endpoint_type = self._get_client_option('nova', 'endpoint_type')
management_url = self.url_for(service_type='compute',
  endpoint_type=endpoint_type)


/heat/common/config.py
=
# these options define baseline defaults that apply to all clients
default_clients_opts = [
cfg.StrOpt('endpoint_type',
   default='publicURL',


My questions:

1. Shouldn't we use the  interalURL instead as default?  In a typical case,
the controller node sits behind a load-balancer, and IP for the publicURLs
are held by the load-balancer. The controller node (so heat-engine) might
not have access to the publicURL at all.

2. Instead of creating and "endpoint_type" entry in heat.conf for each and
every service,  is there a simpler way to force using the internalURL?

Thanks in advance,
Attila


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] core team nomination

[Randall Burt]

+1

 Original message 
From: Sergey Kraynev
Date:10/20/2015 8:42 AM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: [openstack-dev] [Heat] core team nomination

I'd like to propose new candidates for heat core-team:
Rabi Mishra
Peter Razumovsky

According statistic both candidate made a big effort in Heat as
reviewers and as contributors [1][2].
They were involved in Heat community work  during last several releases and
showed good understanding of Heat code.
I think, that they are ready to became core-reviewers.

Heat-cores, please vote with +/- 1.

[1] http://stackalytics.com/report/contribution/heat-group/180
[2] http://stackalytics.com/?module=heat-group&metric=person-day
--
Regards,
Sergey.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] creating a stack with a config_drive

[Randall Burt]

The drive will contain the user data. Its an alternative to the metadata 
service and isn't a "normal" drive. Its created, mounted, and populated by Nova.

 Original message 
From: Maish Saidel-Keesing
Date:08/07/2015 8:35 AM (GMT-06:00)
To: Randall Burt , maishsk+openst...@maishsk.com, "OpenStack Development 
Mailing List (not for usage questions)"
Subject: Re: [openstack-dev] [Heat] creating a stack with a config_drive

On 08/07/15 16:22, Randall Burt wrote:
config_drive: true just tells the instance to mount the drive. You pass data 
via the user_data property.

Thanks Randall that is what I was thinking.

But I am confused.

When booting an instance with nova boot, I can configure a local file/directory 
to be mounted as a config drive on the instance upon boot. I can also provide 
information and commands regularly through the user_data

Through Heat I can provide configuration through user_data. And I can also 
mount a config_drive.

Where do I define what that config_drive contains?


 Original message 
From: Maish Saidel-Keesing
Date:08/07/2015 8:08 AM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: [openstack-dev] [Heat] creating a stack with a config_drive

I have been looking for a working example to create Heat stack with a
config_drive attached.

I know it is possible to deploy a nova instance with the CLI [1]

I see that OS::Nova::Server has a config_drive property that is a
Boolean value [2]

What I cannot find is how this can be used. Where is the path defined
for the config file?
Or am I completely missing what and how this should be used?

Anyone with more info on this - I would be highly grateful.

Thanks.

[1] http://docs.openstack.org/user-guide/cli_config_drive.html
[2]
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server


--
Best Regards,
Maish Saidel-Keesing

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
Best Regards,
Maish Saidel-Keesing
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] creating a stack with a config_drive

[Randall Burt]

config_drive: true just tells the instance to mount the drive. You pass data 
via the user_data property.

 Original message 
From: Maish Saidel-Keesing
Date:08/07/2015 8:08 AM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: [openstack-dev] [Heat] creating a stack with a config_drive

I have been looking for a working example to create Heat stack with a
config_drive attached.

I know it is possible to deploy a nova instance with the CLI [1]

I see that OS::Nova::Server has a config_drive property that is a
Boolean value [2]

What I cannot find is how this can be used. Where is the path defined
for the config file?
Or am I completely missing what and how this should be used?

Anyone with more info on this - I would be highly grateful.

Thanks.

[1] http://docs.openstack.org/user-guide/cli_config_drive.html
[2]
http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Nova::Server


--
Best Regards,
Maish Saidel-Keesing

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [app-catalog] conditional resource exposure - second thoughts

[Randall Burt]

A feedback mechanism for users is obviously a "good thing", but IMO, not 
germane to the threads original purpose of how and when to expose supported 
resources in Heat. I cannot imagine us implementing such a feature directly.

This may be a good discussion to have in the context of app catalog exclusively 
and not in the context of Heat since we seem to be discussing a generally 
available catalog vs Heat running in a specific environment. These two issues 
are quite different in terms of "what's supported". 

In the Heat case, the documentation in OpenStack is "good enough" IMO for "what 
are all the things Heat can possibly let me do" while the chosen endpoint is 
the place to answer "what are the things *this* installation of Heat will let 
me do". If the answer to the latter is unsatisfactory, then the user should 
already have mechanisms to encourage the operator to provide what's missing.

On Jul 14, 2015, at 5:31 PM, "Fox, Kevin M" 
 wrote:

> Without a feedback loop of some kind, how does one solve issues like the 
> following:
> 
> * Operator decides users don't need neutron NaaS because its "too complicated 
> and users don't need it" (Seen on the mailing list multiple times)
> * Software developer writes cloud templates that deploys software that needs 
> private networks to work (For example, ElasticSearch)
> * User wants to deploy said software but can't discover a working version.
> 
> User is sad because they can't find a working template to do what they want. 
> They either reinvent the wheel, or give up and don't use the cloud for that 
> task.
> 
> Being able to close the loop and let the operator easily know the users 
> actually need something they aren't providing gives them the opportunity to 
> fix the issue, benefiting all 3 parties.
> 
> Thanks,
> Kevin
> 
> 
> From: Randall Burt [randall.b...@rackspace.com]
> Sent: Tuesday, July 14, 2015 2:40 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Heat] [app-catalog] conditional resource 
> exposure - secondthoughts
> 
> Making users complain to admins that may have little to no control over what 
> is and isn't available isn't a healthy strategy for user experience. 
> Purposefully engineering hardship to try and influence operators to "do the 
> right thing" in someone else's opinion sounds pretty counter productive to 
> adoption as well.
> 
> FWIW, as a user, I don't want to see things I can't use because it just 
> wastes my time. I agree that the docs are a good place to see "all the 
> things" while querying the service should tell me what's available to me at 
> the time.
> 
> On Jul 14, 2015, at 4:20 PM, "Fox, Kevin M" 
> wrote:
> 
>> We're kind of debating the same thing for the app catalog. Do we show 
>> templates that don't work on a given cloud since they wont work, potentially 
>> making useful tools hard to discover, or do we view it as an opportunity for 
>> users to complain to their admins that they need X feature in order to do 
>> what they need to do. Last time we talked about it, we were leaning towards 
>> the latter.
>> 
>> Maybe a happy middle ground is to have enough smarts in the system to show 
>> the templates, identify what parts won't work, gray out the template but 
>> provide a ui to "notifify the admin" of desire for X to work. That way users 
>> can easily feed back their desires.
>> 
>> Thanks,
>> Kevin
>> From: Pavlo Shchelokovskyy [pshchelokovs...@mirantis.com]
>> Sent: Tuesday, July 14, 2015 11:34 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: [openstack-dev] [Heat] conditional resource exposure - second 
>> thoughts
>> 
>> Hi Heaters,
>> currently we already expose to the user only resources for services deployed 
>> in the cloud [1], and soon we will do the same based on whether actual user 
>> roles allow creating specific resources [2]. Here I would like to get your 
>> opinion on some of my thoughts regarding behavior of resource-type-list, 
>> resource-type-show and template-validate with this new features.
>> resource-type-list
>> We already (or soon will) hide unavailable in the cloud / for the user 
>> resources from the listing. But what if we add an API flag e.g. --all to 
>> show all registered in the engine resources? That would give any user a 
>> glimpse of what their Orchestration service can manage in principle, so they 
>> can nag the cloud o

Re: [openstack-dev] [Heat] [app-catalog] conditional resource exposure - second thoughts

[Randall Burt]

Making users complain to admins that may have little to no control over what is 
and isn't available isn't a healthy strategy for user experience. Purposefully 
engineering hardship to try and influence operators to "do the right thing" in 
someone else's opinion sounds pretty counter productive to adoption as well.

FWIW, as a user, I don't want to see things I can't use because it just wastes 
my time. I agree that the docs are a good place to see "all the things" while 
querying the service should tell me what's available to me at the time.

On Jul 14, 2015, at 4:20 PM, "Fox, Kevin M" 
 wrote:

> We're kind of debating the same thing for the app catalog. Do we show 
> templates that don't work on a given cloud since they wont work, potentially 
> making useful tools hard to discover, or do we view it as an opportunity for 
> users to complain to their admins that they need X feature in order to do 
> what they need to do. Last time we talked about it, we were leaning towards 
> the latter.
> 
> Maybe a happy middle ground is to have enough smarts in the system to show 
> the templates, identify what parts won't work, gray out the template but 
> provide a ui to "notifify the admin" of desire for X to work. That way users 
> can easily feed back their desires.
> 
> Thanks,
> Kevin
> From: Pavlo Shchelokovskyy [pshchelokovs...@mirantis.com]
> Sent: Tuesday, July 14, 2015 11:34 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: [openstack-dev] [Heat] conditional resource exposure - second 
> thoughts
> 
> Hi Heaters,
> currently we already expose to the user only resources for services deployed 
> in the cloud [1], and soon we will do the same based on whether actual user 
> roles allow creating specific resources [2]. Here I would like to get your 
> opinion on some of my thoughts regarding behavior of resource-type-list, 
> resource-type-show and template-validate with this new features.
> resource-type-list
> We already (or soon will) hide unavailable in the cloud / for the user 
> resources from the listing. But what if we add an API flag e.g. --all to show 
> all registered in the engine resources? That would give any user a glimpse of 
> what their Orchestration service can manage in principle, so they can nag the 
> cloud operator to install additional OpenStack components or give them 
> required roles :)
> resource-type-show
> Right now the plan is to disable "showing" unavailable to the user resources. 
> But may be we should leave this as it is, for the same purpose as above (or 
> again add a --all flag or such)?
> template-validate
> Right now Heat is failing validation for templates containing resource types 
> not registered in the engine (e.g. typo). Should we also make this call 
> available services- and roles-sensitive? Or should we leave a way for a user 
> to check validity of any template with any in principle supported resources?
> The bottom line is we are good in making Heat service to be as much 
> self-documented via its own API as possible - let's keep doing that and make 
> any Heat deployment to be the Heat primer :)
> Eager to hear your opinions.
> [1] 
> http://specs.openstack.org/openstack/heat-specs/specs/liberty/conditional-resource-exposure-services.html
> [2] 
> http://specs.openstack.org/openstack/heat-specs/specs/liberty/conditional-resource-exposure-roles.html
> Best regards,
> -- 
> Dr. Pavlo Shchelokovskyy
> Senior Software Engineer
> Mirantis Inc
> www.mirantis.com
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Show attribute is a collection of other attributes or not?

[Randall Burt]

On Jul 2, 2015, at 2:35 PM, Steve Baker 
 wrote:

> On 03/07/15 06:03, Randall Burt wrote:
>> Maybe use "all" for all attributes in the schema and use "show" for the raw 
>> output from the service (as is done today for server and neutron stuff).
> Instead of "all", how about allowing a special form of {get_attr: 
> [resource_name]} with no extra arguments to return a dict of all attributes? 
> This would be consistent with how extra arguments traverse attribute data.

+1 (Hope you can read this despite my bobo client).

>> On Jul 2, 2015, at 12:46 PM, Steven Hardy 
>>  wrote:
>> 
>>> On Thu, Jul 02, 2015 at 04:40:49PM +0300, Sergey Kraynev wrote:
>>>>   Hi Heaters.
>>>>   I don't think that my question is very huge for openstack-dev, but it
>>>>   affects a lot of Heat resourcesA
>>>>   and need collect more opinions before apply some of follow approaches.
>>>>   I recently uploaded initial approach for implementation common 'show'
>>>>   attribute [1]A
>>>>   On one of this review was raised one interesting suggestion:
>>>>   'show' attribute should return map of all resource's attributes, i.e.
>>>>   for each attr in self.attributes_schema:
>>>>   A  A outputs[attr] = A _resolve_attribute(attr)
>>>>   return outputs
>>>>   I agree, that it's more easier than separate show_resource method for 
>>>> each
>>>>   resource and it's the same, what returns Neutron API on "show" request.
>>>>   However, we already has opposite example, when OS::Nova::Server resource
>>>>   has bunch of attributes which are not similar on current 'show' attribute
>>>>   output:
>>>>   
>>>> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L918
>>>>   I suppose, that the same situation will be and for other resources.
>>>>   So I want to ask about way, which we would like to follow?
>>>>   [1] "show" as collection of attributes
>>>>   [2] "show" as the same output for command " A >> I think [2] is the most useful, and most consistent with both the nova and
>>> all neutron resources:
>>> 
>>> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/neutron/neutron.py#L129
>>> 
>>> Another advantage of this "transparent" passthrough of the data returned by
>>> the client is folks have a workaround in the event heat attributes schema
>>> lack some new value that the client returns.  Obviously when it's added
>>> to the attributes schema, it'll be better to use that instead.
>>> 
>>> Steve
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Show attribute is a collection of other attributes or not?

[Randall Burt]

Maybe use "all" for all attributes in the schema and use "show" for the raw 
output from the service (as is done today for server and neutron stuff).

On Jul 2, 2015, at 12:46 PM, Steven Hardy 
 wrote:

> On Thu, Jul 02, 2015 at 04:40:49PM +0300, Sergey Kraynev wrote:
>>   Hi Heaters.
>>   I don't think that my question is very huge for openstack-dev, but it
>>   affects a lot of Heat resourcesA 
>>   and need collect more opinions before apply some of follow approaches.
>>   I recently uploaded initial approach for implementation common 'show'
>>   attribute [1]A 
>>   On one of this review was raised one interesting suggestion:
>>   'show' attribute should return map of all resource's attributes, i.e.
>>   for each attr in self.attributes_schema:
>>   A  A outputs[attr] = A _resolve_attribute(attr)
>>   return outputs
>>   I agree, that it's more easier than separate show_resource method for each
>>   resource and it's the same, what returns Neutron API on "show" request.
>>   However, we already has opposite example, when OS::Nova::Server resource
>>   has bunch of attributes which are not similar on current 'show' attribute
>>   output:
>>   
>> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/nova/server.py#L918
>>   I suppose, that the same situation will be and for other resources.
>>   So I want to ask about way, which we would like to follow?
>>   [1] "show" as collection of attributes
>>   [2] "show" as the same output for command " A  
> I think [2] is the most useful, and most consistent with both the nova and
> all neutron resources:
> 
> https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/neutron/neutron.py#L129
> 
> Another advantage of this "transparent" passthrough of the data returned by
> the client is folks have a workaround in the event heat attributes schema
> lack some new value that the client returns.  Obviously when it's added
> to the attributes schema, it'll be better to use that instead.
> 
> Steve
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Solum] [ Supporting swift downloads for operator languagepacks

[Randall Burt]

A bit of a tangent, but it seems like the url would be to a public Swift 
system. I am unclear if a "source git repo" would be relevant but, assuming 
Swift would be "optional", perhaps users could host catalog LP's in git or some 
other distribution mechanism and have a method by which solum could import them 
from the catalog into that deployment's object store.

On Jun 17, 2015, at 1:58 PM, "Fox, Kevin M" 
 wrote:

> This question may be off on a tangent, or may be related.
> 
> As part of the application catalog project, (http://apps.openstack.org/) 
> we're trying to provide globally accessible resources that can be easily 
> consumed in OpenStack Clouds. How would these global Language Packs fit in? 
> Would the url record in the app catalog be required to point to an Internet 
> facing public Swift system then? Or, would it point to the source git repo 
> that Solum would use to generate the LP still?
> 
> Thanks,
> Kevin
> 
> From: Randall Burt [randall.b...@rackspace.com]
> Sent: Wednesday, June 17, 2015 11:38 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Solum] Supporting swift   downloads   for   
>   operatorlanguagepacks
> 
> Yes. If an operator wants to make their LP publicly available outside of 
> Solum, I was thinking they could just make GET's on the container public. 
> That being said, I'm unsure if this is realistically do-able if you still 
> have to have an authenticated tenant to access the objects. Scratch that; 
> http://blog.fsquat.net/?p=40 may be helpful.
> 
> On Jun 17, 2015, at 1:27 PM, Adrian Otto 
> wrote:
> 
>> To be clear, Randall is referring to a swift container (directory).
>> 
>> Murali has a good idea of attempting to use swift client first, as it has 
>> performance optimizations that can speed up the process more than naive file 
>> transfer tools. I did mention to him that wget does have a retiree feature, 
>> and that we could see about using curl instead to allow for chunked encoding 
>> as additional optimizations.
>> 
>> Randall, are you suggesting that we could use swift client for both private 
>> and public LP uses? That sounds like a good suggestion to me.
>> 
>> Adrian
>> 
>>> On Jun 17, 2015, at 11:10 AM, Randall Burt  
>>> wrote:
>>> 
>>> Can't an operator make the target container public therefore removing the 
>>> need for multiple access strategies?
>>> 
>>>  Original message 
>>> From: Murali Allada
>>> Date:06/17/2015 11:41 AM (GMT-06:00)
>>> To: "OpenStack Development Mailing List (not for usage questions)"
>>> Subject: [openstack-dev] [Solum] Supporting swift downloads for operator 
>>> languagepacks
>>> 
>>> Hello Solum Developers,
>>> 
>>> When we were designing the operator languagepack feature for Solum, we 
>>> wanted to make use of public urls to download operator LPs, such as those 
>>> available for CDN backed swift containers we have at Rackspace, or any 
>>> publicly accessible url. This would mean that when a user chooses to build 
>>> applications on to​​p of a languagepack provided by the operator, we use a 
>>> url to 'wget' the LP image.
>>> 
>>> Recently, we have started noticing a number of failures because of 
>>> corrupted docker images downloaded using 'wget'. The docker images work 
>>> fine when we download them manually with a swift client and use them. The 
>>> corruption seem to be happening when we try to download a large image using 
>>> 'wget' and there are dropped packets or intermittent network issues.
>>> 
>>> My thinking is to start using the swift client to download operator LPs by 
>>> default instead of wget. The swift client already implements retry logic, 
>>> downloading large images in chunks, etc. This means we would not get the 
>>> niceties of using publicly accessible urls. However, the feature will be 
>>> more reliable and robust.
>>> 
>>> The implementation would be as follows:
>>> • ​We'll use the existing service tenant configuration available in the 
>>> solum config file to authenticate and store operator languagepacks using 
>>> the swift client. We were using a different tenant to build and host LPs, 
>>> but now that we require the tenants credentials in the config file, it's 
>>> best to reuse the exist

Re: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks

[Randall Burt]

Yes. If an operator wants to make their LP publicly available outside of Solum, 
I was thinking they could just make GET's on the container public. That being 
said, I'm unsure if this is realistically do-able if you still have to have an 
authenticated tenant to access the objects. Scratch that; 
http://blog.fsquat.net/?p=40 may be helpful.

On Jun 17, 2015, at 1:27 PM, Adrian Otto 
 wrote:

> To be clear, Randall is referring to a swift container (directory).
> 
> Murali has a good idea of attempting to use swift client first, as it has 
> performance optimizations that can speed up the process more than naive file 
> transfer tools. I did mention to him that wget does have a retiree feature, 
> and that we could see about using curl instead to allow for chunked encoding 
> as additional optimizations. 
> 
> Randall, are you suggesting that we could use swift client for both private 
> and public LP uses? That sounds like a good suggestion to me.
> 
> Adrian
> 
>> On Jun 17, 2015, at 11:10 AM, Randall Burt  
>> wrote:
>> 
>> Can't an operator make the target container public therefore removing the 
>> need for multiple access strategies? 
>> 
>>  Original message 
>> From: Murali Allada 
>> Date:06/17/2015 11:41 AM (GMT-06:00) 
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> Subject: [openstack-dev] [Solum] Supporting swift downloads for operator 
>> languagepacks
>> 
>> Hello Solum Developers, 
>>  
>> When we were designing the operator languagepack feature for Solum, we 
>> wanted to make use of public urls to download operator LPs, such as those 
>> available for CDN backed swift containers we have at Rackspace, or any 
>> publicly accessible url. This would mean that when a user chooses to build 
>> applications on to​​p of a languagepack provided by the operator, we use a 
>> url to 'wget' the LP image.
>> 
>> Recently, we have started noticing a number of failures because of corrupted 
>> docker images downloaded using 'wget'. The docker images work fine when we 
>> download them manually with a swift client and use them. The corruption seem 
>> to be happening when we try to download a large image using 'wget' and there 
>> are dropped packets or intermittent network issues.
>> 
>> My thinking is to start using the swift client to download operator LPs by 
>> default instead of wget. The swift client already implements retry logic, 
>> downloading large images in chunks, etc. This means we would not get the 
>> niceties of using publicly accessible urls. However, the feature will be 
>> more reliable and robust.
>> 
>> The implementation would be as follows:
>>  • ​We'll use the existing service tenant configuration available in the 
>> solum config file to authenticate and store operator languagepacks using the 
>> swift client. We were using a different tenant to build and host LPs, but 
>> now that we require the tenants credentials in the config file, it's best to 
>> reuse the existing service tenant creds. Note: If we don't, we'll have 3 
>> separate tenants to maintain. 
>>  • ​Service tenant 
>>  • Operator languagepack tenant
>>  • Global admin tenant 
>>  • I'll keep the option to download the operator languagepacks from a 
>> publicly available url. I'll allow operators to choose which method they 
>> want to use by changing a setting in the solum config file.
>> FYI: In my tests, I've noticed that downloading an image using the swift 
>> client is twice as fast as downloading the same image using 'wget' from a 
>> CDN url.
>> 
>> Thanks,
>> Murali
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks

[Randall Burt]

Can't an operator make the target container public therefore removing the need 
for multiple access strategies?

 Original message 
From: Murali Allada
Date:06/17/2015 11:41 AM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: [openstack-dev] [Solum] Supporting swift downloads for operator 
languagepacks

Hello Solum Developers,

When we were designing the operator languagepack feature for Solum, we wanted 
to make use of public urls to download operator LPs, such as those available 
for CDN backed swift containers we have at Rackspace, or any publicly 
accessible url. This would mean that when a user chooses to build applications 
on to??p of a languagepack provided by the operator, we use a url to 'wget' the 
LP image.

Recently, we have started noticing a number of failures because of corrupted 
docker images downloaded using 'wget'. The docker images work fine when we 
download them manually with a swift client and use them. The corruption seem to 
be happening when we try to download a large image using 'wget' and there are 
dropped packets or intermittent network issues.

My thinking is to start using the swift client to download operator LPs by 
default instead of wget. The swift client already implements retry logic, 
downloading large images in chunks, etc. This means we would not get the 
niceties of using publicly accessible urls. However, the feature will be more 
reliable and robust.

The implementation would be as follows:

  *   ?We'll use the existing service tenant configuration available in the 
solum config file to authenticate and store operator languagepacks using the 
swift client. We were using a different tenant to build and host LPs, but now 
that we require the tenants credentials in the config file, it's best to reuse 
the existing service tenant creds. Note: If we don't, we'll have 3 separate 
tenants to maintain.
 *   ?Service tenant
 *   Operator languagepack tenant
 *   Global admin tenant
  *   I'll keep the option to download the operator languagepacks from a 
publicly available url. I'll allow operators to choose which method they want 
to use by changing a setting in the solum config file.

FYI: In my tests, I've noticed that downloading an image using the swift client 
is twice as fast as downloading the same image using 'wget' from a CDN url.

Thanks,
Murali

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Solum] Should logs be deleted when we delete an app?

[Randall Burt]

While I agree with what you're saying, the way the OpenStack clients are 
traditionally written/designed, the CLI *is* the SDK for those users who want 
to do scripting in a shell rather than in Python. If we go with your 
suggestion, we'd probably also want to have the ability to suppress those 
prompts for folks that want to shell script.

On Jun 16, 2015, at 4:42 PM, Keith Bray 
 wrote:

> Isn't that what the SDK is for?   To chip in with a Product Management type 
> hat on, I'd think the CLI should be primarily focused on user experience 
> interaction, and the SDK should be primarily targeted for developer 
> automation needs around programmatically interacting with the service.   So, 
> I would argue that the target market for the CLI should not be the developer 
> who wants to script.
> 
> -Keith
> 
> From: Adrian Otto 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Tuesday, June 16, 2015 12:24 PM
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Subject: Re: [openstack-dev] [Solum] Should logs be deleted when we delete an 
> app?
> 
>> Interactive choices like that one can make it more confusing for developers 
>> who want to script with the CLI. My preference would be to label the app 
>> delete help text to clearly indicate that it deletes logs
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Solum] Should logs be deleted when we delete an app?

[Randall Burt]

+1 Murali. AFIAK, there is no precedent for what Keith proposes, but that 
doesn't mean its a bad thing.

On Jun 16, 2015, at 12:21 AM, Murali Allada  wrote:

> I agree, users should have a mechanism to keep logs around.
> 
> I implemented the logs deletion feature after we got a bunch of requests from 
> users to delete logs once they delete an app, so they don't get charged for 
> storage once the app is deleted.
> 
> My implementation deletes the logs by default and I think that is the right 
> behavior. Based on user requests, that is exactly what they were asking for. 
> I'm planning to add a --keep-logs flag in a follow up patch. The command will 
> look as follows
> 
> Solum delete app MyApp --keep-logs
> 
> -Murali
> 
> 
> 
> 
> 
> On Jun 15, 2015, at 11:19 PM, Keith Bray  wrote:
> 
>> Regardless of what the API defaults to, could we have the CLI prompt/warn so 
>> that the user easily knows that both options exist?  Is there a precedent 
>> within OpenStack for a similar situation?
>> 
>> E.g. 
>> > solum app delete MyApp
>>  Do you want to also delete your logs? (default is Yes):  [YES/no]
>>   NOTE, if you choose No, application logs will remain on your 
>> account. Depending on your service provider, you may incur on-going storage 
>> charges.  
>> 
>> Thanks,
>> -Keith
>> 
>> From: Devdatta Kulkarni 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Date: Monday, June 15, 2015 9:56 AM
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Subject: Re: [openstack-dev] [Solum] Should logs be deleted when we delete 
>> an app?
>> 
>> Yes, the log deletion should be optional.
>> 
>> 
>> The question is what should be the default behavior. Should the default be 
>> to delete the logs and provide a flag to keep them, or keep the logs by 
>> default and provide a override flag to delete them?
>> 
>> Delete-by-default is consistent with the view that when an app is deleted, 
>> all its artifacts are deleted (the app's meta data, the deployment units 
>> (DUs), and the logs). This behavior is also useful in our current state when 
>> the app resource and the CLI are in flux. For now, without a way to specify 
>> a flag, either to delete the logs or to keep them, delete-by-default 
>> behavior helps us clean all the log files from the application's cloud files 
>> container when an app is deleted.
>> 
>> This is very useful for our CI jobs. Without this, we end up with lots of 
>> log files in the application's container,
>> 
>> and have to resort to separate scripts to delete them up after an app is 
>> deleted.
>> 
>> 
>> Once the app resource and CLI stabilize it should be straightforward to 
>> change the default behavior if required.
>> 
>> - Devdatta
>> 
>> From: Adrian Otto 
>> Sent: Friday, June 12, 2015 6:54 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: [openstack-dev] [Solum] Should logs be deleted when we delete an 
>> app?
>>  
>> Team,
>> 
>> We currently delete logs for an app when we delete the app[1]. 
>> 
>> https://bugs.launchpad.net/solum/+bug/1463986
>> 
>> Perhaps there should be an optional setting at the tenant level that 
>> determines whether your logs are deleted or not by default (set to off 
>> initially), and an optional parameter to our DELETE calls that allows for 
>> the opposite action from the default to be specified if the user wants to 
>> override it at the time of the deletion. Thoughts?
>> 
>> Thanks,
>> 
>> Adrian
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] COMMERCIAL: [heat] Stack/Resource updated_at conventions

[Randall Burt]

2 sounds right to me, but does the in-memory representation get updated or are 
we forced into a refetch at every change?

On Apr 27, 2015, at 10:46 AM, Steven Hardy  wrote:

> Hi all,
> 
> I've been looking into $subject recently, I raised this bug:
> 
> https://bugs.launchpad.net/heat/+bug/1448155
> 
> Basically we've got some historically weird and potentially inconsistent
> behavior around updated_at, and I'm trying to figure out the best way to
> proceed.
> 
> Now, we selectively update updated_at only on the transition to
> UPDATE_COMPLETE, where we store the time that we moved into
> UPDATE_IN_PROGRESS.  During the update, there's no way to derive the
> time we started the update.
> 
> Also, we inconsistently store the time associated with the transition into
> IN_PROGRESS for suspend, resume, snapshot, restore and check actions (even
> though many of these don't modify the stack definition).
> 
> The reason I need this is the hook/breakpoint API - the only way to detect
> if you've hit a breakpoint is via events, and to detect you've hit a hook
> during multiple sequential updates (some of which may fail or time out with
> hooks pending), you need to filter the events to only consider those with a
> timestamp newer than the transition of the stack to the update IN_PROGRESS.
> 
> AFAICT there's two options:
> 
> 1. Update the stack.Stack so we store "now" at every transition (e.g in
> state_set)
> 
> 2. Stop trying to explicitly control updated_at, and just allow the oslo
> TimestampMixin to do it's job and update updated_at every time the DB model
> is updated.
> 
> What are peoples thoughts?  Either will solve my problem, but I'm leaning
> towards (2) as the cleanest and most technically correct solution.
> 
> Similar problems exist for resource.Resource AFAICT.
> 
> Steve
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Stepping down from core

[Randall Burt]

Sorry to see you go. Thanks so much for your contribution and best of luck on 
your new project!

On Feb 27, 2015, at 2:23 PM, Jeff Peeler 
 wrote:

> As discussed during the previous Heat meeting, I'm going to be stepping down 
> from core on the Heat project. My day to day focus is going to be more 
> focused on TripleO for the foreseeable future, and I hope to be able to soon 
> focus on reviews there.
> 
> Being part of Heat core since day 0 has been a good experience, but keeping 
> up with multiple projects is a lot to manage. I don't know how some of you do 
> it!
> 
> Jeff
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] core team changes

[Randall Burt]

+1

On Jan 27, 2015, at 7:36 PM, Angus Salkeld 
 wrote:

> Hi all
> 
> After having a look at the stats:
> http://stackalytics.com/report/contribution/heat-group/90
> http://stackalytics.com/?module=heat-group&metric=person-day
> 
> I'd like to propose the following changes to the Heat core team:
> 
> Add:
> Qiming Teng
> Huang Tianhua
> 
> Remove:
> Bartosz Górski (Bartosz has indicated that he is happy to be removed and 
> doesn't have the time to work on heat ATM).
> 
> Core team please respond with +/- 1.
> 
> Thanks
> Angus
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How can I write at milestone section of blueprint?

[Randall Burt]

Its been discussed at several summits. We have settled on a general solution 
using Zaqar, but no work has been done that I know of. I was just pointing out 
that similar blueprints/specs exist and you may want to look through those to 
get some ideas about writing your own and/or basing your proposal off of one of 
them.

On Dec 22, 2014, at 12:19 AM, Yasunori Goto 
 wrote:

> Rundal-san,
> 
>> There should already be blueprints in launchpad for very similar 
>> functionality.
>> For example: https://blueprints.launchpad.net/heat/+spec/lifecycle-callbacks.
>> While that specifies Heat sending notifications to the outside world,
>> there has been discussion around debugging that would allow the receiver to
>> send notifications back. I only point this out so you can see there should be
>> similar blueprints and specs that you can reference and use as examples.
> 
> Thank you for pointing it out.
> But do you know current status about it?
> Though the above blueprint is not approved, and it seems to be discarded.
> 
> Bye,
> 
>> 
>> On Dec 19, 2014, at 4:17 AM, Steven Hardy 
>> wrote:
>> 
>>> On Fri, Dec 19, 2014 at 05:02:04PM +0900, Yasunori Goto wrote:
 
 Hello,
 
 This is the first mail at Openstack community,
>>> 
>>> Welcome! :)
>>> 
 and I have a small question about how to write blueprint for Heat.
 
 Currently our team would like to propose 2 interfaces
 for users operation in HOT. 
 (One is "Event handler" which is to notify user's defined event to heat.
 Another is definitions of action when heat catches the above notification.)
 So, I'm preparing the blueprint for it.
>>> 
>>> Please include details of the exact use-case, e.g the problem you're trying
>>> to solve (not just the proposed solution), as it's possible we can suggest
>>> solutions based on exiting interfaces.
>>> 
 However, I can not find how I can write at the milestone section of 
 blueprint.
 
 Heat blueprint template has a section for Milestones.
 "Milestones -- Target Milestone for completeion:"
 
 But I don't think I can decide it by myself.
 In my understanding, it should be decided by PTL.
>>> 
>>> Normally, it's decided by when the person submitting the spec expects to
>>> finish writing the code by.  The PTL doesn't really have much control over
>>> that ;)
>>> 
 In addition, probably the above our request will not finish
 by Kilo. I suppose it will be "L" version or later.
>>> 
>>> So to clarify, you want to propose the feature, but you're not planning on
>>> working on it (e.g implementing it) yourself?
>>> 
 So, what should I write at this section?
 "Kilo-x", "L version", or empty?
>>> 
>>> As has already been mentioned, it doesn't matter that much - I see it as a
>>> statement of intent from developers.  If you're just requesting a feature,
>>> you can even leave it blank if you want and we'll update it when an
>>> assignee is found (e.g during the spec review).
>>> 
>>> Thanks,
>>> 
>>> Steve
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> -- 
> Yasunori Goto 
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How can I write at milestone section of blueprint?

[Randall Burt]

There should already be blueprints in launchpad for very similar functionality. 
For example: https://blueprints.launchpad.net/heat/+spec/lifecycle-callbacks. 
While that specifies Heat sending notifications to the outside world, there has 
been discussion around debugging that would allow the receiver to send 
notifications back. I only point this out so you can see there should be 
similar blueprints and specs that you can reference and use as examples.

On Dec 19, 2014, at 4:17 AM, Steven Hardy 
 wrote:

> On Fri, Dec 19, 2014 at 05:02:04PM +0900, Yasunori Goto wrote:
>> 
>> Hello,
>> 
>> This is the first mail at Openstack community,
> 
> Welcome! :)
> 
>> and I have a small question about how to write blueprint for Heat.
>> 
>> Currently our team would like to propose 2 interfaces
>> for users operation in HOT. 
>> (One is "Event handler" which is to notify user's defined event to heat.
>> Another is definitions of action when heat catches the above notification.)
>> So, I'm preparing the blueprint for it.
> 
> Please include details of the exact use-case, e.g the problem you're trying
> to solve (not just the proposed solution), as it's possible we can suggest
> solutions based on exiting interfaces.
> 
>> However, I can not find how I can write at the milestone section of 
>> blueprint.
>> 
>> Heat blueprint template has a section for Milestones.
>> "Milestones -- Target Milestone for completeion:"
>> 
>> But I don't think I can decide it by myself.
>> In my understanding, it should be decided by PTL.
> 
> Normally, it's decided by when the person submitting the spec expects to
> finish writing the code by.  The PTL doesn't really have much control over
> that ;)
> 
>> In addition, probably the above our request will not finish
>> by Kilo. I suppose it will be "L" version or later.
> 
> So to clarify, you want to propose the feature, but you're not planning on
> working on it (e.g implementing it) yourself?
> 
>> So, what should I write at this section?
>> "Kilo-x", "L version", or empty?
> 
> As has already been mentioned, it doesn't matter that much - I see it as a
> statement of intent from developers.  If you're just requesting a feature,
> you can even leave it blank if you want and we'll update it when an
> assignee is found (e.g during the spec review).
> 
> Thanks,
> 
> Steve
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Rework auto-scaling support in Heat

[Randall Burt]

Per our discussion in Paris, I'm partial to Option B. I think a separate API 
endpoint is a lower priority at this point compared to cleaning up and 
normalizing the autoscale code on the back-end. Once we've refactored the 
engine code and solidified the RPC interface, it would be trivial to add an API 
on top of it. Additionally, we could even keep the privileged RPC interface for 
the Heat AS resources (assuming they stick around in some form) as an option 
for deployers. While certainly disruptive, I think we can handle this in small 
and/or isolated enough chances that reviews shouldn't be too difficult, 
especially if its possible to take the existing code largely unchanged at first 
and wrap an RPC abstraction around it.

On Nov 28, 2014, at 1:33 AM, Qiming Teng 
 wrote:

> Dear all,
> 
> Auto-Scaling is an important feature supported by Heat and needed by
> many users we talked to.  There are two flavors of AutoScalingGroup
> resources in Heat today: the AWS-based one and the Heat native one.  As
> more requests coming in, the team has proposed to separate auto-scaling
> support into a separate service so that people who are interested in it
> can jump onto it.  At the same time, Heat engine (especially the resource
> type code) will be drastically simplified.  The separated AS service
> could move forward more rapidly and efficiently.
> 
> This work was proposed a while ago with the following wiki and
> blueprints (mostly approved during Havana cycle), but the progress is
> slow.  A group of developers now volunteer to take over this work and
> move it forward.
> 
> wiki: https://wiki.openstack.org/wiki/Heat/AutoScaling
> BPs:
> - https://blueprints.launchpad.net/heat/+spec/as-lib-db
> - https://blueprints.launchpad.net/heat/+spec/as-lib
> - https://blueprints.launchpad.net/heat/+spec/as-engine-db
> - https://blueprints.launchpad.net/heat/+spec/as-engine
> - https://blueprints.launchpad.net/heat/+spec/autoscaling-api
> - https://blueprints.launchpad.net/heat/+spec/autoscaling-api-client
> - https://blueprints.launchpad.net/heat/+spec/as-api-group-resource
> - https://blueprints.launchpad.net/heat/+spec/as-api-policy-resource
> - https://blueprints.launchpad.net/heat/+spec/as-api-webhook-trigger-resource
> - https://blueprints.launchpad.net/heat/+spec/autoscaling-api-resources
> 
> Once this whole thing lands, Heat engine will talk to the AS engine in
> terms of ResourceGroup, ScalingPolicy, Webhooks.  Heat engine won't care
> how auto-scaling is implemented although the AS engine may in turn ask
> Heat to create/update stacks for scaling's purpose.  In theory, AS
> engine can create/destroy resources by directly invoking other OpenStack
> services.  This new AutoScaling service may eventually have its own DB,
> engine, API, api-client.  We can definitely aim high while work hard on
> real code.
> 
> After reviewing the BPs/Wiki and some communication, we get two options
> to push forward this.  I'm writing this to solicit ideas and comments
> from the community.
> 
> Option A: Top-Down Quick Split
> --
> 
> This means we will follow a roadmap shown below, which is not 100% 
> accurate yet and very rough:
> 
>  1) Get the separated REST service in place and working
>  2) Switch Heat resources to use the new REST service
> 
> Pros:
>  - Separate code base means faster review/commit cycle
>  - Less code churn in Heat
> Cons:
>  - A new service need to be installed/configured/launched
>  - Need commitments from dedicated, experienced developers from very
>beginning
> 
> Option B: Bottom-Up Slow Growth
> ---
> 
> The roadmap is more conservative, with many (yes, many) incremental
> patches to migrate things carefully.
> 
>  1) Separate some of the autoscaling logic into libraries in Heat
>  2) Augment heat-engine with new AS RPCs
>  3) Switch AS related resource types to use the new RPCs
>  4) Add new REST service that also talks to the same RPC
> (create new GIT repo, API endpoint and client lib...) 
> 
> Pros:
>  - Less risk breaking user lands with each revision well tested
>  - More smooth transition for users in terms of upgrades
> 
> Cons:
>  - A lot of churn within Heat code base, which means long review cycles
>  - Still need commitments from cores to supervise the whole process
> 
> There could be option C, D... but the two above are what we came up with
> during the discussion.
> 
> Another important thing we talked about is about the open discussion on
> this.  OpenStack Wiki seems a good place to document settled designs but
> not for interactive discussions.  Probably we should leverage etherpad
> and the mailinglist when moving forward.  Suggestions on this are also
> welcomed.
> 
> Thanks.
> 
> Regards,
> Qiming
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_

Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

[Randall Burt]

On Nov 12, 2014, at 10:42 AM, Zane Bitter 
 wrote:

> On 12/11/14 10:10, Clint Byrum wrote:
>> Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
>>> On 11/11/14 13:34, Ryan Brown wrote:
 I am strongly against allowing arbitrary Javascript functions for
 complexity reasons. It's already difficult enough to get meaningful
 errors when you  up your YAML syntax.
>>> 
>>> Agreed, and FWIW literally everyone that Clint has pitched the JS idea
>>> to thought it was crazy ;)
>>> 
>> 
>> So far nobody has stepped up to defend me,
> 
> I'll defend you, but I can't defend the idea :)
> 
>> so I'll accept that maybe
>> people do think it is crazy. What I'm really confused by is why we have
>> a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
> 
> Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is 
> probably not the way to win over Python developers :D
> 
>> and that would somehow be less crazy than a well known mature language
>> that has always been meant for embedding such as javascript.
> 
> JS is a Turing-complete language, it's an entirely different kettle of fish 
> to a domain-specific language that is inherently safe to interpret from user 
> input. Sure, we can try to lock it down. It's a very tricky job to get right. 
> (Plus it requires a new external dependency of unknown quality... honestly if 
> you're going to embed a Turing-complete language, Python is a much more 
> obvious choice than JS.)
> 
>> Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
>> HOT just right. Users will want to do things we don't expect. I say, let
>> them, or large sections of the users will simply move on to something
>> else.
> 
> The other side of that argument is that users are doing one of two things 
> with data they have obtained from resources in the template:
> 
> 1) Passing data to software deployments
> 2) Passing data to other resources
> 
> In case (1) they can easily transform the data into whatever format they want 
> using their own scripts, running on their own server.
> 
> In case (2), if it's not easy for them to just do what they want without 
> having to perform this kind of manipulation, we have failed to design good 
> resources. And if we give people the tools to just paper over the problem, 
> we'll never hear about it so we can correct it at the source, just launch a 
> thousand hard-to-maintain hacks into the world.

I disagree with this last bit. Having some manner of data manipulation facility 
as part of the template language doesn't mean the resources are badly designed 
or implemented, it just means that users have constraints that we don't 
directly address. Having something more general purpose (whatever it may be) 
seems like a longer term and more maintainable solution to me than eternal tail 
chasing where 1000 specific intrinsic functions bloom. IMO, intrinsics have a 
place, but supporting something like YAQL (or whatever) seems like a general 
solution to a lot of common problems our users come up against.

As for "hard-to-maintain hacks", isn't that what drives a lot of our users to 
ask for these sorts of solutions in the first place? If doing something via 
YAQL is perceived by a user as some sub-optimal hack for their use case, I see 
no reason for that to be any less visible than any other pain point or 
usability issue we have.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

[Randall Burt]

I like this approach and seems to have greater utility beyond the original 
proposal. I'm not fussed on YAQL or straight up JSONPath, but something along 
those lines seems to make sense.

On Nov 11, 2014, at 9:35 AM, Alexis Lee 
 wrote:

> Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +:
>> How about we support YAQL expressions? https://github.com/ativelkov/yaql
>> Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
>> etc?
> 
> We could also use YAQL to provide the HOFs.
> 
>> Here's first_nonnull:
>> 
>>  config:
>>Fn::Select
>>  - 0
>>  filter:
>>- yaql: "$.0 != null"
>>- 
>>- 
> 
>  config:
>yaql: "$[$ != null][0]"
>- 
>- 
> 
> This approach requires less change to Heat, at the price of learning
> more YAQL.
> 
> 
> Alexis
> -- 
> Nova Engineer, HP Cloud.  AKA lealexis, lxsli.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] stevedore plugins (and wait conditions)

[Randall Burt]

On Jul 10, 2014, at 9:21 AM, Zane Bitter 
 wrote:

> On 10/07/14 05:34, Steven Hardy wrote:
 > >The other approach is to set up a new container, owned by the user, 
 > >every time. In that case, a provider selecting this implementation 
 > >would need to make it clear to customers if they would be billed for a 
 > >WaitCondition resource. I'd prefer to avoid this scenario though 
 > >(regardless of the plug-point).
>>> >
>>> >Why? If we won't let the user choose, then why wouldn't we let the 
>>> >provider make this choice? I don't think its wise of us to make decisions 
>>> >based on what a theoretical operator may theoretically do. If the same 
>>> >theoretical provider were to also charge users to create a trust, would we 
>>> >then be concerned about that implementation as well? What if said provider 
>>> >decides charges the user per resource in a stack regardless of what they 
>>> >are? Having Heat own the container(s) as suggested above doesn't preclude 
>>> >that operator from charging the stack owner for those either.
>>> >
>>> >While I agree that these examples are totally silly, I'm just trying to 
>>> >illustrate that we shouldn't deny an operator an option so long as its 
>>> >understood what that option entails from a technical/usage perspective.
>> I don't really get why this question is totally silly - I made a genuine
>> request for education based on near-zero knowledge of public cloud provider
>> pricing models.
> 
> The way I read it Randall was not saying that the question was silly, he was 
> acknowledging that his own examples (like charging per-resource) were 
> contrived (to the point of absurdity) to illustrate his argument.

Yes. I didn't mean to imply the questions or any of the responses were silly, 
only my contrived examples.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] stevedore plugins (and wait conditions)

[Randall Burt]

On Jul 9, 2014, at 4:38 PM, Zane Bitter 
 wrote:
> On 08/07/14 17:17, Steven Hardy wrote:
> 
>> Regarding forcing deployers to make a one-time decision, I have a question
>> re cost (money and performance) of the Swift approach vs just hitting the
>> Heat API
>> 
>> - If folks use the Swift resource and it stores data associated with the
>>   signal in Swift, does that incurr cost to the user in a public cloud
>>   scenario?
> 
> Good question. I believe the way WaitConditions work in AWS is that it sets 
> up a pre-signed URL in a bucket owned by CloudFormation. If we went with that 
> approach we would probably want some sort of quota, I imagine.

Just to clarify, you suggest that the swift-based signal mechanism use 
containers that Heat owns rather than ones owned by the user?

> The other approach is to set up a new container, owned by the user, every 
> time. In that case, a provider selecting this implementation would need to 
> make it clear to customers if they would be billed for a WaitCondition 
> resource. I'd prefer to avoid this scenario though (regardless of the 
> plug-point).

Why? If we won't let the user choose, then why wouldn't we let the provider 
make this choice? I don't think its wise of us to make decisions based on what 
a theoretical operator may theoretically do. If the same theoretical provider 
were to also charge users to create a trust, would we then be concerned about 
that implementation as well? What if said provider decides charges the user per 
resource in a stack regardless of what they are? Having Heat own the 
container(s) as suggested above doesn't preclude that operator from charging 
the stack owner for those either.

While I agree that these examples are totally silly, I'm just trying to 
illustrate that we shouldn't deny an operator an option so long as its 
understood what that option entails from a technical/usage perspective.

>> - What sort of overhead are we adding, with the signals going to swift,
>>   then in the current implementation being copied back into the heat DB[1]?
> 
> I wasn't aware we were doing that, and I'm a bit unsure about it myself. I 
> don't think it's a big overhead, though.

In the current implementation, I think it is minor as well, just a few extra 
Swift API calls which should be pretty minor overhead considering the stack as 
a whole. Plus, it minimizes the above concern around potentially costly user 
containers in that it gets rid of them as soon as its done.

>> It seems to me at the moment that the swift notification method is good if
>> you have significant data associated with the signals, but there are
>> advantages to the simple API signal approach I've been working on when you
>> just need a simple "one shot" low overhead way to get data back from an
>> instance.
>> 
>> FWIW, the reason I revived these patches was I found that
>> SoftwareDeployments did not meet my needs for a really simple signalling
>> mechanism when writing tempest tests:
>> 
>> https://review.openstack.org/#/c/90143/16/tempest/scenario/orchestration/test_volumes_create_from_backup.yaml
>> 
>> These tests currently use the AWS WaitCondition resources, and I wanted a
>> native alternative, without the complexity of using SoftwareDeployments
>> (which also won't work with minimal cirros images without some pretty hacky
>> workarounds[2])
> 
> Yep, I am all for this. I think that Swift is the best way when we have it, 
> but not every cloud has Swift (and the latest rumours from DefCore are that 
> it's likely to stay that way), so we need operators (& developers!) to be 
> able to plug in an alternative implementation.

Very true, but not every cloud has trusts either. Many may have trusts, but 
they don't employ the EC2 extensions to Keystone and therefore can't use the 
"native" signals either (as I understand them anyway). Point being that either 
way, we already impose requirements on a cloud you want to run Heat against. I 
think it in our interest to make the effort to provide choices with obvious 
trade-offs.

>> I'm all for making things simple, avoiding duplication and confusion for
>> users, but I'd like to ensure that making this a one-time deployer level
>> decision definitely makes sense, vs giving users some choice over what
>> method is used.
> 
> Agree, this is an important question to ask. The downside to leaving the 
> choice to the user is that it reduces interoperability between clouds. (In 
> fact, it's unclear whether operators _would_ give users a choice, or just 
> deploy one implementation anyway.) It's not insurmountable (thanks to 
> environments), but it does add friction to the ecosystem so we have to weigh 
> up the trade-offs.

Agreed that this is an important concern, but one of mine is that no other 
resource has "selectable" back-ends. The way an operator controls this today is 
via the global environment where they have the option to disable one or more of 
these resources or even alias one to the other. Seems a large

Re: [openstack-dev] [Heat] stevedore plugins (and wait conditions)

[Randall Burt]

On Jul 9, 2014, at 3:15 PM, Zane Bitter 
 wrote:

> On 08/07/14 17:13, Angus Salkeld wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>> 
>> On 08/07/14 09:14, Zane Bitter wrote:
>>> I see that the new client plugins are loaded using stevedore, which is
>>> great and IMO absolutely the right tool for that job. Thanks to Angus &
>>> Steve B for implementing it.
>>> 
>>> Now that we have done that work, I think there are more places we can
>>> take advantage of it too - for example, we currently have competing
>>> native wait condition resource types being implemented by Jason[1] and
>>> Steve H[2] respectively, and IMHO that is a mistake. We should have
>>> *one* native wait condition resource type, one AWS-compatible one,
>>> software deployments and any custom plugin that require signalling; and
>>> they should all use a common SignalResponder implementation that would
>>> call an API that is pluggable using stevedore. (In summary, what we're
>> 
>> what's wrong with using the environment for that? Just have two resources
>> and you do something like this:
>> https://github.com/openstack/heat/blob/master/etc/heat/environment.d/default.yaml#L7
> 
> It doesn't cover other things that need signals, like software deployments 
> (third-party plugin authors are also on their own). We only want n 
> implementations not n*(number of resources that use signals) implementations.
> 
>>> trying to make configurable is an implementation that should be
>>> invisible to the user, not an interface that is visible to the user, and
>>> therefore the correct unit of abstraction is an API, not a resource.)
>>> 
>> 
>> Totally depends if we want this to be operator configurable (config file or 
>> plugin)
>> or end user configurable (use their environment to choose the 
>> implementation).
>> 
>>> 
>>> I just noticed, however, that there is an already-partially-implemented
>>> blueprint[3] and further pending patches[4] to use stevedore for *all*
>>> types of plugins - particularly resource plugins[5] - in Heat. I feel
>>> very strongly that stevedore is _not_ a good fit for all of those use
>>> cases. (Disclaimer: obviously I _would_ think that, since I implemented
>>> the current system instead of using stevedore for precisely that reason.)
>> 
>> haha.
>> 
>>> 
>>> The stated benefit of switching to stevedore is that it solves issues
>>> like https://launchpad.net/bugs/1292655 that are caused by the current
>>> convoluted layout of /contrib. I think the layout stems at least in part
>> 
>> I think another great reason is consistency with how all other plugins are 
>> openstack
>> are written (stevedore).
> 
> Sure, consistency is nice, sometimes even at the expense of being not quite 
> the right tool for the job. But there are limits to that trade-off.
> 
>> Also I *really* don't think we should optimize for our contrib plugins
>> but for:
>> 1) our built in plugins
>> 2) out of tree plugins
> 
> I completely agree, which is why I was surprised by this change. It seems to 
> be deprecating a system that is working well for built-in and out-of-tree 
> plugins in order to make minor improvements to how we handle contrib.

FWIW, when it comes to deploying Heat with non-built-in, there's no substantive 
difference in the experience between contrib and out-of-tree plugins, so 
neither system is more or less optimized for either. However, with the current 
system, there's no "easy" way to get rid of the "built-in" ones you don't want.

> 
>>> from a misunderstanding of how the current plugin_manager works. The
>>> point of the plugin_manager is that each plugin directory does *not*
>>> have to be a Python package - it can be any directory. Modules in the
>>> directory then appear in the package heat.engine.plugins once imported.
>>> So there is no need to do what we are currently doing, creating a
>>> resources package, and then a parent package that contains the tests
>>> package as well, and then in the tests doing:
>>> 
>>>from ..resources import docker_container  ## noqa
>>> 
>>> All we really need to do is throw the resources in any old directory,
>>> add that directory to the plugin_dirs list, stick the tests in any old
>>> package, and from the tests do
>>> 
>>>from heat.engine.plugins import docker_container
>>> 
>>> The main reason we haven't done this seems to be to avoid having to list
>>> the various contrib plugin dirs separately. Stevedore "solves" this by
>>> forcing us to list not only each directory but each class in each module
>>> in each directory separately. The tricky part of fixing the current
>>> layout is ensuring the contrib plugin directories get added to the
>>> plugin_dirs list during the unit tests and only during the unit tests.
>>> However, I'm confident that could be fixed with no more difficulty than
>>> the stevedore changes and with far less disruption to existing operators
>>> using custom plugins.
>>> 
>>> Stevedore is ideal for configuring an implementation for a small nu

Re: [openstack-dev] [Ceilometer] [Heat] Ceilometer aware people, please advise us on processing notifications..

[Randall Burt]

On Jun 26, 2014, at 5:25 PM, Zane Bitter 
 wrote:

> On 23/06/14 19:25, Clint Byrum wrote:
>> Hello! I would like to turn your attention to this specification draft
>> that I've written:
>> 
>> https://review.openstack.org/#/c/100012/1/specs/convergence-continuous-observer.rst
>> 
>> Angus has suggested that perhaps Ceilometer is a better place to handle
>> this. Can you please comment on the review, or can we have a brief
>> mailing list discussion about how best to filter notifications?
>> 
>> Basically in Heat when a user boots an instance, we would like to act as
>> soon as it is active, and not have to poll the nova API to know when
>> that is. Angus has suggested that perhaps we can just tell ceilometer to
>> hit Heat with a web hook when that happens.
> 
> I'm all in favour of having Ceilometer filter the firehose for us if we can :)
> 
> Webhooks would seem to add a lot of overhead though (set up + tear down a 
> connection for every notification), that could perhaps be avoided by using a 
> message bus? Given that both setting up and receiving these notifications 
> would be admin-only operations, is there any benefit to handling them through 
> a webhook API rather than through oslo.messaging?
> 
> cheers,
> Zane.

In larger OpenStack deployments, the different services probably don't share 
the same message bus. While I certainly agree oslo.messaging and/or 
oslo.notifications should be an option (and probably the default one at that), 
I think there should still be an option to use ceilometer or some other 
notification mechanism. As long as its pluggable, I don't think anyone would be 
too fussed.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Sergey Kraynev for heat-core

[Randall Burt]

On Jun 26, 2014, at 5:08 PM, Steve Baker  wrote:

> I'd like to nominate Sergey Kraynev for heat-core. His reviews are
> valuable and prolific, and his commits have shown a sound understanding
> of heat internals.
> 
> http://stackalytics.com/report/contribution/heat-group/60

+1


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] fine grained quotas

[Randall Burt]

On Jun 19, 2014, at 4:17 PM, Clint Byrum  wrote:

> I was made aware of the following blueprint today:
> 
> http://blueprints.launchpad.net/heat/+spec/add-quota-api-for-heat
> http://review.openstack.org/#/c/96696/14
> 
> Before this goes much further.. I want to suggest that this work be
> cancelled, even though the code looks excellent. The reason those limits
> are in the config file is that these are not billable items and they
> have a _tiny_ footprint in comparison to the physical resources they
> will allocate in Nova/Cinder/Neutron/etc.
> 
> IMO we don't need fine grained quotas in Heat because everything the
> user will create with these templates will cost them and have its own
> quota system. The limits (which I added) are entirely to prevent a DoS
> of the engine.

What's more, I don't think this is something we should expose via API other 
than to perhaps query what those quota values are. It is possible that some 
provider would want to bill on number of stacks, etc (I personally agree with 
Clint here), it seems that is something that could/should be handled external 
to Heat itself.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [HEAT] Discussion: How to list nested stack resources.

[Randall Burt]

I've submitted the spec (finally) and will work on some initial patches this 
afternoon/tomorrow. Please provide any feedback and thanks!

https://review.openstack.org/#/c/98219

On Jun 5, 2014, at 11:35 AM, Randall Burt  wrote:

> Hey, sorry for the slow follow. I have to put some finishing touches on a 
> spec and submit that for review. I'll reply to the list with the link later 
> today. Hope to have an initial patch up as well in the next day or so.
> 
> On Jun 5, 2014, at 10:03 AM, Nilakhya Chatterjee 
> 
> wrote:
> 
>> HI Guys, 
>> 
>> It was gr8 to find your interest in solving the nested stack resource 
>> listing.
>> 
>> Lets move ahead by finishing any discussions left over the BP and getting an 
>> approval on It.
>> 
>> Till now what makes sense to me are : 
>> 
>> a) an additional flag in the client call  --nested (randall)
>> b) A flattened  DS in the output  (tim) 
>> 
>> 
>> Thanks all ! 
>> 
>> 
>> On Wed, May 21, 2014 at 12:42 AM, Randall Burt  
>> wrote:
>> Bartosz, would that be in addition to --nested? Seems like id want to be 
>> able to say "all of it" as well as "some of it".
>> 
>> On May 20, 2014, at 1:24 PM, Bartosz Górski 
>> wrote:
>> 
>>> Hi Tim,
>>> 
>>> Maybe instead of just a flag like --nested (bool value) to resource-list we 
>>> can add optional argument like --depth X or --nested-level X (X - integer 
>>> value) to limit the depth for recursive listing of nested resources?
>>> 
>>> Best,
>>> Bartosz
>>> 
>>> On 05/19/2014 09:13 PM, Tim Schnell wrote:
>>>> Blueprint:
>>>> https://blueprints.launchpad.net/heat/+spec/explode-nested-resources
>>>> 
>>>> Spec: https://wiki.openstack.org/wiki/Heat/explode-resource-list
>>>> 
>>>> Tim
>>>> 
>>>> On 5/19/14 1:53 PM, "Tim Schnell"  wrote:
>>>> 
>>>>> On 5/19/14 12:35 PM, "Randall Burt"  wrote:
>>>>> 
>>>>> 
>>>>>> On May 19, 2014, at 11:39 AM, Steven Hardy 
>>>>>> wrote:
>>>>>> 
>>>>>>> On Mon, May 19, 2014 at 03:26:22PM +, Tim Schnell wrote:
>>>>>>>> Hi Nilakhya,
>>>>>>>> 
>>>>>>>> As Randall mentioned we did discuss this exact issue at the summit. I
>>>>>>>> was
>>>>>>>> planning on putting a blueprint together today to continue the
>>>>>>>> discussion.
>>>>>>>> The Stack Preview call is already doing the necessary recursion to
>>>>>>>> gather
>>>>>>>> the resources so we discussed being able to pass a stack id to the
>>>>>>>> preview
>>>>>>>> endpoint to get all of the resources.
>>>>>>>> 
>>>>>>>> However, after thinking about it some more, I agree with Randall that
>>>>>>>> maybe this should be an extra query parameter passed to the
>>>>>>>> resource-list
>>>>>>>> call. I'Ll have the blueprint up later today, unless you have already
>>>>>>>> started on it.
>>>>>>> Note there is a patch from Anderson/Richard which may help with this:
>>>>>>> 
>>>>>>> https://review.openstack.org/#/c/85781/
>>>>>>> 
>>>>>>> The idea was to enable easier introspection of resources backed by
>>>>>>> nested
>>>>>>> stacks in a UI, but it could be equally useful to generate a "tree"
>>>>>>> resource view in the CLI client by walking the links.
>>>>>>> 
>>>>>>> This would obviously be less efficient than recursing inside the
>>>>>>> engine,
>>>>>>> but arguably the output would be much more useful if it retains the
>>>>>>> nesting
>>>>>>> structure, as opposed to presenting a fully flattened "soup" of
>>>>>>> resources
>>>>>>> with no idea which stack/layer they belong to.
>>>>>>> 
>>>>>>> Steve
>>>>>> Could we simply add stack name/id to this output if the flag is passed? I
>>>>>> agree that we currently have the capabilit

Re: [openstack-dev] [HEAT] Discussion: How to list nested stack resources.

[Randall Burt]

I have submitted a new/expanded spec for this feature: 
https://review.openstack.org/#/c/98219/. I hope to start some WiP patches this 
afternoon/tomorrow morning. Spec reviews and input most welcome.

On Jun 5, 2014, at 11:35 AM, Randall Burt  wrote:

> Hey, sorry for the slow follow. I have to put some finishing touches on a 
> spec and submit that for review. I'll reply to the list with the link later 
> today. Hope to have an initial patch up as well in the next day or so.
> 
> On Jun 5, 2014, at 10:03 AM, Nilakhya Chatterjee 
> 
> wrote:
> 
>> HI Guys, 
>> 
>> It was gr8 to find your interest in solving the nested stack resource 
>> listing.
>> 
>> Lets move ahead by finishing any discussions left over the BP and getting an 
>> approval on It.
>> 
>> Till now what makes sense to me are : 
>> 
>> a) an additional flag in the client call  --nested (randall)
>> b) A flattened  DS in the output  (tim) 
>> 
>> 
>> Thanks all ! 
>> 
>> 
>> On Wed, May 21, 2014 at 12:42 AM, Randall Burt  
>> wrote:
>> Bartosz, would that be in addition to --nested? Seems like id want to be 
>> able to say "all of it" as well as "some of it".
>> 
>> On May 20, 2014, at 1:24 PM, Bartosz Górski 
>> wrote:
>> 
>>> Hi Tim,
>>> 
>>> Maybe instead of just a flag like --nested (bool value) to resource-list we 
>>> can add optional argument like --depth X or --nested-level X (X - integer 
>>> value) to limit the depth for recursive listing of nested resources?
>>> 
>>> Best,
>>> Bartosz
>>> 
>>> On 05/19/2014 09:13 PM, Tim Schnell wrote:
>>>> Blueprint:
>>>> https://blueprints.launchpad.net/heat/+spec/explode-nested-resources
>>>> 
>>>> Spec: https://wiki.openstack.org/wiki/Heat/explode-resource-list
>>>> 
>>>> Tim
>>>> 
>>>> On 5/19/14 1:53 PM, "Tim Schnell"  wrote:
>>>> 
>>>>> On 5/19/14 12:35 PM, "Randall Burt"  wrote:
>>>>> 
>>>>> 
>>>>>> On May 19, 2014, at 11:39 AM, Steven Hardy 
>>>>>> wrote:
>>>>>> 
>>>>>>> On Mon, May 19, 2014 at 03:26:22PM +, Tim Schnell wrote:
>>>>>>>> Hi Nilakhya,
>>>>>>>> 
>>>>>>>> As Randall mentioned we did discuss this exact issue at the summit. I
>>>>>>>> was
>>>>>>>> planning on putting a blueprint together today to continue the
>>>>>>>> discussion.
>>>>>>>> The Stack Preview call is already doing the necessary recursion to
>>>>>>>> gather
>>>>>>>> the resources so we discussed being able to pass a stack id to the
>>>>>>>> preview
>>>>>>>> endpoint to get all of the resources.
>>>>>>>> 
>>>>>>>> However, after thinking about it some more, I agree with Randall that
>>>>>>>> maybe this should be an extra query parameter passed to the
>>>>>>>> resource-list
>>>>>>>> call. I'Ll have the blueprint up later today, unless you have already
>>>>>>>> started on it.
>>>>>>> Note there is a patch from Anderson/Richard which may help with this:
>>>>>>> 
>>>>>>> https://review.openstack.org/#/c/85781/
>>>>>>> 
>>>>>>> The idea was to enable easier introspection of resources backed by
>>>>>>> nested
>>>>>>> stacks in a UI, but it could be equally useful to generate a "tree"
>>>>>>> resource view in the CLI client by walking the links.
>>>>>>> 
>>>>>>> This would obviously be less efficient than recursing inside the
>>>>>>> engine,
>>>>>>> but arguably the output would be much more useful if it retains the
>>>>>>> nesting
>>>>>>> structure, as opposed to presenting a fully flattened "soup" of
>>>>>>> resources
>>>>>>> with no idea which stack/layer they belong to.
>>>>>>> 
>>>>>>> Steve
>>>>>> Could we simply add stack name/id to this output if the flag is passed? I
>>>>>> agree that we currently have the capability to 

Re: [openstack-dev] [HEAT] Discussion: How to list nested stack resources.

[Randall Burt]

Hey, sorry for the slow follow. I have to put some finishing touches on a spec 
and submit that for review. I'll reply to the list with the link later today. 
Hope to have an initial patch up as well in the next day or so.

On Jun 5, 2014, at 10:03 AM, Nilakhya Chatterjee 

 wrote:

> HI Guys, 
> 
> It was gr8 to find your interest in solving the nested stack resource listing.
> 
> Lets move ahead by finishing any discussions left over the BP and getting an 
> approval on It.
> 
> Till now what makes sense to me are : 
> 
> a) an additional flag in the client call  --nested (randall)
> b) A flattened  DS in the output  (tim) 
> 
> 
> Thanks all ! 
> 
> 
> On Wed, May 21, 2014 at 12:42 AM, Randall Burt  
> wrote:
> Bartosz, would that be in addition to --nested? Seems like id want to be able 
> to say "all of it" as well as "some of it".
> 
> On May 20, 2014, at 1:24 PM, Bartosz Górski 
>  wrote:
> 
> > Hi Tim,
> >
> > Maybe instead of just a flag like --nested (bool value) to resource-list we 
> > can add optional argument like --depth X or --nested-level X (X - integer 
> > value) to limit the depth for recursive listing of nested resources?
> >
> > Best,
> > Bartosz
> >
> > On 05/19/2014 09:13 PM, Tim Schnell wrote:
> >> Blueprint:
> >> https://blueprints.launchpad.net/heat/+spec/explode-nested-resources
> >>
> >> Spec: https://wiki.openstack.org/wiki/Heat/explode-resource-list
> >>
> >> Tim
> >>
> >> On 5/19/14 1:53 PM, "Tim Schnell"  wrote:
> >>
> >>> On 5/19/14 12:35 PM, "Randall Burt"  wrote:
> >>>
> >>>
> >>>> On May 19, 2014, at 11:39 AM, Steven Hardy 
> >>>> wrote:
> >>>>
> >>>>> On Mon, May 19, 2014 at 03:26:22PM +, Tim Schnell wrote:
> >>>>>> Hi Nilakhya,
> >>>>>>
> >>>>>> As Randall mentioned we did discuss this exact issue at the summit. I
> >>>>>> was
> >>>>>> planning on putting a blueprint together today to continue the
> >>>>>> discussion.
> >>>>>> The Stack Preview call is already doing the necessary recursion to
> >>>>>> gather
> >>>>>> the resources so we discussed being able to pass a stack id to the
> >>>>>> preview
> >>>>>> endpoint to get all of the resources.
> >>>>>>
> >>>>>> However, after thinking about it some more, I agree with Randall that
> >>>>>> maybe this should be an extra query parameter passed to the
> >>>>>> resource-list
> >>>>>> call. I'Ll have the blueprint up later today, unless you have already
> >>>>>> started on it.
> >>>>> Note there is a patch from Anderson/Richard which may help with this:
> >>>>>
> >>>>> https://review.openstack.org/#/c/85781/
> >>>>>
> >>>>> The idea was to enable easier introspection of resources backed by
> >>>>> nested
> >>>>> stacks in a UI, but it could be equally useful to generate a "tree"
> >>>>> resource view in the CLI client by walking the links.
> >>>>>
> >>>>> This would obviously be less efficient than recursing inside the
> >>>>> engine,
> >>>>> but arguably the output would be much more useful if it retains the
> >>>>> nesting
> >>>>> structure, as opposed to presenting a fully flattened "soup" of
> >>>>> resources
> >>>>> with no idea which stack/layer they belong to.
> >>>>>
> >>>>> Steve
> >>>> Could we simply add stack name/id to this output if the flag is passed? I
> >>>> agree that we currently have the capability to traverse the tree
> >>>> structure of nested stacks, but several folks have requested this
> >>>> capability, mostly for UI/UX purposes. It would be faster if you want the
> >>>> flat structure and we still retain the capability to create your own
> >>>> tree/widget/whatever by following the links. Also, I think its best to
> >>>> include this in the API directly since not all users are integrating
> >>>> using the python-heatclient.
> >>> +1 for adding the stack name/id to the output to maintain a reference to
> >>

Re: [openstack-dev] [Heat]Heat template parameters encryption

[Randall Burt]

On Jun 4, 2014, at 7:30 PM, Clint Byrum 
 wrote:

> Excerpts from Randall Burt's message of 2014-06-04 17:17:07 -0700:
>> On Jun 4, 2014, at 7:05 PM, Clint Byrum 
>> wrote:
>> 
>>> Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
 On 04/06/14 15:58, Vijendar Komalla wrote:
> Hi Devs,
> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
> for Heat parameters encryption blueprint
> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
> This quick and dirty implementation encrypts all the parameters on on
> Stack 'store' and decrypts on on Stack 'load'.
> Following are couple of improvements I am thinking about;
> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
> all the parameters together as a dictionary  [something like
> crypt.encrypt(json.dumps(param_dictionary))]
 
 Yeah, definitely don't encrypt them individually.
 
> 2. Just encrypt parameters that were marked as 'hidden', instead of
> encrypting all parameters
> 
> I would like to hear your feedback/suggestions.
 
 Just as a heads-up, we will soon need to store the properties of 
 resources too, at which point parameters become the least of our 
 problems. (In fact, in theory we wouldn't even need to store 
 parameters... and probably by the time convergence is completely 
 implemented, we won't.) Which is to say that there's almost certainly no 
 point in discriminating between hidden and non-hidden parameters.
 
 I'll refrain from commenting on whether the extra security this affords 
 is worth the giant pain it causes in debugging, except to say that IMO 
 there should be a config option to disable the feature (and if it's 
 enabled by default, it should probably be disabled by default in e.g. 
 devstack).
>>> 
>>> Storing secrets seems like a job for Barbican. That handles the giant
>>> pain problem because in devstack you can just tell Barbican to have an
>>> open read policy.
>>> 
>>> I'd rather see good hooks for Barbican than blanket encryption. I've
>>> worked with a few things like this and they are despised and worked
>>> around universally because of the reason Zane has expressed concern about:
>>> debugging gets ridiculous.
>>> 
>>> How about this:
>>> 
>>> parameters:
>>> secrets:
>>>   type: sensitive
>>> resources:
>>> sensitive_deployment:
>>>   type: OS::Heat::StructuredDeployment
>>>   properties:
>>> config: weverConfig
>>> server: myserver
>>> input_values:
>>>   secret_handle: { get_param: secrets }
>>> 
>>> The sensitive type would, on the client side, store the value in Barbican,
>>> never in Heat. Instead it would just pass in a handle which the user
>>> can then build policy around. Obviously this implies the user would set
>>> up Barbican's in-instance tools to access the secrets value. But the
>>> idea is, let Heat worry about being high performing and introspectable,
>>> and then let Barbican worry about sensitive things.
>> 
>> While certainly ideal, it doesn't solve the current problem since we can't 
>> yet guarantee Barbican will even be available in a given release of 
>> OpenStack. In the meantime, Heat continues to store sensitive user 
>> information unencrypted in its database. Once Barbican is integrated, I'd be 
>> all for changing this implementation, but until then, we do need an interim 
>> solution. Sure, debugging is a pain and as developers we can certainly 
>> grumble, but leaking sensitive user information because we were too fussed 
>> to protect data at rest seems worse IMO. Additionally, the solution as 
>> described sounds like we're imposing a pretty awkward process on a user to 
>> save ourselves from having to decrypt some data in the cases where we can't 
>> access the stack information directly from the API or via debugging running 
>> Heat code (where the data isn't encrypted anymore).
>> 
> 
> I have made that exact, reasoned argument before, and then later seen
> giant swathes of code with things like
> 
> if CONF.dont_encrypt:
>  ...
> 
> The next thing that happens is one by one the production deployments
> eventually end up with dont_encrypt because they can't debug anything
> and they've all had a multi-hour downtime event while they dealt with
> the encrypted database on several levels.
> 
> I'm not coddling developers. I'm coddling operators.
> 
> It's a different story if you only encrypt the sensitive "leaf" data,
> like passwords, credit card numbers, or personally identifying data. The
> operator can still have some clue what is going on if they can see the
> non-sensitive data.
> 
> The design I suggested above isn't really that awkward and will
> be infinitely easier to understand for an operator than encrypting
> everything.

So, you're simply advocating a more granular approach in which we only encrypt 
the values for inputs marked as hidden in the interim?
___

Re: [openstack-dev] [Heat]Heat template parameters encryption

[Randall Burt]

On Jun 4, 2014, at 7:05 PM, Clint Byrum 
 wrote:

> Excerpts from Zane Bitter's message of 2014-06-04 16:19:05 -0700:
>> On 04/06/14 15:58, Vijendar Komalla wrote:
>>> Hi Devs,
>>> I have submitted an WIP review (https://review.openstack.org/#/c/97900/)
>>> for Heat parameters encryption blueprint
>>> https://blueprints.launchpad.net/heat/+spec/encrypt-hidden-parameters
>>> This quick and dirty implementation encrypts all the parameters on on
>>> Stack 'store' and decrypts on on Stack 'load'.
>>> Following are couple of improvements I am thinking about;
>>> 1. Instead of encrypting individual parameters, on Stack 'store' encrypt
>>> all the parameters together as a dictionary  [something like
>>> crypt.encrypt(json.dumps(param_dictionary))]
>> 
>> Yeah, definitely don't encrypt them individually.
>> 
>>> 2. Just encrypt parameters that were marked as 'hidden', instead of
>>> encrypting all parameters
>>> 
>>> I would like to hear your feedback/suggestions.
>> 
>> Just as a heads-up, we will soon need to store the properties of 
>> resources too, at which point parameters become the least of our 
>> problems. (In fact, in theory we wouldn't even need to store 
>> parameters... and probably by the time convergence is completely 
>> implemented, we won't.) Which is to say that there's almost certainly no 
>> point in discriminating between hidden and non-hidden parameters.
>> 
>> I'll refrain from commenting on whether the extra security this affords 
>> is worth the giant pain it causes in debugging, except to say that IMO 
>> there should be a config option to disable the feature (and if it's 
>> enabled by default, it should probably be disabled by default in e.g. 
>> devstack).
> 
> Storing secrets seems like a job for Barbican. That handles the giant
> pain problem because in devstack you can just tell Barbican to have an
> open read policy.
> 
> I'd rather see good hooks for Barbican than blanket encryption. I've
> worked with a few things like this and they are despised and worked
> around universally because of the reason Zane has expressed concern about:
> debugging gets ridiculous.
> 
> How about this:
> 
> parameters:
>  secrets:
>type: sensitive
> resources:
>  sensitive_deployment:
>type: OS::Heat::StructuredDeployment
>properties:
>  config: weverConfig
>  server: myserver
>  input_values:
>secret_handle: { get_param: secrets }
> 
> The sensitive type would, on the client side, store the value in Barbican,
> never in Heat. Instead it would just pass in a handle which the user
> can then build policy around. Obviously this implies the user would set
> up Barbican's in-instance tools to access the secrets value. But the
> idea is, let Heat worry about being high performing and introspectable,
> and then let Barbican worry about sensitive things.

While certainly ideal, it doesn't solve the current problem since we can't yet 
guarantee Barbican will even be available in a given release of OpenStack. In 
the meantime, Heat continues to store sensitive user information unencrypted in 
its database. Once Barbican is integrated, I'd be all for changing this 
implementation, but until then, we do need an interim solution. Sure, debugging 
is a pain and as developers we can certainly grumble, but leaking sensitive 
user information because we were too fussed to protect data at rest seems worse 
IMO. Additionally, the solution as described sounds like we're imposing a 
pretty awkward process on a user to save ourselves from having to decrypt some 
data in the cases where we can't access the stack information directly from the 
API or via debugging running Heat code (where the data isn't encrypted anymore).


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [solum] reviews for the new API

[Randall Burt]

Sorry to poke my head in, but doesn't that beg the question of why you'd want 
to expose some third party DSL in the first place? If its an advanced feature, 
I wonder why it would be even considered before the 90% solution works, much 
less take a dependency on another non-integrated service. IMO, the value of 
Solum initially lies in addressing that 90% CI/CD/ALM solution well and in a 
way that doesn't require me to deploy and maintain an additional service that's 
not part of integrated OpenStack. At best, it would seem prudent to me to 
simply allow for a basic way for me to insert my own CI/CD steps and be 
prescriptive about how those custom steps participate in Solum's workflow 
rather than locking me into some specific workflow DSL. If I then choose to use 
Mistral to do some custom work, I can, but Solum shouldn't care what I use.

If Solum isn't fairly opinionated (at least early on) about the basic 
CI/CD->ALM lifecycle and the steps therein, I would question its utility if its 
a wrapper over my existing Jenkins jobs and a workflow service.

On Jun 4, 2014, at 1:10 PM, Julien Vey  wrote:

> Murali, Roshan.
> 
> I think there is a misunderstood. By default, the user wouldn't see any 
> "workflow" dsl. If the user does not specify anything, we would use a 
> pre-defined mistral workbook defined by Solum, as Adrian described
> 
> If the user needs more, mistral is not so complicated. Have a look at this 
> example Angus has done 
> https://review.openstack.org/#/c/95709/4/etc/solum/workbooks/build.yaml
> We can define anything as solum actions, and the users would just have to 
> call one of this actions. Solum takes care of the implementation. If we have 
> comments about the DSL, Mistral's team is willing to help.
> 
> Our end-users will be developers, and a lot of them will need a custom 
> workflow at some point. For instance, if Jenkins has so many plugins, it's 
> because there are as many custom build workflows, specific to each company. 
> If we add an abstraction on top of mistral or any other workflow engine, we 
> will lock developers in our own decisions, and any additional feature would 
> require a new development in Solum, whereas exposing (when users want it) 
> mistral, we would allow for any customization.
> 
> Julien
> 
> 
> 
> 
> 2014-06-04 19:50 GMT+02:00 Roshan Agrawal :
> Agreeing with what Murali said below. We should make things really simple for 
> the 99 percentile of the users, and not force the complexity needed by the 
> minority of the “advanced users” on the rest of the 99 percentile users.  
> 
>  
> 
> Mistral is a generic workflow DSL, we do not need to expose all that 
> complexity to the Solum user that wants to customize the pipeline. 
> “Non-advanced” users will have a need to customize the pipeline. In this 
> case, the user is not necessarily the developer persona, but typically an 
> admin/release manager persona.  
> 
>  
> 
> Pipeline customization should be doable easily, without having the understand 
> or author a generic workflow DSL.
> 
>  
> 
> For the really advanced user who needs to have a finer grained need to tweak 
> the mistral workflow DSL (I am not sure if there will be a use case for this 
> if we have the right customizations exposed via the pipeline API), we should 
> have the “option” for the user to tweak the mistral DSL directly, but we 
> should not expect 99.9% (or more) of the users to deal with a generic 
> workflow.
> 
>  
> 
>  
> 
> From: Murali Allada [mailto:murali.all...@rackspace.com] 
> Sent: Wednesday, June 04, 2014 12:09 PM
> 
> 
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [solum] reviews for the new API
> 
>  
> 
> Angus/Julien,
> 
>  
> 
> I would disagree that we should expose the mistral DSL to end users.
> 
>  
> 
> What if we decide to use something other than Mistral in the future? We 
> should be able to plug in any workflow system we want without changing what 
> we expose to the end user.
> 
>  
> 
> To me, the pipeline DSL is similar to our plan file. We don't expose a heat 
> template to our end users.
> 
>  
> 
> Murali
> 
>  
> 
>  
> 
>  
> 
> On Jun 4, 2014, at 10:58 AM, Julien Vey 
> 
>  wrote:
> 
> 
> 
> 
> Hi Angus,
> 
>  
> 
> I really agree with you. I would insist on #3, most of our users will use the 
> default workbook, and only advanced users will want to customize the 
> workflow. "advanced" users should easily understand a mistral workbook, cause 
> they are "advanced"
> 
>  
> 
> To add to the cons of creating our own DSL, it will require a lot more work, 
> more design discussions, more maintenance... We might end up doing what 
> mistral is already doing. If we have some difficulties with Mistral's DSL, we 
> can talk with the team, and contribute back our experience of using Mistral.
> 
>  
> 
> Julien
> 
>  
> 
>  
> 
>  
> 
>  
> 
> 2014-06-04 14:11 GMT+02:00 Angus Salkeld :
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> H

[openstack-dev] [Barbican][Heat] Reviews requested for Barbican resources

[Randall Burt]

Hello Barbican devs. I was wondering if we could get some of you to weigh in on 
a couple of reviews for adding Barbican support in Heat. We seem to be churning 
a bit around current and future features supported by the resources and could 
use some expert opinions.

Blueprint: https://blueprints.launchpad.net/heat/+spec/barbican-resources
Order Resource: https://review.openstack.org/81906
Secret Resource: https://review.openstack.org/79355

Thanks in advance for your time.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [HEAT] Discussion: How to list nested stack resources.

[Randall Burt]

Bartosz, would that be in addition to --nested? Seems like id want to be able 
to say "all of it" as well as "some of it".

On May 20, 2014, at 1:24 PM, Bartosz Górski 
 wrote:

> Hi Tim,
> 
> Maybe instead of just a flag like --nested (bool value) to resource-list we 
> can add optional argument like --depth X or --nested-level X (X - integer 
> value) to limit the depth for recursive listing of nested resources?
> 
> Best,
> Bartosz
> 
> On 05/19/2014 09:13 PM, Tim Schnell wrote:
>> Blueprint:
>> https://blueprints.launchpad.net/heat/+spec/explode-nested-resources
>> 
>> Spec: https://wiki.openstack.org/wiki/Heat/explode-resource-list
>> 
>> Tim
>> 
>> On 5/19/14 1:53 PM, "Tim Schnell"  wrote:
>> 
>>> On 5/19/14 12:35 PM, "Randall Burt"  wrote:
>>> 
>>> 
>>>> On May 19, 2014, at 11:39 AM, Steven Hardy 
>>>> wrote:
>>>> 
>>>>> On Mon, May 19, 2014 at 03:26:22PM +, Tim Schnell wrote:
>>>>>> Hi Nilakhya,
>>>>>> 
>>>>>> As Randall mentioned we did discuss this exact issue at the summit. I
>>>>>> was
>>>>>> planning on putting a blueprint together today to continue the
>>>>>> discussion.
>>>>>> The Stack Preview call is already doing the necessary recursion to
>>>>>> gather
>>>>>> the resources so we discussed being able to pass a stack id to the
>>>>>> preview
>>>>>> endpoint to get all of the resources.
>>>>>> 
>>>>>> However, after thinking about it some more, I agree with Randall that
>>>>>> maybe this should be an extra query parameter passed to the
>>>>>> resource-list
>>>>>> call. I'Ll have the blueprint up later today, unless you have already
>>>>>> started on it.
>>>>> Note there is a patch from Anderson/Richard which may help with this:
>>>>> 
>>>>> https://review.openstack.org/#/c/85781/
>>>>> 
>>>>> The idea was to enable easier introspection of resources backed by
>>>>> nested
>>>>> stacks in a UI, but it could be equally useful to generate a "tree"
>>>>> resource view in the CLI client by walking the links.
>>>>> 
>>>>> This would obviously be less efficient than recursing inside the
>>>>> engine,
>>>>> but arguably the output would be much more useful if it retains the
>>>>> nesting
>>>>> structure, as opposed to presenting a fully flattened "soup" of
>>>>> resources
>>>>> with no idea which stack/layer they belong to.
>>>>> 
>>>>> Steve
>>>> Could we simply add stack name/id to this output if the flag is passed? I
>>>> agree that we currently have the capability to traverse the tree
>>>> structure of nested stacks, but several folks have requested this
>>>> capability, mostly for UI/UX purposes. It would be faster if you want the
>>>> flat structure and we still retain the capability to create your own
>>>> tree/widget/whatever by following the links. Also, I think its best to
>>>> include this in the API directly since not all users are integrating
>>>> using the python-heatclient.
>>> +1 for adding the stack name/id to the output to maintain a reference to
>>> the initial stack that the resource belongs to. The original stated
>>> use-case that I am aware of was to have a flat list of all resources
>>> associated with a stack to be displayed in the UI when the user prompts to
>>> delete a stack. This would prevent confusion about what and why different
>>> resources are being deleted due to the stack delete.
>>> 
>>> This use-case does not require any information about the nested stacks but
>>> I can foresee that information being useful in the future. I think a
>>> flattened data structure (with a reference to stack id) is still the most
>>> efficient solution. The patch landed by Anderson/Richard provides an
>>> alternate method to drill down into nested stacks if the hierarchy is
>>> important information though this is not the optimal solution in this
>>> case.
>>> 
>>> Tim
>>>> 
>>>> ___
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev@lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [HEAT] Discussion: How to list nested stack resources.

[Randall Burt]

On May 19, 2014, at 11:39 AM, Steven Hardy 
 wrote:

> On Mon, May 19, 2014 at 03:26:22PM +, Tim Schnell wrote:
>> Hi Nilakhya,
>> 
>> As Randall mentioned we did discuss this exact issue at the summit. I was
>> planning on putting a blueprint together today to continue the discussion.
>> The Stack Preview call is already doing the necessary recursion to gather
>> the resources so we discussed being able to pass a stack id to the preview
>> endpoint to get all of the resources.
>> 
>> However, after thinking about it some more, I agree with Randall that
>> maybe this should be an extra query parameter passed to the resource-list
>> call. I'Ll have the blueprint up later today, unless you have already
>> started on it.
> 
> Note there is a patch from Anderson/Richard which may help with this:
> 
> https://review.openstack.org/#/c/85781/
> 
> The idea was to enable easier introspection of resources backed by nested
> stacks in a UI, but it could be equally useful to generate a "tree"
> resource view in the CLI client by walking the links.
> 
> This would obviously be less efficient than recursing inside the engine,
> but arguably the output would be much more useful if it retains the nesting
> structure, as opposed to presenting a fully flattened "soup" of resources
> with no idea which stack/layer they belong to.
> 
> Steve

Could we simply add stack name/id to this output if the flag is passed? I agree 
that we currently have the capability to traverse the tree structure of nested 
stacks, but several folks have requested this capability, mostly for UI/UX 
purposes. It would be faster if you want the flat structure and we still retain 
the capability to create your own tree/widget/whatever by following the links. 
Also, I think its best to include this in the API directly since not all users 
are integrating using the python-heatclient.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [HEAT] Discussion: How to list nested stack resources.

[Randall Burt]

Nilakhya, We discussed this a bit at the summit and I think the consensus was 
that this would be a good thing to do by passing a flag to resource-list that 
would "flatten" the structure of nested stacks in the call. Tim Schnell brought 
this up as well and may be interested in helping define the use case and spec.

On May 14, 2014, at 4:32 PM, Nilakhya Chatterjee 
 wrote:

> Hi All,
> 
> I recently tried to create a nested stack with the following example : 
> 
> http://paste.openstack.org/show/79156/
> 
> heat resource-list  gives only "MyStack" but intention should be to list all 
> the resources created by the nested templates, as also pointed by the command 
> help:
> 
> resource-list   Show list of resources belonging to a stack
> 
> 
> Let me know if this requires a BP to be created for discussion.
> 
> Thanks.
> 
> -- 
> 
> Nilakhya | Consultant Engineering
> GlobalLogic
> P +x.xxx.xxx.  M +91.989.112.5770  S skype
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] How to cross-reference resources within OS::Heat::ResourceGroup

[Randall Burt]

It would be a bit difficult if not impossible since you either address the 
aggregate values of all attributes in the group or the attribute of one 
specific member of the group. The ResourceGroup doesn't have a mechanism for 
applying sequential values to the members. That might be something you could 
raise a blueprint for.

On May 6, 2014, at 3:07 PM, "Janczuk, Tomasz" 
 wrote:

> Could this be accomplished with 3 resource groups instead of one? The
> first would create the ports, the second floating IPs, and the last the
> VMs? In that case, would there be a way to construct a reference to a
> particular instance of, say, a port, when creating an instance of a
> floating IP?
> 
> On 5/6/14, 12:41 PM, "Randall Burt"  wrote:
> 
>> A resource group's definition contains only one resource and you seem to
>> want groups of multiple resources. You would need to use a nested stack
>> or provider template to do what you're proposing.
>> 
>> On May 6, 2014, at 2:23 PM, "Janczuk, Tomasz" 
>> wrote:
>> 
>>> I am trying to create an OS::Heat::ResourceGroup of VMs and assign each
>>> VM a floating IP. As far as I know this requires cross-referencing the
>>> VM, port, and floating IP resources. How can I do that within a
>>> OS::Heat::ResourceGroup definition?
>>> 
>>> The `port: { get_resource: vm_cluster.vm_port.0 }` below is rejected by
>>> Heat.
>>> 
>>> Any help appreciated.
>>> 
>>> Thanks,
>>> Tomasz
>>> 
>>> vm_cluster:
>>>   type: OS::Heat::ResourceGroup
>>>   properties:
>>> count: { get_param: num_instances }
>>> resource_def:
>>>   vm:
>>> type: OS::Nova::Server
>>> properties:
>>>   key_name: { get_param: key_name }
>>>   flavor: { get_param: flavor }
>>>   image: { get_param: image }
>>>   networks:
>>> - port: { get_resource: vm_cluster.vm_port.0 }
>>>   vm_port:
>>> type: OS::Neutron::Port
>>> properties:
>>>   network_id: { get_param: private_net_id }
>>>   fixed_ips:
>>> - subnet_id: { get_param: private_subnet_id }
>>>   security_groups: [{ get_resource: rabbit_security_group }]
>>>   vm_floating_ip:
>>> type: OS::Neutron::FloatingIP
>>> properties:
>>>   floating_network_id: { get_param: public_net_id }
>>>   port_id: { get_resource: vm_cluster.vm_port.0 }
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] How to cross-reference resources within OS::Heat::ResourceGroup

[Randall Burt]

A resource group's definition contains only one resource and you seem to want 
groups of multiple resources. You would need to use a nested stack or provider 
template to do what you're proposing.

On May 6, 2014, at 2:23 PM, "Janczuk, Tomasz" 
 wrote:

> I am trying to create an OS::Heat::ResourceGroup of VMs and assign each VM a 
> floating IP. As far as I know this requires cross-referencing the VM, port, 
> and floating IP resources. How can I do that within a OS::Heat::ResourceGroup 
> definition?
> 
> The `port: { get_resource: vm_cluster.vm_port.0 }` below is rejected by Heat.
> 
> Any help appreciated.
> 
> Thanks,
> Tomasz
> 
>  vm_cluster:
>type: OS::Heat::ResourceGroup
>properties:
>  count: { get_param: num_instances }
>  resource_def:
>vm:
>  type: OS::Nova::Server
>  properties:
>key_name: { get_param: key_name }
>flavor: { get_param: flavor }
>image: { get_param: image }
>networks:
>  - port: { get_resource: vm_cluster.vm_port.0 }
>vm_port:
>  type: OS::Neutron::Port
>  properties:
>network_id: { get_param: private_net_id }
>fixed_ips:
>  - subnet_id: { get_param: private_subnet_id }
>security_groups: [{ get_resource: rabbit_security_group }]
>vm_floating_ip:
>  type: OS::Neutron::FloatingIP
>  properties:
>floating_network_id: { get_param: public_net_id }
>port_id: { get_resource: vm_cluster.vm_port.0 }
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Proposing Thomas Spatzier for heat-core

[Randall Burt]

+1

On Apr 22, 2014, at 1:43 PM, Zane Bitter  wrote:

> Resending with [Heat] in the subject line. My bad.
> 
> On 22/04/14 14:21, Zane Bitter wrote:
>> I'd like to propose that we add Thomas Spatzier to the heat-core team.
>> 
>> Thomas has been involved in and consistently contributing to the Heat
>> community for around a year, since the time of the Havana design summit.
>> His code reviews are of extremely high quality IMO, and he has been
>> reviewing at a rate consistent with a member of the core team[1].
>> 
>> One thing worth addressing is that Thomas has only recently started
>> expanding the focus of his reviews from HOT-related changes out into the
>> rest of the code base. I don't see this as an obstacle - nobody is
>> familiar with *all* of the code, and we trust core reviewers to know
>> when we are qualified to give +2 and when we should limit ourselves to
>> +1 - and as far as I know nobody else is bothered either. However, if
>> you have strong feelings on this subject nobody will take it personally
>> if you speak up :)
>> 
>> Heat Core team members, please vote on this thread. A quick reminder of
>> your options[2]:
>> +1  - five of these are sufficient for acceptance
>>  0  - abstention is always an option
>> -1  - this acts as a veto
>> 
>> cheers,
>> Zane.
>> 
>> 
>> [1] http://russellbryant.net/openstack-stats/heat-reviewers-30.txt
>> [2]
>> https://wiki.openstack.org/wiki/Heat/CoreTeam#Adding_or_Removing_Members
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [Glance] How about managing heat template like flavors in nova?

[Randall Burt]

We discussed this with the Glance community back in January and it was agreed 
that we should extend Glance's scope to include Heat templates as well as other 
artifacts. I'm planning on submitting some patches around this during Juno.

Adding the Glance tag as this is relevant to them as well.


 Original message 
From: Mike Spreitzer
Date:04/19/2014 9:43 PM (GMT-06:00)
To: "OpenStack Development Mailing List (not for usage questions)"
Subject: Re: [openstack-dev] [Heat] How about managing heat template like 
flavors in nova?

Gouzongmei  wrote on 04/19/2014 10:37:02 PM:

> We can supply APIs for getting, putting, adding and deleting current
> templates in the system, then when creating heat stacks, we just
> need to specify the name of the template.

Look for past discussion of Heat Template Repository (Heater).  Here is part of 
it: https://wiki.openstack.org/wiki/Heat/htr

Regards,
Mike
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Custom Resource

[Randall Burt]

On Apr 14, 2014, at 8:08 AM, Rabi Mishra 
 wrote:

> Hi Steve,
> 
> Thanks a lot for your prompt response. I can't agree more that the CFN custom 
> resource implementation is complex with it's dependency on SNS and SQS. 
> However, it  decouples the implementation of the resource life-cycle from the 
> resource itself. IMO, this has some advantages from the template complexity 
> and flexibility point of view.

IIRC implementing something like this had been discussed quite a while back. I 
think we discussed the possibility of using web hooks and a defined api/payload 
in place of the SNS/SQS type stuff. I don't think it ever made it to the 
backlog, but I'd be happy to discuss further design and maybe add a design 
session to the summit if you're unable to make it.

> On choices you mentioned:
> 
> 1. Custom Python Plugins - I do think this is the best approach for my 
> use-cases. However, asking a customer to develop custom plugins and 
> maintaining them can be too much ask(asking their 3rd party tool vendors to 
> do it is even more difficult), than plugging in some of their existing infra 
> script snippets.
> 
> 2. Provider Resource - Use of environment files for mapping nested template 
> and exchange of parameters/attributes looks sleek. However, I am yet to 
> understand how to wrap code snippets (many of them existing scripts) for the 
> resource life-cycle in the nested template to achieve these use-cases. 
> 
> With the CFN Custom resource, addition of some bits of code to the existing 
> scripts to parse the JSON snippets based on the stack life-cycle method is 
> what's required. 
> 
> However, my understanding of what's possible with the Provider Resource is 
> limited at the moment. I'll spend more time and go through it before coming 
> back with an answer to the use-case feasibility and constraints.
> 
> 
> Regards,
> Rabi Mishra
> 
> - Original Message -
>> Hi Rabi,
>> 
>> On Mon, Apr 14, 2014 at 06:44:44AM -0400, Rabi Mishra wrote:
>>> Hi All,
>>> 
>>> Recently, I've come across some requirements for external
>>> integrations/resources that can be managed like stack resources
>>> (create,update,delete) from the stack.
>>> 
>>> 1. Adding/Removing DNS records for instances created as part of a stack.
>>> 2. Integration with IPAM solutions for allocate/release of IPs (IP
>>> allocation pool for provider network)
>>> 3. Other custom integration for dynamic parameters to stacks.
>>> 
>>> IMHO, it would probably make sense to create a custom resource like 'AWS
>>> CFN Custom Resource'[1] that can be used for these kind of use cases. I
>>> have created a blueprint[2] for this.
>> 
>> Heat already has a couple of ways for custom resources to be defined.
>> 
>> The one which probably matches your requirements best is the "provider
>> resource" interface, which allows template defined resources to be mapped
>> to user-definable resource types, via an environment file:
>> 
>> http://hardysteven.blogspot.co.uk/2013/10/heat-providersenvironments-101-ive.html
>> http://docs.openstack.org/developer/heat/template_guide/environment.html
>> 
>> Provider resources can be defined by both users, and deployers (who can use
>> templates to e.g wrap an existing resource with something like DNS
>> registration logic, and expose the type transparently to the end-user)
>> 
>> For deployer requirements not satisfied by provider resources (for example
>> integration with third-party services), Heat also provides a python plugin
>> API, which enables deployers to create their own resource plugins as
>> needed:
>> 
>> http://docs.openstack.org/developer/heat/pluginguide.html
>> 
>> Personally, I think these two models provide sufficient flexibility that we
>> should be able to avoid the burden of maintaining a CFN compatible custom
>> resource plugin API.  I've not looked at it in detail, but the CFN model
>> you refer to has always seemed pretty complex to me, and seems like
>> something we don't necessarily want to replicate.
>> 
>> If there are gaps where things are not yet possible via the provider
>> resource interface, I'd rather discuss incremental improvements to that
>> instead of wholesale reimplementation of something compatible with AWS.
>> 
>> Can you provide any more feedback on your use-cases, and whether the
>> interfaces I linked can be used to satisfy them?
>> 
>> Steve
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [TripleO] Better handling of lists in Heat - a proposal to add a map function

[Randall Burt]

On Apr 4, 2014, at 1:56 PM, Zane Bitter 
 wrote:

> On 19/02/14 02:48, Clint Byrum wrote:
>> Since picking up Heat and trying to think about how to express clusters
>> of things, I've been troubled by how poorly the CFN language supports
>> using lists. There has always been the Fn::Select function for
>> dereferencing arrays and maps, and recently we added a nice enhancement
>> to HOT to allow referencing these directly in get_attr and get_param.
>> 
>> However, this does not help us when we want to do something with all of
>> the members of a list.
>> 
>> In many applications I suspect the template authors will want to do what
>> we want to do now in TripleO. We have a list of identical servers and
>> we'd like to fetch the same attribute from them all, join it with other
>> attributes, and return that as a string.
>> 
>> The specific case is that we need to have all of the hosts in a cluster
>> of machines addressable in /etc/hosts (please, Designate, save us,
>> eventually. ;). The way to do this if we had just explicit resources
>> named NovaCompute0, NovaCompute1, would be:
>> 
>>   str_join:
>> - "\n"
>> - - str_join:
>> - ' '
>> - get_attr:
>>   - NovaCompute0
>>   - networks.ctlplane.0
>> - get_attr:
>>   - NovaCompute0
>>   - name
>>   - str_join:
>> - ' '
>> - get_attr:
>>   - NovaCompute1
>>   - networks.ctplane.0
>> - get_attr:
>>   - NovaCompute1
>>   - name
>> 
>> Now, what I'd really like to do is this:
>> 
>> map:
>>   - str_join:
>> - "\n"
>> - - str_join:
>>   - ' '
>>   - get_attr:
>> - "$1"
>> - networks.ctlplane.0
>>   - get_attr:
>> - "$1"
>> - name
>>   - - NovaCompute0
>> - NovaCompute1
>> 
>> This would be helpful for the instances of resource groups too, as we
>> can make sure they return a list. The above then becomes:
>> 
>> 
>> map:
>>   - str_join:
>> - "\n"
>> - - str_join:
>>   - ' '
>>   - get_attr:
>> - "$1"
>> - networks.ctlplane.0
>>   - get_attr:
>> - "$1"
>> - name
>>   - get_attr:
>>   - NovaComputeGroup
>>   - member_resources
>> 
>> Thoughts on this idea? I will throw together an implementation soon but
>> wanted to get this idea out there into the hive mind ASAP.
> 
> Apparently I read this at the time, but completely forgot about it. Sorry 
> about that! Since it has come up again in the context of the "TripleO Heat 
> templates and merge.py" thread, allow me to contribute my 2c.
> 
> Without expressing an opinion on this proposal specifically, consensus within 
> the Heat core team has been heavily -1 on any sort of for-each functionality. 
> I'm happy to have the debate again (and TBH I don't really know what the 
> right answer is), but I wouldn't consider the lack of comment on this as a 
> reliable indicator of lazy consensus in favour; equivalent proposals have 
> been considered and rejected on multiple occasions.
> 
> Since it looks like TripleO will soon be able to move over to using 
> AutoscalingGroups (or ResourceGroups, or something) for groups of similar 
> servers, maybe we could consider baking this functionality into Autoscaling 
> groups instead of as an intrinsic function.
> 
> For example, when you do get_attr on an autoscaling resource it could fetch 
> the corresponding attribute from each member of the group and return them as 
> a list. (It might be wise to prepend "Output." or something similar - maybe 
> "Members." - to the attribute names, as AWS::CloudFormation::Stack does, so 
> that attributes of the autoscaling group itself can remain in a separate 
> namespace.)

FWIW, ResourceGroup supports this now as well as getting the attribute value 
from a given indexed member of the group.

> 
> Since members of your NovaComputeGroup will be nested stacks anyway (using 
> ResourceGroup or some equivalent feature - preferably autoscaling with 
> rolling updates), in the case above you'd define in the scaled template:
> 
>  outputs:
>hosts_entry:
>  description: An /etc/hosts entry for the NovaComputeServer
>  value:
>- str_join:
>  - ' '
>  - - get_attr:
>  - NovaComputeServer
>  - networks
>  - ctlplane
>  - 0
>- get_attr:
>  - NovaComputeServer
>  - name
> 
> And then in the main template (containing the autoscaling group):
> 
>str_join:
>  - "\n"
>  - get_attr:
>- NovaComputeGroup
>- Members.hosts_entry
> 
> would give the same output as your example would.
> 
> IMHO we should do something like this regardless of whether it solves your 
> use case, because it's fairly easy, requires no changes to the template 
> format, and users have been asking for ways to access e.g. a list of IP 
> addresses from a scaling group. That said, it seems very likely that mak

Re: [openstack-dev] [Heat][Murano][TOSCA] Murano team contrib. to Heat TOSCA activities

[Randall Burt]

On Mar 10, 2014, at 1:26 PM, Georgy Okrokvertskhov 

 wrote:

> Hi,
> 
> Thomas and Zane initiated a good discussion about Murano DSL and TOSCA 
> initiatives in Heat. I think will be beneficial for both teams to contribute 
> into TOSCA.

Wasn't TOSCA developing a "simplified" version in order to converge with HOT?

> While Mirantis is working on organizational part for OASIS. I would like to 
> understand what is the current view on the TOSCA and HOT relations. 
> It looks like TOSCA can cover all aspects of declarative components HOT 
> templates and imperative workflows which can be covered by Murano. What do 
> you think about that?

Aren't workflows covered by Mistral? How would this be different than including 
mistral support in Heat?

> I think TOSCA format can be used a a descriptions of Applications and 
> heat-translator can actually convert TOSCA descriptions to both HOT and 
> Murano files which can be then used for actual Application deployment. Both 
> Het and Murano workflows can coexist in Orchestration program and cover both 
> declarative templates and imperative workflows use cases.
> 
> -- 
> Georgy Okrokvertskhov
> Architect,
> OpenStack Platform Products,
> Mirantis
> http://www.mirantis.com
> Tel. +1 650 963 9828
> Mob. +1 650 996 3284
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [TripleO] Better handling of lists in Heat - a proposal to add a map function

[Randall Burt]

This may also be relevant: 
https://blueprints.launchpad.net/heat/+spec/override-resource-name-in-resource-group

On Feb 19, 2014, at 1:48 AM, Clint Byrum 
 wrote:

> Since picking up Heat and trying to think about how to express clusters
> of things, I've been troubled by how poorly the CFN language supports
> using lists. There has always been the Fn::Select function for
> dereferencing arrays and maps, and recently we added a nice enhancement
> to HOT to allow referencing these directly in get_attr and get_param.
> 
> However, this does not help us when we want to do something with all of
> the members of a list.
> 
> In many applications I suspect the template authors will want to do what
> we want to do now in TripleO. We have a list of identical servers and
> we'd like to fetch the same attribute from them all, join it with other
> attributes, and return that as a string.
> 
> The specific case is that we need to have all of the hosts in a cluster
> of machines addressable in /etc/hosts (please, Designate, save us,
> eventually. ;). The way to do this if we had just explicit resources
> named NovaCompute0, NovaCompute1, would be:
> 
>  str_join:
>- "\n"
>- - str_join:
>- ' '
>- get_attr:
>  - NovaCompute0
>  - networks.ctlplane.0
>- get_attr:
>  - NovaCompute0
>  - name
>  - str_join:
>- ' '
>- get_attr:
>  - NovaCompute1
>  - networks.ctplane.0
>- get_attr:
>  - NovaCompute1
>  - name
> 
> Now, what I'd really like to do is this:
> 
> map:
>  - str_join:
>- "\n"
>- - str_join:
>  - ' '
>  - get_attr:
>- "$1"
>- networks.ctlplane.0
>  - get_attr:
>- "$1"
>- name
>  - - NovaCompute0
>- NovaCompute1
> 
> This would be helpful for the instances of resource groups too, as we
> can make sure they return a list. The above then becomes:
> 
> 
> map:
>  - str_join:
>- "\n"
>- - str_join:
>  - ' '
>  - get_attr:
>- "$1"
>- networks.ctlplane.0
>  - get_attr:
>- "$1"
>- name
>  - get_attr:
>  - NovaComputeGroup
>  - member_resources
> 
> Thoughts on this idea? I will throw together an implementation soon but
> wanted to get this idea out there into the hive mind ASAP.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Nominate Jason Dunsmore for heat-core

[Randall Burt]

Very +1

 Original message 
From: Steve Baker
Date:02/09/2014 4:41 PM (GMT-06:00)
To: OpenStack Development Mailing List
Subject: [openstack-dev] [heat] Nominate Jason Dunsmore for heat-core

I would like to nominate Jason Dunsmore for heat-core.

His reviews are valuable and prolific, his code contributions have
demonstrated a good knowledge of heat internals, and he has endured a
sound hazing to get multi-engine into heat.

http://russellbryant.net/openstack-stats/heat-reviewers-60.txt
http://www.stackalytics.com/?release=icehouse&metric=commits&user_id=jasondunsmore

Heat cores, please reply with your vote.

cheers

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] About LaunchConfiguration and Autoscaling

[Randall Burt]

On Jan 30, 2014, at 12:09 PM, Clint Byrum 
 wrote:

> Excerpts from Zane Bitter's message of 2014-01-30 07:38:38 -0800:
>> On 30/01/14 06:01, Thomas Herve wrote:
>>> Hi all,
>>> 
>>> While talking to Zane yesterday, he raised an interesting question about 
>>> whether or not we want to keep a LaunchConfiguration object for the native 
>>> autoscaling resources.
>>> 
>>> The LaunchConfiguration object basically holds properties to be able to 
>>> fire new servers in a scaling group. In the new design, we will be able to 
>>> start arbitrary resources, so we can't keep a strict LaunchConfiguration 
>>> object as it exists, as we can have arbitrary properties.
>>> 
>>> It may be still be interesting to store it separately to be able to reuse 
>>> it between groups.
>>> 
>>> So either we do this:
>>> 
>>> group:
>>>   type: OS::Heat::ScalingGroup
>>>   properties:
>>> scaled_resource: OS::Nova::Server
>>> resource_properties:
>>>   image: my_image
>>>   flavor: m1.large
>> 
>> The main advantages of this that I see are:
>> 
>> * It's one less resource.
>> * We can verify properties against the scaled_resource at the place the 
>> LaunchConfig is defined. (Note: in _both_ models these would be verified 
>> at the same place the _ScalingGroup_ is defined.)

This looks a lot like OS::Heat::ResourceGroup, which I believe already 
addresses some of Zane's concerns around "dynamic" property validation.

>> 
>>> Or:
>>> 
>>> group:
>>>   type: OS::Heat::ScalingGroup
>>>   properties:
>>> scaled_resource: OS::Nova::Server
>>> launch_configuration: server_config
>>> server_config:
>>>   type: OS::Heat::LaunchConfiguration
>>>   properties:
>>> image: my_image
>>> flavor: m1.large
>> 
>> 
>> I favour this one for a few reasons:
>> 
>> * A single LaunchConfiguration can be re-used by multiple scaling 
>> groups. Reuse is good, and is one of the things we have been driving 
>> toward with e.g. software deployments.
> 
> I agree with the desire for re-use. In fact I am somewhat desperate to
> have it as we try to write templates which allow assembling different
> topologies of OpenStack deployment.
> 
> I would hope we would solve that at a deeper level, rather than making
> resources for the things we think will need re-use. I think nested stacks
> allow this level of re-use already anyway. Software config just allows
> sub-resource composition.

Agreed. Codifying re-use inside specific resource types is a game of catch-up I 
don't think we can win in the end.

> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Nomination for heat-core

[Randall Burt]

+1


Sent from my Verizon Wireless 4G LTE Smartphone



 Original message 
From: Steve Baker 
Date: 12/18/2013 8:28 PM (GMT-06:00)
To: OpenStack Development Mailing List 
Subject: [openstack-dev] [heat] Nomination for heat-core


I would like to nominate Bartosz Górski to be a heat-core reviewer. His reviews 
to date have been valuable and his other contributions to the project have 
shown a sound understanding of how heat works.

Here is his review history:
https://review.openstack.org/#/q/reviewer:bartosz.gorski%2540ntti3.com+project:openstack/heat,n,z

If you are heat-core please reply with your vote.

cheers
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 11, 2013, at 5:44 PM, Georgy Okrokvertskhov 

 wrote:

> Hi,
> 
> To keep this thread alive I would like to share the small screencast I've 
> recorded for Murano Metadata repository. I would like to share with you what 
> we have in Murano and start a conversation about metadata repository 
> development in OpenStack. Here is a link to screencast 
> http://www.youtube.com/watch?v=Yi4gC4ZhvPg Here is a link  to a detailed 
> specification of PoC for metadata repository currently implemented in Murano.
> 
> There is an etherpad (here) for new MetadataRepository design we started to 
> write after lesson learn phase of PoC. This is a future version of repository 
> we want to have. This proposal can be used as an initial basis for metadata 
> repository design conversation.
> 
> It will be great if we start conversation with Glance team to understand how 
> this work can be organized. As it was revealed in this thread, the most 
> probable candidate for metadata repository service implementation is Glance 
> program. 
> 
> Thanks,
> Georgy

Thanks for the link and info. I think the general consensus is this belongs in 
Glance, however I think details are being deferred until the mid-summit meet up 
in Washington D.C. (I could be totally wrong about this). In any case, I think 
I'll also start converting the existing HeatR blueprints to Glance ones. 
Perhaps it would be a good idea at this point to propose specific blueprints 
and have further ML discussions focused on specific changes?

> On Mon, Dec 9, 2013 at 3:24 AM, Thierry Carrez  wrote:
> Vishvananda Ishaya wrote:
> > On Dec 6, 2013, at 10:07 AM, Georgy Okrokvertskhov
> > mailto:gokrokvertsk...@mirantis.com>> wrote:
> >
> >> I am really inspired by this thread. Frankly saying, Glance for Murano
> >> was a kind of sacred entity, as it is a service with a long history in
> >> OpenStack.  We even did not think in the direction of changing Glance.
> >> Spending a night with these ideas, I am kind of having a dream about
> >> unified catalog where the full range of different entities are
> >> presented. Just imagine that we have everything as  first class
> >> citizens of catalog treated equally: single VM (image), Heat template
> >> (fixed number of VMs\ autoscaling groups), Murano Application
> >> (generated Heat templates), Solum assemblies
> >>
> >> Projects like Solum will highly benefit from this catalog as it can
> >> use all varieties of VM configurations talking with one service.
> >> This catalog will be able not just list all possible deployable
> >> entities but can be also a registry for already deployed
> >> configurations. This is perfectly aligned with the goal for catalog to
> >> be a kind of market place which provides billing information too.
> >>
> >> OpenStack users also will benefit from this as they will have the
> >> unified approach for manage deployments and deployable entities.
> >>
> >> I doubt that it could be done by a single team. But if all teams join
> >> this effort we can do this. From my perspective, this could be a part
> >> of Glance program and it is not necessary to add a new program for
> >> that. As it was mentioned earlier in this thread an idea of market
> >> place for images in Glance was here for some time. I think we can
> >> extend it to the idea of creating a marketplace for a deployable
> >> entity regardless of the way of deployment. As Glance is a core
> >> project which means it always exist in OpenStack deployment it makes
> >> sense to as a central catalog for everything.
> >
> > +1
> 
> +1 too.
> 
> I don't think that Glance is collapsing under its current complexity
> yet, so extending Glance to a general catalog service that can serve
> more than just reference VM images makes sense IMHO.
> 
> --
> Thierry Carrez (ttx)
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> Georgy Okrokvertskhov
> Technical Program Manager,
> Cloud and Infrastructure Services,
> Mirantis
> http://www.mirantis.com
> Tel. +1 650 963 9828
> Mob. +1 650 996 3284
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Stack preview

[Randall Burt]

On Dec 10, 2013, at 3:46 PM, Zane Bitter  wrote:

> On 10/12/13 15:10, Randall Burt wrote:
>> On Dec 10, 2013, at 1:27 PM, Zane Bitter 
>>  wrote:
>> 
>>> On 10/12/13 12:46, Richard Lee wrote:
>>>> Hey all,
>>>> 
>>>> We're working on a blueprint
>>>> <https://blueprints.launchpad.net/heat/+spec/preview-stack> that adds
>>>> the ability to preview what a given template+parameters would create in
>>>> terms of resources.  We think this would provide significant value for
>>>> blueprint authors and for other heat users that want to see what
>>>> someone's template would create before actually launching resources (and
>>>> possibly having to pay for them).
>>> 
>>> +1 for this use case.
>>> 
>>> BTW AWS supports something similar, which we never bothered to implement in 
>>> the compatibility API. You might want to do some research on that as a 
>>> starting point:
>>> 
>>> http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_EstimateTemplateCost.html
>>> 
>>> However the fact that we have pluggable resource types would make it very 
>>> difficult for us to do cost calculations inside Heat (and, in fact, 
>>> CloudFormation doesn't do that either, it just spits out a URL for their 
>>> separate calculator) - e.g. it's very hard to know which resources will 
>>> create, say, a Nova server unless they are all annotated in some way.
>>> 
>>> Are you thinking the API will simply return a list of resource types and 
>>> counts? e.g.:
>>> 
>>> {
>>>   "OS::Nova::Server": 2,
>>>   "OS::Cinder::Volume": 1,
>>>   "OS::Neutron::FloatingIP: 1
>>> }
>>> 
>>> If so, +1 for that implementation too. Don't forget that you will have to 
>>> recurse through provider templates, which may not contain what they say on 
>>> the tin.
>> 
>> That sounds more than reasonable to me. I don't think we could begin to do 
>> any sort of meaningful "cost" calculation without having to mostly punt to 
>> the service provider anyway.
> 
> Yeah, exactly.
> 
> Although it occurs to me that we may want more detail than I originally 
> thought... e.g. knowing the flavor of any Nova servers is probably quite 
> important. Any ideas?
> 
> The first thing that comes to mind is that we could annotate resource types 
> with the list of parameters we want to group by. That would enable something 
> like:
> 
> {
>  "OS::Nova::Server":
>[{config: {"flavor": m1.small}, count: 1},
> {config: {"flavor": m3.large}, count: 1}],
>  "OS::Cinder::Volume":
>[{config: {"size": 10}, count: 1}],
>  "OS::Neutron::FloatingIP":
>[{config: {}, count: 1}],
> }
> 
> - ZB

Yeah, that makes a lot of sense from a "I want to calculate what this stack is 
going to cost me" use case. My only concern is that a given service provider 
may have different ideas as to what's "important" WRT a stack's value, but we 
could always extend this with something in the global environment similar to 
how we discussed resource support status in those reviews.

So it sounds to me like we just need to add a field to the property schema that 
says "this property is important to the preview call".

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Stack convergence first steps

[Randall Burt]

On Dec 10, 2013, at 1:03 PM, Anderson Mesquita 
mailto:anderson...@thoughtworks.com>>
 wrote:

To try and keep this conversation moving forward, is it safe to say that we at 
least need to change the current status attribute to something like 
action_status? And the same with status_reason being changed to 
action_status_reason? Does anybody see a reason why we shouldn't go this way, 
since it's really what status currently refers to?

IMO, this sort of change should be proposed for the v2 api, since there are 
already expectations in the v1 api as to what these things mean. For v1, I 
wouldn't be opposed to adding a synthetic resource_state attribute that 
reflects the actual "status" of the underlying resource (ACTIVE, RESIZE, etc).

2013/12/8 Mitsuru Kanabuchi 
mailto:kanabuchi.mits...@po.ntts.co.jp>>

On Thu, 5 Dec 2013 22:13:18 -0600
Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>> wrote:

> On Thu, Dec 5, 2013 at 7:25 PM, Randall Burt 
> mailto:randall.b...@rackspace.com>>wrote:
>
> >  On Dec 5, 2013, at 6:25 PM, Christopher Armstrong <
> > chris.armstr...@rackspace.com<mailto:chris.armstr...@rackspace.com>>
> >  wrote:
> >
> >   On Thu, Dec 5, 2013 at 3:50 PM, Anderson Mesquita <
> > anderson...@thoughtworks.com<mailto:anderson...@thoughtworks.com>> wrote:
> >
> >> Hey stackers,
> >>
> >> We've been working towards making stack convergence (
> >> https://blueprints.launchpad.net/heat/+spec/stack-convergence) one step
> >> closer to being ready at a time.  After the first patch was submitted we
> >> got positive feedback on it as well as some good suggestions as to how to
> >> move it forward.
> >>
> >> The first step (https://blueprints.launchpad.net/heat/+spec/stack-check)
> >> is to get all the statuses back from the real world resources and update
> >> our stacks accordingly so that we'll be able to move on to the next step:
> >> converge it to the desired state, fixing any errors that may have happened.
> >>
> >> We just submitted another WiP for review, and as we were doing it, a few
> >> questions were raised and we'd like to get everybody's input on them. Our
> >> main concern is around the use and purpose of the `status` of a
> >> stack/resource.  `status` currently appears to represent the status of the
> >> last action taken, and it seems that we may need to repurpose it or
> >> possibly create something else to represent a stack's "health" (i.e.
> >> everything is up and running as expected, something smells fishy, something
> >> broke, stack's is doomed).  We described this thoroughly here:
> >> https://etherpad.openstack.org/p/heat-convergence
> >>
> >> Any thoughts?
> >>
> >> Cheers,
> >>
> >>
> >  I think a lot of OpenStack projects use "status" fields as "status of
> > the most recent operation", and I think it's totally wrong. "status" should
> > be a known state of the _object_, not an action, and if we need statuses
> > for operations, then those operations should be addressable REST objects.
> > Of course there are cases where object status should be updated to reflect
> > an operating status if it's a completely exclusive operation (BUILDING and
> > DELETING make sense, for example).
> >
> >  Actually, I think most projects are the opposite where "status" means
> > "what's the state of the resource" (Nova, Trove, Cinder, etc), whereas Heat
> > uses status as the state of the last operation. Probably wouldn't be too
> > terrible to have a new "state" for stacks and their resources then perhaps
> > deprecate and use "status" in the accepted way in the v2 API?
>
> Well, my point is that it's done inconsistently. Yes, it's mostly used as
> an object status, but nova for example uses it as an operation status for
> things like resize.

Nova's status of in resize is "RESIZE" and "VERITY_RESIZE".
This status means "Currently, Instance is ACTIVE and resize in progress".
I think Heat can assume resource status is "ACTIVE" in this case.

Thus, several status that contain operation status have to map resource(object)
status. However in my impression, a status that should assume another status
isn't a lot.

In my opinion, status mapping table is reasonable in present time.

Regards

--
Mitsuru Kanabuchi


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Stack preview

[Randall Burt]

On Dec 10, 2013, at 1:27 PM, Zane Bitter 
 wrote:

> On 10/12/13 12:46, Richard Lee wrote:
>> Hey all,
>> 
>> We're working on a blueprint
>>  that adds
>> the ability to preview what a given template+parameters would create in
>> terms of resources.  We think this would provide significant value for
>> blueprint authors and for other heat users that want to see what
>> someone's template would create before actually launching resources (and
>> possibly having to pay for them).
> 
> +1 for this use case.
> 
> BTW AWS supports something similar, which we never bothered to implement in 
> the compatibility API. You might want to do some research on that as a 
> starting point:
> 
> http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_EstimateTemplateCost.html
> 
> However the fact that we have pluggable resource types would make it very 
> difficult for us to do cost calculations inside Heat (and, in fact, 
> CloudFormation doesn't do that either, it just spits out a URL for their 
> separate calculator) - e.g. it's very hard to know which resources will 
> create, say, a Nova server unless they are all annotated in some way.
> 
> Are you thinking the API will simply return a list of resource types and 
> counts? e.g.:
> 
> {
>   "OS::Nova::Server": 2,
>   "OS::Cinder::Volume": 1,
>   "OS::Neutron::FloatingIP: 1
> }
> 
> If so, +1 for that implementation too. Don't forget that you will have to 
> recurse through provider templates, which may not contain what they say on 
> the tin.

That sounds more than reasonable to me. I don't think we could begin to do any 
sort of meaningful "cost" calculation without having to mostly punt to the 
service provider anyway.

> 
> cheers,
> Zane.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 6, 2013, at 5:04 PM, Clint Byrum 
 wrote:

> Excerpts from Randall Burt's message of 2013-12-06 14:43:05 -0800:
>> I too have warmed to this idea but wonder about the actual implementation 
>> around it. While I like where Edmund is going with this, I wonder if it 
>> wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates 
>> to Glance (/assemblies, /applications, etc) along side /images.  Initially, 
>> we could have separate endpoints and data structures for these different 
>> asset types, refactoring the easy bits along the way and leveraging the 
>> existing data storage and caching bits, but leaving more disruptive changes 
>> alone. That can get the functionality going, prove some concepts, and allow 
>> all of the interested parties to better plan a more general v3 api.
>> 
> 
> +1 on bolting the different views for things on as new v2 pieces instead
> of trying to solve the API genericism immediately.
> 
> I would strive to make this a facade, and start immediately on making
> Glance more generic under the hood.  Otherwise these will just end up
> as silos inside Glance instead of silos inside OpenStack.

Totally agreed. Where it makes sense to refactor we should do that rather than 
implementing essentially different services underneath.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

I too have warmed to this idea but wonder about the actual implementation 
around it. While I like where Edmund is going with this, I wonder if it 
wouldn't be valuable in the short-to-mid-term (I/J) to just add /templates to 
Glance (/assemblies, /applications, etc) along side /images.  Initially, we 
could have separate endpoints and data structures for these different asset 
types, refactoring the easy bits along the way and leveraging the existing data 
storage and caching bits, but leaving more disruptive changes alone. That can 
get the functionality going, prove some concepts, and allow all of the 
interested parties to better plan a more general v3 api.

On Dec 6, 2013, at 4:23 PM, Edmund Troche 
mailto:edmund.tro...@us.ibm.com>>
 wrote:


I agree with what seems to also be the general consensus, that Glance can 
"become" Heater+Glance (the service that manages images in OS today). Clearly, 
if someone looks at the Glance DB schema, APIs and service type (as returned by 
keystone service-list), all of the terminology is about images, so we would 
need to more formally define what are the characteristics or "image", 
"template", maybe "assembly", "components" etc and find what is a good 
generalization. When looking at the attributes for "image" (image table), I can 
see where there are a few that would be generic enough to apply to "image", 
"template" etc, so those could be taken to be the base set of attributes, and 
then based on the "type" (image, template, etc) we could then have attributes 
that are type-specific (maybe by leveraging what is today "image_properties").

As I read through the discussion, the one thing that came to mind is "asset 
management". I can see where if someone bothers to create an image, or a 
template, then it is for a good reason, and that perhaps you'd like to maintain 
it as an IT asset. Along those lines, it occurred to me that maybe what we need 
is to make Glance some sort of asset management service that can be leveraged 
by Service Catalogs, Nova, etc. Instead of storing "images" and "templates"  we 
store assets of one kind or another, with artifacts (like files, image content, 
etc), and associated metadata. There is some work we could borrow from, 
conceptually at least, from OSLC's Asset Management specification: 
http://open-services.net/wiki/asset-management/OSLC-Asset-Management-2.0-Specification/.
 Looking at this spec, it probably has more than we need, but there's plenty we 
could borrow from it.


Edmund Troche


Georgy Okrokvertskhov ---12/06/2013 01:34:13 PM---As a Murano team 
we will be happy to contribute to Glance. Our Murano metadata repository is a 
stand

From: Georgy Okrokvertskhov 
mailto:gokrokvertsk...@mirantis.com>>
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>,
Date: 12/06/2013 01:34 PM
Subject: Re: [openstack-dev] [heat] [glance] Heater Proposal





As a Murano team we will be happy to contribute to Glance. Our Murano metadata 
repository is a standalone component (with its own git repository)which is not 
tightly coupled with Murano itself. We can easily add our functionality to 
Glance as a new component\subproject.

Thanks
Georgy


On Fri, Dec 6, 2013 at 11:11 AM, Vishvananda Ishaya 
mailto:vishvana...@gmail.com>> wrote:

On Dec 6, 2013, at 10:38 AM, Clint Byrum 
mailto:cl...@fewbar.com>> wrote:

> Excerpts from Jay Pipes's message of 2013-12-05 21:32:54 -0800:
>> On 12/05/2013 04:25 PM, Clint Byrum wrote:
>>> Excerpts from Andrew Plunk's message of 2013-12-05 12:42:49 -0800:
> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>> On Dec 5, 2013, at 10:10 AM, Clint Byrum > fewbar.com>
>>  wrote:
>>
>>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
 Why not just use glance?

>>>
>>> I've asked that question a few times, and I think I can collate the
>>> responses I've received below. I think enhancing glance to do these
>>> things is on the table:
>>>
>>> 1. Glance is for big blobs of data not tiny templates.
>>> 2. Versioning of a single resource is desired.
>>> 3. Tagging/classifying/listing/sorting
>>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>>
>>> My responses:
>>>
>>> 1: Irrelevant. Smaller things will fit in it just fine.
>>
>> Fitting is one thing, optimizations around particular assumptions about 
>> the size of data and the frequency of reads/writes might be an issue, 
>> but I admit to ignorance about those details in Glance.
>>
>
> Optimizations can be improved for various use cases. The design, however,
> has no assumptions that I know about that would invalidate storing blobs
> of yaml/json vs. blobs of kernel/qcow2/raw image.

 I think we are getting out into the weeds a little bit here. It is 
 im

Re: [openstack-dev] [heat][horizon]Heat UI related requirements & roadmap

[Randall Burt]

I hope I'm not re-opening worm cans here, and that's not my intent, but I just 
wanted to get a little clarification in-line below:

On Dec 6, 2013, at 3:24 PM, Tim Schnell 
 wrote:

> To resolve this thread, I have created 5 blueprints based on this mailing
> list discussion. I have attempted to distill the proposed specification
> down to what seemed generally agreed upon but if you feel strongly that I
> have incorrectly captured something let's talk about it!
> 
> Here are the blueprints:
> 
> 1) Stack Keywords
> blueprint: https://blueprints.launchpad.net/heat/+spec/stack-keywords
> spec: https://wiki.openstack.org/wiki/Heat/UI#Stack_Keywords

As proposed, these look like template keywords and not stack keywords.

I may be mis-remembering the conversation around this, but it would seem to me 
this mixes tagging templates and tagging stacks. In my mind, these are separate 
things. For the stack part, it seems like I could just pass something like 
"--keyword blah" multiple times to python-heatclient and not have to edit the 
template I'm passing. This lets me organize my stacks the way I want rather 
than relying on the template author (who may not be me) to organize things for 
me. Alternatively, I'd at least like the ability to accept, replace, and/or 
augment the keywords the template author proposes.

> 
> 2) Parameter Grouping and Ordering
> blueprint: 
> https://blueprints.launchpad.net/heat/+spec/parameter-grouping-ordering
> spec: 
> https://wiki.openstack.org/wiki/Heat/UI#Parameter_Grouping_and_Ordering
> 
> 3) Parameter Help Text
> blueprint: 
> https://blueprints.launchpad.net/heat/+spec/add-help-text-to-template
> spec: https://wiki.openstack.org/wiki/Heat/UI#Help_Text
> 
> 4) Parameter Label
> blueprint: 
> https://blueprints.launchpad.net/heat/+spec/add-parameter-label-to-template
> spec: https://wiki.openstack.org/wiki/Heat/UI#Parameter_Label
> 
> 
> This last blueprint did not get as much discussion so I have added it with
> the discussion flag set. I think this will get more important in the
> future but I don't need to implement right now. I'd love to hear more
> thoughts about it.
> 
> 5) Get Parameters API Endpoint
> blueprint: 
> https://blueprints.launchpad.net/heat/+spec/get-parameters-from-api
> spec: 
> https://wiki.openstack.org/wiki/Heat/UI#New_API_Endpoint_for_Returning_Temp
> late_Parameters

History around validate_template aside, I wonder if this doesn't re-open the 
discussion around having an endpoint that will translate an entire template 
into the native format (HOT). I understand that the idea is that we normalize 
parameter values to relieve user interfaces from having to understand several 
formats supported by Heat, but it just seems to me that there's a more general 
use case here.

> 
> Thanks,
> Tim

I know its probably nit-picky, but I would prefer these specs be individual 
wiki pages instead of lumped all together. At any rate, thanks for organizing 
all this!

> 
> On 11/28/13 4:55 AM, "Zane Bitter"  wrote:
> 
>> On 27/11/13 23:37, Fox, Kevin M wrote:
>>> Hmm... Yeah. when you tell heat client the url to a template file, you
>>> could set a flag telling the heat client it is in a git repo. It could
>>> then automatically look for repo information and set a stack metadata
>>> item pointing back to it.
>> 
>> Or just store the URL.
>> 
>>> If you didn't care about taking a performance hit, heat client could
>>> always try and check to see if it was a git repo url. That may add
>>> several extra http requests though...
>>> 
>>> Thanks,
>>> Kevin
>>> 
>>> From: Clint Byrum [cl...@fewbar.com]
>>> Sent: Wednesday, November 27, 2013 1:04 PM
>>> To: openstack-dev
>>> Subject: Re: [openstack-dev] [heat][horizon]Heat UI related
>>> requirements &  roadmap
>>> 
>>> Excerpts from Fox, Kevin M's message of 2013-11-27 08:58:16 -0800:
 This use case is sort of a providence case. Where did the stack come
 from so I can find out more about it.
 
>>> 
>>> This exhibits similar problems to our Copyright header problems. Relying
>>> on authors to maintain their authorship information in two places is
>>> cumbersome and thus the one that is not automated will likely fall out
>>> of sync fairly quickly.
>>> 
 You could put a git commit field in the template itself but then it
 would be hard to keep updated.
 
>>> 
>>> Or you could have Heat able to pull from any remote source rather than
>>> just allowing submission of the template directly. It would just be
>>> another column in the stack record. This would allow said support person
>>> to see where it came from by viewing the stack, which solves the use
>>> case.
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@

Re: [openstack-dev] [heat] Stack convergence first steps

[Randall Burt]

On Dec 5, 2013, at 6:25 PM, Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>>
 wrote:

On Thu, Dec 5, 2013 at 3:50 PM, Anderson Mesquita 
mailto:anderson...@thoughtworks.com>> wrote:
Hey stackers,

We've been working towards making stack convergence 
(https://blueprints.launchpad.net/heat/+spec/stack-convergence) one step closer 
to being ready at a time.  After the first patch was submitted we got positive 
feedback on it as well as some good suggestions as to how to move it forward.

The first step (https://blueprints.launchpad.net/heat/+spec/stack-check) is to 
get all the statuses back from the real world resources and update our stacks 
accordingly so that we'll be able to move on to the next step: converge it to 
the desired state, fixing any errors that may have happened.

We just submitted another WiP for review, and as we were doing it, a few 
questions were raised and we'd like to get everybody's input on them. Our main 
concern is around the use and purpose of the `status` of a stack/resource.  
`status` currently appears to represent the status of the last action taken, 
and it seems that we may need to repurpose it or possibly create something else 
to represent a stack's "health" (i.e. everything is up and running as expected, 
something smells fishy, something broke, stack's is doomed).  We described this 
thoroughly here: https://etherpad.openstack.org/p/heat-convergence

Any thoughts?

Cheers,


I think a lot of OpenStack projects use "status" fields as "status of the most 
recent operation", and I think it's totally wrong. "status" should be a known 
state of the _object_, not an action, and if we need statuses for operations, 
then those operations should be addressable REST objects. Of course there are 
cases where object status should be updated to reflect an operating status if 
it's a completely exclusive operation (BUILDING and DELETING make sense, for 
example).

Actually, I think most projects are the opposite where "status" means "what's 
the state of the resource" (Nova, Trove, Cinder, etc), whereas Heat uses status 
as the state of the last operation. Probably wouldn't be too terrible to have a 
new "state" for stacks and their resources then perhaps deprecate and use 
"status" in the accepted way in the v2 API?

--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 4:45 PM, Steve Baker 
mailto:sba...@redhat.com>>
 wrote:

On 12/06/2013 10:46 AM, Mark Washenberger wrote:



On Thu, Dec 5, 2013 at 1:05 PM, Vishvananda Ishaya 
mailto:vishvana...@gmail.com>> wrote:

On Dec 5, 2013, at 12:42 PM, Andrew Plunk 
mailto:andrew.pl...@rackspace.com>> wrote:

>> Excerpts from Randall Burt's message of 2013-12-05 09:05:44 -0800:
>>> On Dec 5, 2013, at 10:10 AM, Clint Byrum >> fewbar.com>
>>> wrote:
>>>
 Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
> Why not just use glance?
>

 I've asked that question a few times, and I think I can collate the
 responses I've received below. I think enhancing glance to do these
 things is on the table:

 1. Glance is for big blobs of data not tiny templates.
 2. Versioning of a single resource is desired.
 3. Tagging/classifying/listing/sorting
 4. Glance is designed to expose the uploaded blobs to nova, not users

 My responses:

 1: Irrelevant. Smaller things will fit in it just fine.
>>>
>>> Fitting is one thing, optimizations around particular assumptions about the 
>>> size of data and the frequency of reads/writes might be an issue, but I 
>>> admit to ignorance about those details in Glance.
>>>
>>
>> Optimizations can be improved for various use cases. The design, however,
>> has no assumptions that I know about that would invalidate storing blobs
>> of yaml/json vs. blobs of kernel/qcow2/raw image.
>
> I think we are getting out into the weeds a little bit here. It is important 
> to think about these apis in terms of what they actually do, before the 
> decision of combining them or not can be made.
>
> I think of HeatR as a template storage service, it provides extra data and 
> operations on templates. HeatR should not care about how those templates are 
> stored.
> Glance is an image storage service, it provides extra data and operations on 
> images (not blobs), and it happens to use swift as a backend.

This is not completely correct. Glance already supports something akin to 
templates. You can create an "image" with metadata properties that specifies a 
complex block device mapping which would allow for multiple volumes and images 
to connected to the vm at boot time. This is functionally a template for a 
single vm.

Glance is pretty useless if is just an "image storage" service, we already have 
other places that can store bits (swift, cinder). It is much more valuable as a 
searchable repository of bootable templates. I don't see any reason why this 
idea couldn't be extended to include more complex templates that could include 
more than one vm.

FWIW I agree with all of this. I think Glance's real role in OpenStack is as a 
helper and optionally as a gatekeeper for the category of "stuff Nova can 
boot". So any parameter that affects what Nova is going to boot should in my 
view be something Glance can be aware of. This list of parameters *could* grow 
to include multiple device images, attached volumes, and other things that 
currently live in the realm of flavors such as extra hardware requirements and 
networking aspects.

Just so things don't go too crazy, I'll add that since Nova is generally 
focused on provisioning individual VMs, anything above the level of an 
individual VM should be out of scope for Glance.

I think Glance should alter its approach to be less generally agnostic about 
the contents of the objects it hosts. Right now, we are just starting to do 
this with images, as we slowly advance on offering server side format 
conversion. We could find similar use cases for single vm templates.


The average heat template would provision more than one VM, plus any number of 
other cloud resources.

An image is required to provision a single nova server;
a template is required to provision a single heat stack.

Hopefully the above "single vm" policy could be reworded to be agnostic to the 
service which consumes the object that glance is storing.

To add to this, is it that Glance wants to be *more* integrated and geared 
towards vm or container images or that Glance wishes to have more intimate 
knowledge of the things its cataloging *regardless of what those things 
actually might be*? The reason I ask is that Glance supporting only "single vm 
templates" when Heat orchestrates the entire (or almost entire) spectrum of 
core and integrated projects means that its suitability as a candidate for a 
template repository plummets quite a bit.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 4:08 PM, Georgy Okrokvertskhov 
mailto:gokrokvertsk...@mirantis.com>>
 wrote:

Hi,

I am really glad to see the line of thinking close to what we at Murano see as 
a right direction for OpenStack development. This is a good initiative which 
potentially will be useful for other projects.  We have very similar idea about 
repository in Murano project and we even implemented the first version of it. 
We are very open for collaboration and exchanging the ideas.


In terms of overlap with Murano, I can see the overlap in the area of Murano 
Metadata Repository. We already have done some work in this area and you can 
find the detailed description here 
https://wiki.openstack.org/wiki/Murano/SimplifiedMetadataRepository. The 
implementation of the first version is already done and we plan to include the 
implementation in Murano 0.4 release which will go out in a week.


For the future roadmap with more advanced functionality we have created 
etherpad:  https://etherpad.openstack.org/p/MuranoMetadata


My concerns around Heater lie in two areas:

- Fit for OpenStack Orchestration program

Do you mean to imply that a repository of orchestration templates is a bad fit 
for the orchestration program?

- Too narrow focus as it formulated right now making hard for other projects 
like Murano take advantage of this services as general purpose metadata 
repository

That's what the discussion around using Glance is about though. The proposal 
started out as a separate service, but arguments are being made that the use 
cases fit into Glance. The use cases don't change as their focused on templates 
and not general object cataloging, but that's something to sort if/when we land 
on an implementation.

I am not sure how metadata repository related to orchestration program as it 
does not orchestrate anything. I would rather consider creating separate 
Service Catalog/Metadata Repository program or consider storage programs like 
Glance or Swift as Heater has the similar feature set. If you replace 
“template” with “object” you will actually propose a new swift implementation 
with replacing existing Swift’s versioning, acl, and metadata for objects.

Doesn't that same argument hold for Murano Metadata Repository as well? And, as 
initially proposed, its not a generic metadata repository but a template 
cataloging system. The difference maybe academic, but I think its important. 
That being said, maybe there's a case for something even more generic (store 
some meta information about some consumable artifact and a pointer to where to 
get it), but IMO, the arguments for Glance then become more compelling (not 
that I've bought in to that completely yet).

Murano as Application Catalog also could be a fit, but I don’t insist :)

It sounds to me like conceptually it would suffer from the same scoping issues 
we're already discussing though.

At the current moment Heat is not opinionated about template placement and this 
provides a lot of flexibility for other projects which use Heat under the hood. 
With your proposal, you are creating new metadata repository solution for 
specific use case of template storage making Heat much more prescriptive.

I'm not sure where this impression comes from. The Heat orchestration 
api/engine would in no way be dependent on the repository. Heat would still 
accept and orchestrate any template you passed it. At best, Heat would be 
extended to be aware of catalog urls and template id's, but in no way was it 
ever meant to imply that Heat would ever be modified to *only* work with 
templates from a catalog or require any of the catalog metadata to function in 
its core role as an orchestration engine.

Combined with TC policy which enforces projects to use existing code, this 
could be a big problem because other projects might want to keep not only Heat 
template but other components and metadata. Murano is a good example for that. 
It has multiple objects associated with Application and Heat template is only 
one of them.  That would mean that other projects would either need to 
duplicate the functionality of the catalog or significantly restrict their own 
functionality.

Or Murano could also extend Glance functionality to include these sorts of 
composite artifacts similar to how Vish described earlier in the thread.

The most scary thing for me is that you propose to add metadata information to 
the HOT template. In Murano we keep UI information in a separate file and this 
provides us a flexibility how to render UI for the same Heat template in 
different Applications. This is a question of separation of concerns as Heat as 
a deployment orchestration should not interfere to UI.

While I agree that things related to specific UI strategies don't really belong 
in the template, I think you may have misconstrued the example cited. As I 
understood it as a "if this then wouldn't we have to do this unsightly thing"? 
Could be that I misunderstood though, and in that cas

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 11:10 AM, Clint Byrum 
 wrote:

> Excerpts from James Slagle's message of 2013-12-05 08:35:12 -0800:
>> On Thu, Dec 5, 2013 at 11:10 AM, Clint Byrum  wrote:
>>> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
 Why not just use glance?
 
>>> 
>>> I've asked that question a few times, and I think I can collate the
>>> responses I've received below. I think enhancing glance to do these
>>> things is on the table:
>> 
>> I'm actually interested in the use cases laid out by Heater from both
>> a template perspective and image perspective.  For the templates, as
>> Robert mentioned, Tuskar needs a solution for this requirement, since
>> it's deploying using templates.  For the images, we have the concept
>> of a "golden" image in TripleO and are heavily focused on image based
>> deployments.  Therefore, it seems to make sense that TripleO also
>> needs a way to version/tag known good images.
>> 
>> Given that, I think it makes sense  to do this in a way so that it's
>> consumable for things other than just templates.  In fact, you can
>> almost s/template/image/g on the Heater wiki page, and it pretty well
>> lays out what I'd like to see for images as well.
>> 
>>> 1. Glance is for big blobs of data not tiny templates.
>>> 2. Versioning of a single resource is desired.
>>> 3. Tagging/classifying/listing/sorting
>>> 4. Glance is designed to expose the uploaded blobs to nova, not users
>>> 
>>> My responses:
>>> 
>>> 1: Irrelevant. Smaller things will fit in it just fine.
>>> 
>>> 2: The swift API supports versions. We could also have git as a
>>> backend.
>> 
>> I would definitely like to see a git backend for versioning.  No
>> reason to reimplement a different solution for what already works
>> well.  I'm not sure we'd want to put a whole image into git though.
>> Perhaps just it's manifest (installed components, software versions,
>> etc) in json format would go into git, and that would be associated
>> back to the binary image via uuid.  That would even make it easy to
>> diff changes between versions, etc.
>> 
> 
> Right, git for a big 'ol image makes little sense.
> 
> I'm suggesting that one might want to have two glances, one for images
> which just uses swift versions and would just expose a list of versions,
> and one for templates which would use git and thus expose more features
> like a git remote for the repo. I'm not sure if glance has embraced the
> extension paradigm yet, but this would fall nicely into it.

Alternatively, Glance could have configurable backends for each image type 
allowing for optimization without the (often times messy) extension mechanism? 
This is assuming it doesn't do this already - I really need to start digging 
here. In the spirit of general OpenStack architectural paradigms, as long as 
the service exposes a consistent interface for templates that includes 
versioning support, the back-end store and (possibly) the versioning "engine" 
should certainly be configurable. Swift probably makes a decent first/default 
implementation.

>>> This feels like something we can add as an optional feature
>>> without exploding Glance's scope and I imagine it would actually be a
>>> welcome feature for image authors as well. Think about Ubuntu maintaining
>>> official images. If they can keep the ID the same and just add a version
>>> (allowing users to lock down to a version if updated images cause issue)
>>> that seems like a really cool feature for images _and_ templates.
>>> 
>>> 3: I'm sure glance image users would love to have those too.
>>> 
>>> 4: Irrelevant. Heat will need to download templates just like nova, and
>>> making images publicly downloadable is also a thing in glance.
>>> 
>>> It strikes me that this might be a silo problem instead of an
>>> actual design problem. Folk should not be worried about jumping into
>>> Glance and adding features. Unless of course the Glance folk have
>>> reservations? (adding glance tag to the subject)
>> 
>> I'm +1 for adding these types of features to glance, or at least
>> something common, instead of making it specific to Heat templates.
>> 
> 
> Right, it may be that glance is too limited, but from what I've seen,
> it is not and it already has the base concepts that "HeaTeR" wants to
> have available.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] [glance] Heater Proposal

[Randall Burt]

On Dec 5, 2013, at 10:10 AM, Clint Byrum 
 wrote:

> Excerpts from Monty Taylor's message of 2013-12-04 17:54:45 -0800:
>> Why not just use glance?
>> 
> 
> I've asked that question a few times, and I think I can collate the
> responses I've received below. I think enhancing glance to do these
> things is on the table:
> 
> 1. Glance is for big blobs of data not tiny templates.
> 2. Versioning of a single resource is desired.
> 3. Tagging/classifying/listing/sorting
> 4. Glance is designed to expose the uploaded blobs to nova, not users
> 
> My responses:
> 
> 1: Irrelevant. Smaller things will fit in it just fine.

Fitting is one thing, optimizations around particular assumptions about the 
size of data and the frequency of reads/writes might be an issue, but I admit 
to ignorance about those details in Glance.

> 2: The swift API supports versions. We could also have git as a
> backend. This feels like something we can add as an optional feature
> without exploding Glance's scope and I imagine it would actually be a
> welcome feature for image authors as well. Think about Ubuntu maintaining
> official images. If they can keep the ID the same and just add a version
> (allowing users to lock down to a version if updated images cause issue)
> that seems like a really cool feature for images _and_ templates.

Agreed, though one could argue that using image names and looking up ID's or 
just using ID's as appropriate "sort of" handle this use case, but I agree that 
having image versioning seems a reasonable feature for Glance to have as well.

> 3: I'm sure glance image users would love to have those too.

And image metadata is already there so we don't have to go through those 
discussions all over again ;).

> 4: Irrelevant. Heat will need to download templates just like nova, and
> making images publicly downloadable is also a thing in glance.

Yeah, this was the kicker for me. I'd been thinking of adding the 
tenancy/public/private templates use case to the HeatR spec and realized that 
this was a good argument for Glance since it already has this feature.

> It strikes me that this might be a silo problem instead of an
> actual design problem. Folk should not be worried about jumping into
> Glance and adding features. Unless of course the Glance folk have
> reservations? (adding glance tag to the subject)

Perhaps, and if these use cases make sense for the Glance users in general, I 
wouldn't want to re-invent all those wheels either. I admit there's some appeal 
to being able to pass a template ID to stack-create or as the type of a 
provider resource and have an actual API to call that's already got a known, 
tested client that's already part of the OpenStack ecosystem

In the end, though, even if some and not all of our use cases make sense for 
the Glance folks, we still have the option of creating the HeatR service and 
having Glance as a possible back-end data store.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Solum] CLI minimal implementation

[Randall Burt]

I disagree. If a param is required and has no meaningful default, it should be 
positional IMO. I think this actually reduces confusion as you can tell from 
the signature alone that this is a value the user must supply to have any 
meaningful thing happen.

On Dec 3, 2013, at 10:13 AM, Paul Montgomery 
 wrote:

> I agree.  With many optional parameters possible, positional parameters
> would seem to complicate things a bit (even for end users).
> 
> 
> On 12/3/13 8:14 AM, "Arati Mahimane"  wrote:
> 
>> 
>> 
>> On 12/3/13 7:51 AM, "Roshan Agrawal"  wrote:
>> 
>>> 
>>> 
 -Original Message-
 From: Russell Bryant [mailto:rbry...@redhat.com]
 Sent: Monday, December 02, 2013 8:17 PM
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Solum] CLI minimal implementation
 
 On 12/02/2013 07:03 PM, Roshan Agrawal wrote:
> I have created a child blueprint to define scope for the minimal
 implementation of the CLI to consider for milestone 1.
> 
 https://blueprints.launchpad.net/solum/+spec/cli-minimal-implementatio
> n
> 
> Spec for the minimal CLI @
> 
 https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/CLI-minimal-im
> plementation Etherpad for discussion notes:
> https://etherpad.openstack.org/p/MinimalCLI
> 
> Would look for feedback on the ML, etherpad and discuss more in the
 weekly IRC meeting tomorrow.
 
 What is this R1.N syntax?  How does it relate to development
 milestones?
 Does R1 mean a requirement for milestone-1?
>>> 
>>> These do not relate to development milestones. R1 is a unique identified
>>> for the given requirement. R1.x is a unique requirement Id for something
>>> that is a sub item of the top level requirement R1.
>>> Is there a more "openstack standard way" for generating requirements Id?
>>> 
 For consistency, I would use commands like:
 
   solum app-create
   solum app-delete
   solum assembly-create
   solum assembly-delete
 
 instead of adding a space in between:
 
   solum app create
 
 to be more consistent with other clients, like:
 
   nova flavor-create
   nova flavor-delete
   glance image-create
   glance image-delete
>>> 
>>> The current proposal is an attempt to be consistent with the direction
>>> for the "openstack one CLI". Adrian's addressed it in his other reply.
>>> 
>>> 
 I would make required arguments positional arguments.  So, instead of:
 
   solum app-create --plan=planname
 
 do:
 
   solum app-create 
>>> 
>>> I will make this change unless hear objections
>> 
>> In my opinion, since most of the parameters (listed here
>> https://wiki.openstack.org/wiki/Solum/FeatureBlueprints/ApplicationDeploym
>> e
>> ntAndManagement#Solum-R1.12_app_create:_CLI) are optional,
>> it would be easier to specify the parameters as =
>> instead of having positional parameters.
>> 
>> 
>>> 
>>> 
 Lastly, everywhere you have a name, I would use a UUID.  Names
 shouldn't
 have to be globally unique (because of multi-tenancy).  UUIDs should
 always
 work, but you can support a name in the client code as a friendly
 shortcut,
 but it should fail if a unique result can not be resolved from the
 name.
>>> 
>>> 
>>> Names do not have to be globally unique; just unique within the tenant
>>> namespace. The Name+tenant combination should map to a unique uuid.
>>> The CLI is a client tool, where as a user working with names is easier.
>>> We will support both, but start with Names (the friendly shortcut), and
>>> map it to uuid behind the scenes.
>>> 
>>> 
 --
 Russell Bryant
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

[Randall Burt]

On Nov 14, 2013, at 1:05 PM, Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>> wrote:

On Thu, Nov 14, 2013 at 11:00 AM, Randall Burt 
mailto:randall.b...@rackspace.com>> wrote:

On Nov 14, 2013, at 12:44 PM, Zane Bitter 
mailto:zbit...@redhat.com>>
 wrote:

> On 14/11/13 18:51, Randall Burt wrote:
>>
>> On Nov 14, 2013, at 11:30 AM, Christopher Armstrong
>> mailto:chris.armstr...@rackspace.com> 
>> <mailto:chris.armstr...@rackspace.com<mailto:chris.armstr...@rackspace.com>>>
>>  wrote:
>>
>>> On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt
>>> mailto:randall.b...@rackspace.com> 
>>> <mailto:randall.b...@rackspace.com<mailto:randall.b...@rackspace.com>>> 
>>> wrote:
>>>Regarding web hook execution and cool down, I think the response
>>>should be something like 307 if the hook is on cool down with an
>>>appropriate retry-after header.
>
> I strongly disagree with this even ignoring the security issue mentioned 
> below. Being in the cooldown period is NOT an error, and the caller should 
> absolutely NOT try again later - the request has been received and correctly 
> acted upon (by doing nothing).

But how do I know nothing was done? I may have very good reasons to re-scale 
outside of ceilometer or other mechanisms and absolutely SHOULD try again 
later.  As it stands, I have no way of knowing that my scaling action didn't 
happen without examining my physical resources. 307 is a legitimate response in 
these cases, but I'm certainly open to other suggestions.


I agree there should be a way to find out what happened, but in a way that 
requires a more strongly authenticated request. My preference would be to use 
an audit log system (I haven't been keeping up with the current thoughts on the 
design for Heat's event/log API) that can be inspected via API.

Fair enough. I'm just thinking of folks who want to set this up but use 
external tools/monitoring solutions for the actual eventing. Having those tools 
grep through event logs seems a tad cumbersome, but I do understand the desire 
to make these un-authenticated secrets makes that terribly difficult.



--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

[Randall Burt]

On Nov 14, 2013, at 12:44 PM, Zane Bitter 
 wrote:

> On 14/11/13 18:51, Randall Burt wrote:
>> 
>> On Nov 14, 2013, at 11:30 AM, Christopher Armstrong
>> mailto:chris.armstr...@rackspace.com>>
>>  wrote:
>> 
>>> On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt
>>> mailto:randall.b...@rackspace.com>> wrote:
>>>Regarding web hook execution and cool down, I think the response
>>>should be something like 307 if the hook is on cool down with an
>>>appropriate retry-after header.
> 
> I strongly disagree with this even ignoring the security issue mentioned 
> below. Being in the cooldown period is NOT an error, and the caller should 
> absolutely NOT try again later - the request has been received and correctly 
> acted upon (by doing nothing).

But how do I know nothing was done? I may have very good reasons to re-scale 
outside of ceilometer or other mechanisms and absolutely SHOULD try again 
later.  As it stands, I have no way of knowing that my scaling action didn't 
happen without examining my physical resources. 307 is a legitimate response in 
these cases, but I'm certainly open to other suggestions.

> 
>>> Indicating whether a webhook was found or whether it actually executed
>>> anything may be an information leak, since webhook URLs require no
>>> additional authentication other than knowledge of the URL itself.
>>> Responding with only 202 means that people won't be able to guess at
>>> random URLs and know when they've found one.
>> 
>> Perhaps, but I also miss important information as a legitimate caller as
>> to whether or not my scaling action actually happened or I've been a
>> little too aggressive with my curl commands. The fact that I get
>> anything other than 404 (which the spec returns if its not a legit hook)
>> means I've found *something* and can simply call it endlessly in a loop
>> causing havoc. Perhaps the web hooks *should* be authenticated? This
>> seems like a pretty large hole to me, especially if I can max someone's
>> resources by guessing the right url.
> 
> Web hooks MUST be authenticated.
> 
> cheers,
> Zane.
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

[Randall Burt]

On Nov 14, 2013, at 11:30 AM, Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>>
 wrote:

On Thu, Nov 14, 2013 at 11:16 AM, Randall Burt 
mailto:randall.b...@rackspace.com>> wrote:
Good stuff! Some questions/comments:

If web hooks are associated with policies and policies are independent 
entities, how does a web hook specify the scaling group to act on? Does calling 
the web hook activate the policy on every associated scaling group?


Not sure what you mean by "policies are independent entities". You may have 
missed that the policy resource lives hierarchically under the group resource. 
Policies are strictly associated with one scaling group, so when a policy is 
executed (via a webhook), it's acting on the scaling group that the policy is 
associated with.

Whoops. Yeah, I missed that.



Regarding web hook execution and cool down, I think the response should be 
something like 307 if the hook is on cool down with an appropriate retry-after 
header.

Indicating whether a webhook was found or whether it actually executed anything 
may be an information leak, since webhook URLs require no additional 
authentication other than knowledge of the URL itself. Responding with only 202 
means that people won't be able to guess at random URLs and know when they've 
found one.

Perhaps, but I also miss important information as a legitimate caller as to 
whether or not my scaling action actually happened or I've been a little too 
aggressive with my curl commands. The fact that I get anything other than 404 
(which the spec returns if its not a legit hook) means I've found *something* 
and can simply call it endlessly in a loop causing havoc. Perhaps the web hooks 
*should* be authenticated? This seems like a pretty large hole to me, 
especially if I can max someone's resources by guessing the right url.


On Nov 14, 2013, at 10:57 AM, Randall Burt 
mailto:randall.b...@rackspace.com>>
 wrote:


On Nov 14, 2013, at 10:19 AM, Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>>
 wrote:

http://docs.heatautoscale.apiary.io/

I've thrown together a rough sketch of the proposed API for autoscaling. It's 
written in API-Blueprint format (which is a simple subset of Markdown) and 
provides schemas for inputs and outputs using JSON-Schema. The source document 
is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp


Things we still need to figure out:

- how to scope projects/domains. put them in the URL? get them from the token?

This may be moot considering the latest from the keystone devs regarding token 
scoping to domains/projects. Basically, a token is scoped to a single 
domain/project from what I understood, so domain/project is implicit. I'm still 
of the mind that the tenant doesn't belong so early in the URI, since we can 
already surmise the actual tenant from the authentication context, but that's 
something for Openstack at large to agree on.

- how webhooks are done (though this shouldn't affect the API too much; they're 
basically just opaque)

Please read and comment :)


--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

[Randall Burt]

Good stuff! Some questions/comments:

If web hooks are associated with policies and policies are independent 
entities, how does a web hook specify the scaling group to act on? Does calling 
the web hook activate the policy on every associated scaling group?

Regarding web hook execution and cool down, I think the response should be 
something like 307 if the hook is on cool down with an appropriate retry-after 
header.

On Nov 14, 2013, at 10:57 AM, Randall Burt 
mailto:randall.b...@rackspace.com>>
 wrote:


On Nov 14, 2013, at 10:19 AM, Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>>
 wrote:

http://docs.heatautoscale.apiary.io/

I've thrown together a rough sketch of the proposed API for autoscaling. It's 
written in API-Blueprint format (which is a simple subset of Markdown) and 
provides schemas for inputs and outputs using JSON-Schema. The source document 
is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp


Things we still need to figure out:

- how to scope projects/domains. put them in the URL? get them from the token?

This may be moot considering the latest from the keystone devs regarding token 
scoping to domains/projects. Basically, a token is scoped to a single 
domain/project from what I understood, so domain/project is implicit. I'm still 
of the mind that the tenant doesn't belong so early in the URI, since we can 
already surmise the actual tenant from the authentication context, but that's 
something for Openstack at large to agree on.

- how webhooks are done (though this shouldn't affect the API too much; they're 
basically just opaque)

Please read and comment :)


--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] rough draft of Heat autoscaling API

[Randall Burt]

On Nov 14, 2013, at 10:19 AM, Christopher Armstrong 
mailto:chris.armstr...@rackspace.com>>
 wrote:

http://docs.heatautoscale.apiary.io/

I've thrown together a rough sketch of the proposed API for autoscaling. It's 
written in API-Blueprint format (which is a simple subset of Markdown) and 
provides schemas for inputs and outputs using JSON-Schema. The source document 
is currently at https://github.com/radix/heat/raw/as-api-spike/autoscaling.apibp


Things we still need to figure out:

- how to scope projects/domains. put them in the URL? get them from the token?

This may be moot considering the latest from the keystone devs regarding token 
scoping to domains/projects. Basically, a token is scoped to a single 
domain/project from what I understood, so domain/project is implicit. I'm still 
of the mind that the tenant doesn't belong so early in the URI, since we can 
already surmise the actual tenant from the authentication context, but that's 
something for Openstack at large to agree on.

- how webhooks are done (though this shouldn't affect the API too much; they're 
basically just opaque)

Please read and comment :)


--
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Do we need to clean up resource_id after deletion?

[Randall Burt]

My thoughts exactly. I meant to dig into the soft-delete code to see if those 
changes handled resource_id differently but I got to traveling and forgot. IMO, 
if it universally needs doing, then it should be done in resource.Resource and 
be cognizant of deletion policy.

From: Clint Byrum [cl...@fewbar.com]
Sent: Friday, November 01, 2013 11:30 PM
To: openstack-dev
Subject: Re: [openstack-dev] [Heat] Do we need to clean up resource_id after
deletion?

Excerpts from Christopher Armstrong's message of 2013-11-01 11:34:56 -0700:
> Vijendar and I are trying to figure out if we need to set the resource_id
> of a resource to None when it's being deleted.
>
> This is done in a few resources, but not everywhere. To me it seems either
>
> a) redundant, since the resource is going to be deleted anyway (thus
> deleting the row in the DB that has the resource_id column)
> b) actively harmful to useful debuggability, since if the resource is
> soft-deleted, you'll not be able to find out what physical resource it
> represented before it's cleaned up.
>
> Is there some specific reason we should be calling resource_id_set(None) in
> a check_delete_complete method?
>

I've often wondered why some do it, and some don't.

Seems to me that it should be done not inside each resource plugin but
in the generic resource handling code.

However, I have not given this much thought. Perhaps others can provide
insight into why it has been done that way.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [heat] Proposal for new heat-core member

[Randall Burt]

Thanks all. I'm honored to join this great team!


 Original message 
From: Steve Baker 
Date: 10/29/2013 3:24 PM (GMT-06:00)
To: openstack-dev@lists.openstack.org
Cc: Randall Burt 
Subject: Re: [openstack-dev] [heat] Proposal for new heat-core member


I count enough core +1s. Congrats Randall, well deserved!

On 10/26/2013 08:12 AM, Steven Dake wrote:
Hi,

I would like to propose Randall Burt for Heat Core.  He has shown interest in 
Heat by participating in IRC and providing high quality reviews.  The most 
important aspect in my mind of joining Heat Core is output and quality of 
reviews.  Randall has been involved in Heat reviews for atleast 6 months.  He 
has had 172 reviews over the last 6 months staying "in the pack" [1] of core 
heat reviewers.  His 90 day stats are also encouraging, with 97 reviews 
(compared to the top reviewer Steve Hardy with 444 reviews).  Finally his 30 
day stats also look good, beating out 3 core reviewers [2] on output with good 
quality reviews.

Please have a vote +1/-1 and take into consideration: 
https://wiki.openstack.org/wiki/Heat/CoreTeam

Regards,
-steve

[1] http://russellbryant.net/openstack-stats/heat-reviewers-180.txt
[2] http://russellbryant.net/openstack-stats/heat-reviewers-30.txt



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Comments on Steve Baker's Proposal on HOT Software Config

[Randall Burt]

On Oct 28, 2013, at 9:49 AM, Steven Hardy  wrote:

> On Mon, Oct 28, 2013 at 02:33:40PM +0000, Randall Burt wrote:
>> 
>> On Oct 28, 2013, at 8:53 AM, Steven Hardy 
>> wrote:
>> 
>>> On Sun, Oct 27, 2013 at 11:23:20PM -0400, Lakshminaraya Renganarayana wrote:
>>>> A few us at IBM studied Steve Baker's proposal on HOT Software
>>>> Configuration. Overall the proposed constructs and syntax are great -- we
>>>> really like the clean syntax and concise specification of components. We
>>>> would like to propose a few minor extensions that help with better
>>>> expression of dependencies among components and resources, and in-turn
>>>> enable cross-vm coordination. We have captured our thoughts on this on the
>>>> following Wiki page
>>>> 
>>>> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-ibm-response
>>> 
>>> Thanks for putting this together.  I'll post inline below with cut/paste
>>> from the wiki followed by my response/question:
>>> 
>>>> E2: Allow usage of component outputs (similar to resources):
>>>> There are fundamental differences between components and resources...
>>> 
>>> So... lately I've been thinking this is not actually true, and that
>>> components are really just another type of resource.  If we can implement
>>> the software-config functionality without inventing a new template
>>> abstraction, IMO a lot of the issues described in your wiki page no longer
>>> exist.
>>> 
>>> Can anyone provide me with a clear argument for what the "fundamental
>>> differences" actually are?
>>> 
>>> My opinion is we could do the following:
>>> - Implement software config "components" as ordinary resources, using the
>>> existing interfaces (perhaps with some enhancements to dependency
>>> declaration)
>>> - Give OS::Nova::Server a components property, which simply takes a list of
>>> resources which describe the software configuration(s) to be applied
>> 
>> I see the appeal here, but I'm leaning toward having the components define 
>> the resources they apply to rather than extending the interfaces of every 
>> compute-related resource we have or may have in the future. True, this may 
>> make things trickier in some respects with regard to bootstrapping the 
>> compute resource, but then again, don't most configuration management 
>> systems work on active compute instances?
> 
> What "every" though?  Don't we have exactly one compute resource,
> OS::Nova::Server?  (I'm assuming this functionality won't be available via
> AWS compatible Instance resource)

Yes, I suppose it wouldn't do to go extending the AWS compatibility interface 
with this functionality, so I withdraw my concern.

> 
> Steve
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Comments on Steve Baker's Proposal on HOT Software Config

[Randall Burt]

On Oct 28, 2013, at 8:53 AM, Steven Hardy 
 wrote:

> On Sun, Oct 27, 2013 at 11:23:20PM -0400, Lakshminaraya Renganarayana wrote:
>> A few us at IBM studied Steve Baker's proposal on HOT Software
>> Configuration. Overall the proposed constructs and syntax are great -- we
>> really like the clean syntax and concise specification of components. We
>> would like to propose a few minor extensions that help with better
>> expression of dependencies among components and resources, and in-turn
>> enable cross-vm coordination. We have captured our thoughts on this on the
>> following Wiki page
>> 
>> https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config-ibm-response
> 
> Thanks for putting this together.  I'll post inline below with cut/paste
> from the wiki followed by my response/question:
> 
>> E2: Allow usage of component outputs (similar to resources):
>> There are fundamental differences between components and resources...
> 
> So... lately I've been thinking this is not actually true, and that
> components are really just another type of resource.  If we can implement
> the software-config functionality without inventing a new template
> abstraction, IMO a lot of the issues described in your wiki page no longer
> exist.
> 
> Can anyone provide me with a clear argument for what the "fundamental
> differences" actually are?
> 
> My opinion is we could do the following:
> - Implement software config "components" as ordinary resources, using the
>  existing interfaces (perhaps with some enhancements to dependency
>  declaration)
> - Give OS::Nova::Server a components property, which simply takes a list of
>  resources which describe the software configuration(s) to be applied

I see the appeal here, but I'm leaning toward having the components define the 
resources they apply to rather than extending the interfaces of every 
compute-related resource we have or may have in the future. True, this may make 
things trickier in some respects with regard to bootstrapping the compute 
resource, but then again, don't most configuration management systems work on 
active compute instances?

> 
> This provides a lot of benefits:
> - Uniformity of interfaces (solves many of the interface-mapping issues you
>  discuss in the wiki)
> - Can use provider resources and environments functionality unmodified
> - Conceptually simple, we don't have to confuse everyone with a new
>  abstraction sub-type and related terminology
> - Resources describing software components will be stateful, as described
>  in (E4), only the states would be the existing resource states, e.g
>  CREATE, IN_PROGRESS == CONFIGURING, and CREATE, COMPLETE ==
>  CONFIG_COMPLETE
> 
> Thoughts?

Completely agree here. So far, I've not seen how components differ from 
resources save for some name and superficial syntax changes. Ordering component 
resources can likely be achieved with the existing "depends-on" functionality 
as well.

> 
> Steve
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] Network topologies [and more]

[Randall Burt]

On Oct 28, 2013, at 9:07 AM, Mike Spreitzer 
mailto:mspre...@us.ibm.com>>
 wrote:

Zane Bitter mailto:zbit...@redhat.com>> wrote on 10/28/2013 
06:47:50 AM:
> On 27/10/13 16:37, Edgar Magana wrote:
> > Heat Developers,
> >
> > I am one of the core developers for Neutron who is lately working on the
> > concept of "Network Topologies". I want to discuss with you if the
> > following blueprint will make sense to have in heat or neutron code:
> > https://blueprints.launchpad.net/neutron/+spec/network-topologies-api
> >
> > ...
>
> It sounds to me like the only thing there that Heat is not already doing
> is to dump the existing network configuration. What if you were to
> implement just that part and do it in the format of a Heat template? (An
> independent tool to convert the JSON output to a Heat template would
> also work, I guess.)
>
> ...
>
> It does sound very much like you're trying to solve the same problem as
> Heat.
>

In my templates I have more than a network topology.  How would I combine the 
extracted/shared network topology with the other stuff I want in my heat 
template?

Well, if Neutron generated a Heat template describing a particular topology, 
you could always include that in another template as a provider resource 
assuming said template is generated with meaningful outputs. IMO, there is some 
appeal to having Neutron generate a Heat template for this feature. Its 
something directly consumable by another Openstack service allowing you to not 
only describe, but save and re-create for later any networking configuration 
using the same artifact.


Thanks,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat

[Randall Burt]

On Aug 15, 2013, at 6:20 PM, Angus Salkeld  wrote:

> On 15/08/13 17:50 -0500, Christopher Armstrong wrote:
>> *Introduction and Requirements*
>> 
>> So there's kind of a perfect storm happening around autoscaling in Heat
>> right now. It's making it really hard to figure out how I should compose
>> this email. There are a lot of different requirements, a lot of different
>> cool ideas, and a lot of projects that want to take advantage of
>> autoscaling in one way or another: Trove, OpenShift, TripleO, just to name
>> a few...
>> 
>> I'll try to list the requirements from various people/projects that may be
>> relevant to autoscaling or scaling in general.
>> 
>> 1. Some users want a service like Amazon's Auto Scaling or Rackspace's
>> Otter -- a simple API that doesn't really involve orchestration.
>> 2. If such a API exists, it makes sense for Heat to take advantage of its
>> functionality instead of reimplementing it.
>> 3. If Heat integrates with that separate API, however, that API will need
>> two ways to do its work:
>>  1. native instance-launching functionality, for the "simple" use
>>  2. a way to talk back to Heat to perform orchestration-aware scaling
>> operations.
>> 4. There may be things that are different than AWS::EC2::Instance that we
>> would want to scale (I have personally been playing around with the concept
>> of a ResourceGroup, which would maintain a nested stack of resources based
>> on an arbitrary template snippet).
>> 5. Some people would like to be able to perform manual operations on an
>> instance group -- such as Clint Byrum's recent example of "remove instance
>> 4 from resource group A".
>> 
>> Please chime in with your additional requirements if you have any! Trove
>> and TripleO people, I'm looking at you :-)
>> 
>> 
>> *TL;DR*
>> 
>> Point 3.2. above is the main point of this email: exactly how should the
>> autoscaling API talk back to Heat to tell it to add more instances? I
>> included the other points so that we keep them in mind while considering a
>> solution.
>> 
>> *Possible Solutions*
>> 
>> I have heard at least three possibilities so far:
>> 
>> 1. the autoscaling API should maintain a full template of all the nodes in
>> the autoscaled nested stack, manipulate it locally when it wants to add or
>> remove instances, and post an update-stack to the nested-stack associated
>> with the InstanceGroup.
>> 
>> Pros: It doesn't require any changes to Heat.
>> 
>> Cons: It puts a lot of burden of state management on the autoscale API, and
>> it arguably spreads out the responsibility of "orchestration" to the
>> autoscale API. Also arguable is that automated agents outside of Heat
>> shouldn't be managing an "internal" template, which are typically developed
>> by devops people and kept in version control.
>> 
>> 2. There should be a new custom-built API for doing exactly what the
>> autoscaling service needs on an InstanceGroup, named something unashamedly
>> specific -- like "instance-group-adjust".
>> 
>> Pros: It'll do exactly what it needs to do for this use case; very little
>> state management in autoscale API; it lets Heat do all the orchestration
>> and only give very specific delegation to the external autoscale API.
>> 
>> Cons: The API grows an additional method for a specific use case.
> 
> I like this one above:
> adjust(new_size, victim_list=['i1','i7'])
> 
> So if you are reducing the new_size we look in the victim_list to
> choose those first. This should cover Clint's use case as well.
> 
> -Angus

We could just support victim_list=[1, 7], since these groups are collections of 
identical
resources. Simple indexing should be sufficient, I would think.

Perhaps separating the stimulus from the actions to take would let us 
design/build toward different policy implementations. Initially, we could have 
a HeatScalingPolicy that works with the signals that a scaling group can 
handle. When/if AS becomes an API outside of Heat, we can implement a fairly 
simple NovaScalingPolicy that includes the args to pass to nova boot.

>> 3. the autoscaling API should update the "Size" Property of the
>> InstanceGroup resource in the stack that it is placed in. This would
>> require the ability to PATCH a specific piece of a template (an operation
>> isomorphic to update-stack).

I think a PATCH semantic for updates would be generally useful in terms of 
"quality of life" for API users. Not having to pass the complete state and 
param values for trivial updates would be quite nice regardless of its 
implications to AS.

>> 
>> Pros: The API modification is generic, simply a more optimized version of
>> update-stack; very little state management required in autoscale API.
>> 
>> Cons: This would essentially require manipulating the user-provided
>> template.  (unless we have a concept of "private properties", which perhaps
>> wouldn't appear in the template as provided by the user, but can be
>> manipulated with such an update stack operation?)
>> 
>> 
>> *Addenda*

Re: [openstack-dev] [Heat] Multi region support for Heat

[Randall Burt]

On Jul 23, 2013, at 11:03 AM, Clint Byrum 
 wrote:

> Excerpts from Steve Baker's message of 2013-07-22 21:43:05 -0700:
>> On 07/23/2013 10:46 AM, Angus Salkeld wrote:
>>> On 22/07/13 16:52 +0200, Bartosz Górski wrote:
 Hi folks,
 
 I would like to start a discussion about the blueprint I raised about
 multi region support.
 I would like to get feedback from you. If something is not clear or
 you have questions do not hesitate to ask.
 Please let me know what you think.
 
 Blueprint:
 https://blueprints.launchpad.net/heat/+spec/multi-region-support
 
 Wikipage:
 https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat
 
>>> 
>>> What immediatley looks odd to me is you have a MultiCloud Heat talking
>>> to other Heat's in each region. This seems like unneccessary
>>> complexity to me.
>>> I would have expected one Heat to do this job.
>> 
>> It should be possible to achieve this with a single Heat installation -
>> that would make the architecture much simpler.
>> 
> 
> Agreed that it would be simpler and is definitely possible.
> 
> However, consider that having a Heat in each region means Heat is more
> resilient to failure. So focusing on a way to make multiple Heat's
> collaborate, rather than on a way to make one Heat talk to two regions
> may be a more productive exercise.

Perhaps, but wouldn't having an engine that only requires the downstream 
services running (nova, cinder, etc) in a given region be equally if not more 
resilient? A heat engine in region 1 can still provision resources in region 2 
even if the heat service in region 2 is unavailable. Seems that one could 
handle global availability via any cast, DR strategy or some other routing 
magic rather than having the engine itself implement some support for it.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev