Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On Fri, Sep 28, 2018 at 12:50 AM melanie witt wrote: > On Thu, 27 Sep 2018 17:23:26 -0500, Matt Riedemann wrote: > > On 9/27/2018 3:02 PM, Jay Pipes wrote: > >> A great example of this would be the proposed "deploy template" from > >> [2]. This is nothing more than abusing the placement traits API in order > >> to allow passthrough of instance configuration data from the nova flavor > >> extra spec directly into the nodes.instance_info field in the Ironic > >> database. It's a hack that is abusing the entire concept of the > >> placement traits concept, IMHO. > >> > >> We should have a way *in Nova* of allowing instance configuration > >> key/value information to be passed through to the virt driver's spawn() > >> method, much the same way we provide for user_data that gets exposed > >> after boot to the guest instance via configdrive or the metadata service > >> API. What this deploy template thing is is just a hack to get around the > >> fact that nova doesn't have a basic way of passing through some collated > >> instance configuration key/value information, which is a darn shame and > >> I'm really kind of annoyed with myself for not noticing this sooner. :( > > > > We talked about this in Dublin through right? We said a good thing to do > > would be to have some kind of template/profile/config/whatever stored > > off in glare where schema could be registered on that thing, and then > > you pass a handle (ID reference) to that to nova when creating the > > (baremetal) server, nova pulls it down from glare and hands it off to > > the virt driver. It's just that no one is doing that work. > > If I understood correctly, that discussion was around adding a way to > pass a desired hardware configuration to nova when booting an ironic > instance. And that it's something that isn't yet possible to do using > the existing ComputeCapabilitiesFilter. Someone please correct me if I'm > wrong there. > > That said, I still don't understand why we are talking about deprecating > the ComputeCapabilitiesFilter if there's no supported way to replace it > yet. If boolean traits are not enough to replace it, then we need to > hold off on deprecating it, right? Would the > template/profile/config/whatever in glare approach replace what the > ComputeCapabilitiesFilter is doing or no? Sorry, I'm just not clearly > understanding this yet. > > I just feel some new traits have to be defined, like Jay said, and some work has to be done on the Ironic side to make sure they expose them as traits and not by the old way. That leaves tho a question : does Ironic support custom capabilities ? If so, that leads to Jay's point about the key/pair information that's not intented for traits. If we all agree on the fact that traits shouldn't be allowed for key/value pairs, could we somehow imagine Ironic to change the customization mechanism to be boolean only ? Also, I'm a bit confused whether operators make use of Ironic capabilities for fancy operational queries, like the ones we have in https://github.com/openstack/nova/blob/3716752/nova/scheduler/filters/extra_specs_ops.py#L24-L35 and if Ironic correctly documents how to put such things into traits ? (eg. say CUSTOM_I_HAVE_MORE_THAN_2_GPUS) All of the above makes me a bit worried by a possible ComputeCapabilitiesFilter deprecation, if we aren't yet able to provide a clear upgrade path for our users. -Sylvain -melanie > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On Thu, 27 Sep 2018 17:23:26 -0500, Matt Riedemann wrote: On 9/27/2018 3:02 PM, Jay Pipes wrote: A great example of this would be the proposed "deploy template" from [2]. This is nothing more than abusing the placement traits API in order to allow passthrough of instance configuration data from the nova flavor extra spec directly into the nodes.instance_info field in the Ironic database. It's a hack that is abusing the entire concept of the placement traits concept, IMHO. We should have a way *in Nova* of allowing instance configuration key/value information to be passed through to the virt driver's spawn() method, much the same way we provide for user_data that gets exposed after boot to the guest instance via configdrive or the metadata service API. What this deploy template thing is is just a hack to get around the fact that nova doesn't have a basic way of passing through some collated instance configuration key/value information, which is a darn shame and I'm really kind of annoyed with myself for not noticing this sooner. :( We talked about this in Dublin through right? We said a good thing to do would be to have some kind of template/profile/config/whatever stored off in glare where schema could be registered on that thing, and then you pass a handle (ID reference) to that to nova when creating the (baremetal) server, nova pulls it down from glare and hands it off to the virt driver. It's just that no one is doing that work. If I understood correctly, that discussion was around adding a way to pass a desired hardware configuration to nova when booting an ironic instance. And that it's something that isn't yet possible to do using the existing ComputeCapabilitiesFilter. Someone please correct me if I'm wrong there. That said, I still don't understand why we are talking about deprecating the ComputeCapabilitiesFilter if there's no supported way to replace it yet. If boolean traits are not enough to replace it, then we need to hold off on deprecating it, right? Would the template/profile/config/whatever in glare approach replace what the ComputeCapabilitiesFilter is doing or no? Sorry, I'm just not clearly understanding this yet. -melanie ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On 9/27/2018 3:02 PM, Jay Pipes wrote: A great example of this would be the proposed "deploy template" from [2]. This is nothing more than abusing the placement traits API in order to allow passthrough of instance configuration data from the nova flavor extra spec directly into the nodes.instance_info field in the Ironic database. It's a hack that is abusing the entire concept of the placement traits concept, IMHO. We should have a way *in Nova* of allowing instance configuration key/value information to be passed through to the virt driver's spawn() method, much the same way we provide for user_data that gets exposed after boot to the guest instance via configdrive or the metadata service API. What this deploy template thing is is just a hack to get around the fact that nova doesn't have a basic way of passing through some collated instance configuration key/value information, which is a darn shame and I'm really kind of annoyed with myself for not noticing this sooner. :( We talked about this in Dublin through right? We said a good thing to do would be to have some kind of template/profile/config/whatever stored off in glare where schema could be registered on that thing, and then you pass a handle (ID reference) to that to nova when creating the (baremetal) server, nova pulls it down from glare and hands it off to the virt driver. It's just that no one is doing that work. -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On 09/26/2018 05:48 PM, melanie witt wrote: On Tue, 25 Sep 2018 12:08:03 -0500, Matt Riedemann wrote: On 9/25/2018 8:36 AM, John Garbutt wrote: Another thing is about existing flavors configured for these capabilities-scoped specs. Are you saying during the deprecation we'd continue to use those even if the filter is disabled? In the review I had suggested that we add a pre-upgrade check which inspects the flavors and if any of these are found, we report a warning meaning those flavors need to be updated to use traits rather than capabilities. Would that be reasonable? I like the idea of a warning, but there are features that have not yet moved to traits: https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html There is a more general plan that will help, but its not quite ready yet: https://review.openstack.org/#/c/504952/ As such, I think we can't get pull the plug on flavors including capabilities and passing them to Ironic, but (after a cycle of deprecation) I think we can now stop pushing capabilities from Ironic into Nova and using them for placement. Forgive my ignorance, but if traits are not on par with capabilities, why are we deprecating the capabilities filter? I would like to know the answer to this as well. In short, traits were never designed to be key/value pairs. They are simple strings indicating boolean capabilities. Ironic "capabilities" are key/value metadata pairs. *Some* of those Ironic "capabilities" are possible to create as boolean traits. For example, you can change the boot_mode=uefi and boot_mode=bios Ironic capabilities to be a trait called CUSTOM_BOOT_MODE_UEFI or CUSTOM_BOOT_MODE_BIOS [1]. Other Ironic "capabilities" are not, in fact, capabilities at all. Instead, they are just random key/value pairs that are not boolean in nature nor do they represent a capability of the baremetal hardware. A great example of this would be the proposed "deploy template" from [2]. This is nothing more than abusing the placement traits API in order to allow passthrough of instance configuration data from the nova flavor extra spec directly into the nodes.instance_info field in the Ironic database. It's a hack that is abusing the entire concept of the placement traits concept, IMHO. We should have a way *in Nova* of allowing instance configuration key/value information to be passed through to the virt driver's spawn() method, much the same way we provide for user_data that gets exposed after boot to the guest instance via configdrive or the metadata service API. What this deploy template thing is is just a hack to get around the fact that nova doesn't have a basic way of passing through some collated instance configuration key/value information, which is a darn shame and I'm really kind of annoyed with myself for not noticing this sooner. :( -jay [1] As I've asked for in the past, it would be great to have Ironic contributors push patches to the os-traits library for standardized baremetal capabilities like boot modes. Please do consider contributing there. [2] https://review.openstack.org/#/c/504952/16/specs/approved/deploy-templates.rst ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On Tue, 25 Sep 2018 12:08:03 -0500, Matt Riedemann wrote: On 9/25/2018 8:36 AM, John Garbutt wrote: Another thing is about existing flavors configured for these capabilities-scoped specs. Are you saying during the deprecation we'd continue to use those even if the filter is disabled? In the review I had suggested that we add a pre-upgrade check which inspects the flavors and if any of these are found, we report a warning meaning those flavors need to be updated to use traits rather than capabilities. Would that be reasonable? I like the idea of a warning, but there are features that have not yet moved to traits: https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html There is a more general plan that will help, but its not quite ready yet: https://review.openstack.org/#/c/504952/ As such, I think we can't get pull the plug on flavors including capabilities and passing them to Ironic, but (after a cycle of deprecation) I think we can now stop pushing capabilities from Ironic into Nova and using them for placement. Forgive my ignorance, but if traits are not on par with capabilities, why are we deprecating the capabilities filter? I would like to know the answer to this as well. -melanie ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On 9/25/2018 8:36 AM, John Garbutt wrote: Another thing is about existing flavors configured for these capabilities-scoped specs. Are you saying during the deprecation we'd continue to use those even if the filter is disabled? In the review I had suggested that we add a pre-upgrade check which inspects the flavors and if any of these are found, we report a warning meaning those flavors need to be updated to use traits rather than capabilities. Would that be reasonable? I like the idea of a warning, but there are features that have not yet moved to traits: https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html There is a more general plan that will help, but its not quite ready yet: https://review.openstack.org/#/c/504952/ As such, I think we can't get pull the plug on flavors including capabilities and passing them to Ironic, but (after a cycle of deprecation) I think we can now stop pushing capabilities from Ironic into Nova and using them for placement. Forgive my ignorance, but if traits are not on par with capabilities, why are we deprecating the capabilities filter? -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On Thu, 20 Sep 2018 at 16:02, Matt Riedemann wrote: > On 9/20/2018 4:16 AM, John Garbutt wrote: > > Following on from the PTG discussions, I wanted to bring everyone's > > attention to Nova's plans to deprecate ComputeCapabilitiesFilter, > > including most of the the integration with Ironic Capabilities. > > > > To be specific, this is my proposal in code form: > > https://review.openstack.org/#/c/603102/ > > > > Once the code we propose to deprecate is removed we will stop using > > capabilities pushed up from Ironic for 'scheduling', but we would still > > pass capabilities request in the flavor down to Ironic (until we get > > some standard traits and/or deploy templates sorted for things like > UEFI). > > > > Functionally, we believe all use cases can be replaced by using the > > simpler placement traits (this is more efficient than post placement > > filtering done using capabilities): > > > https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/ironic-driver-traits.html > > > > Please note the recent addition of forbidden traits that helps improve > > the usefulness of the above approach: > > > https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-forbidden-traits.html > > > > For example, a flavor request for GPUs >= 2 could be replaced by a > > custom trait trait that reports if a given Ironic node has > > CUSTOM_MORE_THAN_2_GPUS. That is a bad example (longer term we don't > > want to use traits for this, but that is a discussion for another day) > > but it is the example that keeps being raised in discussions on this > topic. > > > > The main reason for reaching out in this email is to ask if anyone has > > needs that the ResourceClass and Traits scheme does not currently > > address, or can think of a problem with a transition to the newer > approach. > > I left a few comments in the change, but I'm assuming as part of the > deprecation we'd remove the filter from the default enabled_filters list > so new installs don't automatically get warnings during scheduling? > +1 Good point, we totally need to do that. > Another thing is about existing flavors configured for these > capabilities-scoped specs. Are you saying during the deprecation we'd > continue to use those even if the filter is disabled? In the review I > had suggested that we add a pre-upgrade check which inspects the flavors > and if any of these are found, we report a warning meaning those flavors > need to be updated to use traits rather than capabilities. Would that be > reasonable? > I like the idea of a warning, but there are features that have not yet moved to traits: https://specs.openstack.org/openstack/ironic-specs/specs/juno-implemented/uefi-boot-for-ironic.html There is a more general plan that will help, but its not quite ready yet: https://review.openstack.org/#/c/504952/ As such, I think we can't get pull the plug on flavors including capabilities and passing them to Ironic, but (after a cycle of deprecation) I think we can now stop pushing capabilities from Ironic into Nova and using them for placement. Thanks, John ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [ironic] [nova] [tripleo] Deprecation of Nova's integration with Ironic Capabilities and ComputeCapabilitiesFilter
On 9/20/2018 4:16 AM, John Garbutt wrote: Following on from the PTG discussions, I wanted to bring everyone's attention to Nova's plans to deprecate ComputeCapabilitiesFilter, including most of the the integration with Ironic Capabilities. To be specific, this is my proposal in code form: https://review.openstack.org/#/c/603102/ Once the code we propose to deprecate is removed we will stop using capabilities pushed up from Ironic for 'scheduling', but we would still pass capabilities request in the flavor down to Ironic (until we get some standard traits and/or deploy templates sorted for things like UEFI). Functionally, we believe all use cases can be replaced by using the simpler placement traits (this is more efficient than post placement filtering done using capabilities): https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/ironic-driver-traits.html Please note the recent addition of forbidden traits that helps improve the usefulness of the above approach: https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-forbidden-traits.html For example, a flavor request for GPUs >= 2 could be replaced by a custom trait trait that reports if a given Ironic node has CUSTOM_MORE_THAN_2_GPUS. That is a bad example (longer term we don't want to use traits for this, but that is a discussion for another day) but it is the example that keeps being raised in discussions on this topic. The main reason for reaching out in this email is to ask if anyone has needs that the ResourceClass and Traits scheme does not currently address, or can think of a problem with a transition to the newer approach. I left a few comments in the change, but I'm assuming as part of the deprecation we'd remove the filter from the default enabled_filters list so new installs don't automatically get warnings during scheduling? Another thing is about existing flavors configured for these capabilities-scoped specs. Are you saying during the deprecation we'd continue to use those even if the filter is disabled? In the review I had suggested that we add a pre-upgrade check which inspects the flavors and if any of these are found, we report a warning meaning those flavors need to be updated to use traits rather than capabilities. Would that be reasonable? -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
