Excerpts from Jay Pipes's message of 2016-08-03 19:47:37 -0400: >> Hi Novas and anyone interested in how to represent capabilities in a >> consistent fashion. >> >> I spent an hour creating a new os-capabilities Python library this evening: > >> http://github.com/jaypipes/os-capabilities >> Anyway, lemme know your initial thoughts please.
On 8/11/16, 3:52 PM, "Clint Byrum" <cl...@fewbar.com> wrote: > Did we ever resolve the similarities between this and Searchlight's > similar goals of providing consistent metadata to the users? > http://docs.openstack.org/developer/glance/metadefs-concepts.html > I understand your library is for operators and developers to > collaborate, but it seems like there should be some alignment with this > UI that wants to do the same thing for the end user where appropriate. The metadefs catalog wasn’t written just as a UI construct, it was actually a derivative of an effort called graffiti [0] that was entirely about capability and requirement matching and providing a way for portability across clouds. The Graffiti project was proposed at the Atlanta (Juno) OpenStack summit. Since then quite a bit of the concepts have been adopted and are covered as part of multiple different OpenStack Projects. [0] https://wiki.openstack.org/wiki/Graffiti/Architecture A key concept is to both support defining standard metadata that can be attached to various resources as well as providing a common service for deployers to register their own metadata with visibility restrictions. This can be anything from “Gold, Silver, Bronze” to “some hardware capability”. Ultimately it is up to the deployer to activate, publish, or discover the capabilities in their environment and enable them in the catalog. Glance Metadata Definition Catalog (Box 1 in the workflow diagram) * http://docs.openstack.org/developer/glance/metadefs-concepts.html * https://github.com/openstack/glance/tree/master/etc/metadefs * https://youtu.be/zJpHXdBOoeM Horizon features (Box 2 in the diagram – but also CLI) * An admin UI for managing the catalog * (Admin —> Metadata Definitions) (Kilo) * A widget for associating metadata to different resources * (Update Metadata action on each row item below) * admin -> images (Juno) * admin -> flavors (Kilo) * admin —> Host Aggregates (Kilo) * project —> images (Liberty) * project —> instances (Mitaka) * The ability to add metadata at launch time * project —> Launch Instance (ng launch instance enabled) (Mitaka) Searchlight (Box 3 in the workflow diagram) * http://launchpad.net/searchlight * https://wiki.openstack.org/wiki/Searchlight The Searchlight project is primarily intended for high performance searching across the cloud. Fulfilling the concepts of Graffiti is a side effect, but did provide some of the original inspiration for the project. We actually have blueprint we will do in “O” that will provide data mapping from metadefs to the schemas in ElasticSearch [1]. * [1] https://blueprints.launchpad.net/searchlight/+spec/configurable-dynamic-properties In addition, when this popped up in my mailbox today, a search revealed This message from last August with a few points that I’d like to help clarify below: On 8/10/15, 8:05 AM, "Jay Pipes" <jaypi...@gmail.com> wrote: > The Glance metadefs stuff is problematic in a number of ways: > 1) It wasn't written with Nova in mind at all, but rather for UI needs. > This means it introduces a bunch of constants that are different from > the constants in Nova. This is actually not the case. This was originally co-sponsored by Intel to help expose out all the CPU capabilities in Nova. The constants in the metadef catalog all come from combing through the code in Nova was a complete maze and were not available at the time from Nova (or cinder or glance or …) See overview here [2]: [2] https://wiki.openstack.org/wiki/Graffiti > 2) It uses a custom JSON format instead of JSONSchema, so we now need to > figure out the schema for these metadef documents and keep up to date > with that schema as it changes. It uses JSON schema, but surrounds it with a very lightweight envelope. The envelope is called a namespace and is simply a container of JSON schema, allowing us to manage it as a programmatic unit and as a way for cloud deployers to share the capabilities across clouds very easily. We did place a limitation on it that it cannot support nested objects. This was primarily due to the extreme difficulty of representing that construct to users in an easy to understand way: http://docs.openstack.org/developer/glance/metadefs-concepts.html#catalog-terminology > 3) It mixes qualitative things -- CPU model, features, etc -- with > quantitative things -- amount of cores, threads, etc. These two things > are precisely what we are trying to decouple from each other in the next > generation of Nova's "flavors". > > 4) It is still missing the user-facing representation of these things. > Users -- i.e. the people launching VMs in Nova -- do not want or need to > know whether the underlying hardware is running Nehalem processors or > Sandybridge processors. They only need to know the set of generic > features that are exposed to the workloads running on the host. In other > words, we are looking for a way to map "service levels" such as "Silver > Compute" or "Whizbang Amazing Compute" to a set of hardware that exposes > some features. We need that compatibility layer on top of the low-level > hardware specifics that will allow service providers to create these > "product categories" if you will. It is true that the catalog can contain anything. This include you defining a namespace as “Service Levels” with GOLD, SILVER, BRONZE in addition to specific items as mentioned above. The operator could even make the service levels Public (visible via the API to all and hence in the UI) and the other items private (only visible to the project owning them and not visible in the UI). So, the operator could tag images or flavors with both service levels and with specific other properties and decide which ones to expose the definitions to. In Horizon, we have some widgets that honor this and will selectively display metadata based on if the definition is visible to that user. It was originally envisioned to be more similar to some of the specs that have gone through on capabilities in Nova, but due to the various limitations in OpenStack at the time at least, a number of design simplifications were made to work within the way OpenStack services work. For example, all the resource types (Images, Flavors, Aggregates) have a very simple mechanism for adding data to them. Taking advantage of host aggregate filtering with flavors or images properties matching allows operators to group hosts according to an aggregate and to simply tag something like “GOLD” on both the aggregate and the image or the flavor. As hosts are discovered into the environment, they simply are added to the correct aggregate. Of course, that is a very course grained example, but leveraging simple tagging or key value pairs could be done to tag service level info on the flavor and the flavors could be filtered that way. Anyway, all of the above is not meant in any way as a challenge to the current work, but meant as a way to help understand what is already in place and to help clear up a few points so that they may be considered in the current work. Thanks, Travis __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev