Hi Zane, please count me in :-)
Cheers, Lingxian Kong On Wed, Jun 6, 2018 at 4:52 AM, Remo Mattei <[email protected]> wrote: > I will be happy to check it out. > > Remo > > > On Jun 5, 2018, at 9:19 AM, Zane Bitter <[email protected]> wrote: > > I've been doing some investigation into the Service Catalog in Kubernetes > and how we can get OpenStack resources to show up in the catalog for use by > applications running in Kubernetes. (The Big 3 public clouds already > support this.) The short answer is via an implementation of something > called the Open Service Broker API, but there are shortcuts available to > make it easier to do. > > I'm convinced that this is readily achievable and something we ought to do > as a community. > > I've put together a (long-winded) FAQ below to answer all of your > questions about it. > > Would you be interested in working on a new project to implement this > integration? Reply to this thread and let's collect a list of volunteers to > form the initial core review team. > > cheers, > Zane. > > > What is the Open Service Broker API? > ------------------------------------ > > The Open Service Broker API[1] is a standard way to expose external > resources to applications running in a PaaS. It was originally developed in > the context of CloudFoundry, but the same standard was adopted by > Kubernetes (and hence OpenShift) in the form of the Service Catalog > extension[2]. (The Service Catalog in Kubernetes is the component that > calls out to a service broker.) So a single implementation can cover the > most popular open-source PaaS offerings. > > In many cases, the services take the form of simply a pre-packaged > application that also runs inside the PaaS. But they don't have to be - > services can be anything. Provisioning via the service broker ensures that > the services requested are tied in to the PaaS's orchestration of the > application's lifecycle. > > (This is certainly not the be-all and end-all of integration between > OpenStack and containers - we also need ways to tie PaaS-based applications > into the OpenStack's orchestration of a larger group of resources. Some > applications may even use both. But it's an important part of the story.) > > What sorts of services would OpenStack expose? > ---------------------------------------------- > > Some example use cases might be: > > * The application needs a reliable message queue. Rather than spinning up > multiple storage-backed containers with anti-affinity policies and dealing > with the overhead of managing e.g. RabbitMQ, the application requests a > Zaqar queue from an OpenStack cloud. The overhead of running the queueing > service is amortised across all of the applications in the cloud. The queue > gets cleaned up correctly when the application is removed, since it is tied > into the application definition. > > * The application needs a database. Rather than spinning one up in a > storage-backed container and dealing with the overhead of managing it, the > application requests a Trove DB from an OpenStack cloud. > > * The application includes a service that needs to run on bare metal for > performance reasons (e.g. could also be a database). The application > requests a bare-metal server from Nova w/ Ironic for the purpose. (The same > applies to requesting a VM, but there are alternatives like KubeVirt - > which also operates through the Service Catalog - available for getting a > VM in Kubernetes. There are no non-proprietary alternatives for getting a > bare-metal server.) > > AWS[3], Azure[4], and GCP[5] all have service brokers available that > support these and many more services that they provide. I don't know of any > reason in principle not to expose every type of resource that OpenStack > provides via a service broker. > > How is this different from cloud-provider-openstack? > ---------------------------------------------------- > > The Cloud Controller[6] interface in Kubernetes allows Kubernetes itself > to access features of the cloud to provide its service. For example, if k8s > needs persistent storage for a container then it can request that from > Cinder through cloud-provider-openstack[7]. It can also request a load > balancer from Octavia instead of having to start a container running > HAProxy to load balance between multiple instances of an application > container (thus enabling use of hardware load balancers via the cloud's > abstraction for them). > > In contrast, the Service Catalog interface allows the *application* > running on Kubernetes to access features of the cloud. > > What does a service broker look like? > ------------------------------------- > > A service broker provides an HTTP API with 5 actions: > > * List the services provided by the broker > * Create an instance of a resource > * Bind the resource into an instance of the application > * Unbind the resource from an instance of the application > * Delete the resource > > The binding step is used for things like providing a set of DB credentials > to a container. You can rotate credentials when replacing a container by > revoking the existing credentials on unbind and creating a new set on bind, > without replacing the entire resource. > > Is there an easier way? > ----------------------- > > Yes! Folks from OpenShift came up with a project called the Automation > Broker[8]. To add support for a service to Automation Broker you just > create a container with an Ansible playbook to handle each of the actions > (create/bind/unbind/delete). This eliminates the need to write another > implementation of the service broker API, and allows us to simply write > Ansible playbooks instead.[9] > > (Aside: Heat uses a comparable method to allow users to manage an external > resource using Mistral workflows: the OS::Mistral::ExternalResource > resource type.) > > Support for accessing AWS resources through a service broker is also > implemented using these Ansible Playbook Bundles.[3] > > Does this mean maintaining another client interface? > ---------------------------------------------------- > > Maybe not. We already have per-project Python libraries, (deprecated) > per-project CLIs, openstackclient CLIs, openstack-sdk, shade, Heat resource > plugins, and Horizon dashboards. (Mistral actions are generated > automatically from the clients.) Some consolidation is already planned, but > it would be great not to require projects to maintain yet another interface. > > One option is to implement a tool that generates a set of playbooks for > each of the resources already exposed (via shade) in the OpenStack Ansible > modules. Then in theory we'd only need to implement the common parts once, > and then every service with support in shade would get this for free. > Ideally the same broker could be used against any OpenStack cloud (so e.g. > k8s might be running in your private cloud, but you may want its service > catalog to allow you to connect to resources in one or more public clouds) > - using shade is an advantage there because it is designed to abstract the > differences between clouds. > > Another option might be to write or generate Heat templates for each > resource type we want to expose. Then we'd only need to implement a common > way of creating a Heat stack, and just have a different template for each > resource type. This is the approach taken by the AWS playbook bundles > (except with CloudFormation, obviously). An advantage is that this allows > Heat to do any checking and type conversion required on the input > parameters. Heat templates can also be made to be fairly cloud-independent, > mainly because they make it easier to be explicit about things like ports > and subnets than on the command line, where it's more tempting to allow > things to happen in a magical but cloud-specific way. > > I'd prefer to go with the pure-Ansible autogenerated way so we can have > support for everything, but looking at the GCP[5]/Azure[4]/AWS[3] brokers > they have 10, 11 and 17 services respectively, so arguably we could get a > comparable number of features exposed without investing crazy amounts of > time if we had to write templates explicitly. > > How would authentication work? > ------------------------------ > > There are two main deployment topologies we need to consider: Kubernetes > deployed by an OpenStack tenant (Magnum-style, though not necessarily using > Magnum) and accessing resources in that tenant's project in the local > cloud, or accessing resources in some remote OpenStack cloud. > > We also need to take into account that in the second case, the Kubernetes > cluster may 'belong' to a single cloud tenant (as in the first case) or may > be shared by applications that each need to authenticate to different > OpenStack tenants. (Kubernetes has traditionally assumed the former, but I > expect it to move in the direction of allowing the latter, and it's already > fairly common for OpenShift deployments.) > > The way e.g. the AWS broker[3] works is that you can either use the > credentials provisioned to the VM that k8s is installed on (a 'Role' in AWS > parlance - note that this is completely different to a Keystone Role), or > supply credentials to authenticate to AWS remotely. > > OpenStack doesn't yet support per-instance credentials, although we're > working on it. (One thing to keep in mind is that ideally we'll want a way > to provide different permissions to the service broker and > cloud-provider-openstack.) An option in the meantime might be to provide a > way to set up credentials as part of the k8s installation. We'd also need > to have a way to specify credentials manually. Unlike for proprietary > clouds, the credentials also need to include the Keystone auth_url. We > should try to reuse openstacksdk's clouds.yaml/secure.yaml format[10] if > possible. > > The OpenShift Ansible Broker works by starting up an Ansible container on > k8s to run a playbook from the bundle, so presumably credentials can be > passed as regular k8s secrets. > > In all cases we'll want to encourage users to authenticate using Keystone > Application Credentials[11]. > > How would network integration work? > ----------------------------------- > > Kuryr[12] allows us to connect application containers in Kubernetes to > Neutron networks in OpenStack. It would be desirable if, when the user > requests a VM or bare-metal server through the service broker, it were > possible to choose between attaching to the same network as Kubernetes > pods, or to a different network. > > > [1] https://www.openservicebrokerapi.org/ > [2] https://kubernetes.io/docs/concepts/service-catalog/ > [3] https://github.com/awslabs/aws-servicebroker#aws-service-broker > [4] https://github.com/Azure/open-service-broker-azure# > open-service-broker-for-azure > [5] https://github.com/GoogleCloudPlatform/gcp- > service-broker#cloud-foundry-service-broker-for-google-cloud-platform > [6] https://github.com/kubernetes/community/blob/ > master/keps/0002-controller-manager.md#remove-cloud- > provider-code-from-kubernetes-core > [7] https://github.com/kubernetes/cloud-provider- > openstack#openstack-cloud-controller-manager > [8] http://automationbroker.io/ > [9] https://docs.openshift.org/latest/apb_devel/index.html > [10] https://docs.openstack.org/openstacksdk/latest/user/ > config/configuration.html#config-files > [11] https://docs.openstack.org/keystone/latest/user/ > application_credentials.html > [12] https://docs.openstack.org/kuryr/latest/devref/goals_ > and_use_cases.html > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
