Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt
Thanks Devdatta/Maxime for your comments. I am definitely not rigid about implementing the workflow in Nova and it's well known that there can be multiple integration points for this work including that in docker itself. However, there are two prime reasons why we chose Nova as integration point in OpenStack: 1. Minimal changes to a VM boot workflow. No need to depend on Swift or any other service. 2. Faster boot up times - since the downloading of the virtual machine image is negated. Downloading the docker filesystems should be more or less easier. Some comments inline. Thanks, Sudipto On 27/07/16 11:59 PM, Maxime Belanger wrote: +1 on this, Still you loose all the great stuff about the containers but it is a first step towards native container orchestration platform. IMHO, it is not about just losing stuff. We are not emulating a docker workflow. The expectation is to have the ability to run a container inside a virtual machine and then take that filesystem out and run it natively on the hardware as desired. You can debate on whether it's really needed in Nova or elsewhere and I think that's a fair debate. I am sure there are further technical challenges to overcome, if we want to think in this direction. *From:* Devdatta Kulkarni <devdatta.kulka...@rackspace.com> *Sent:* July 27, 2016 12:21:30 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Hi Sudipta, There is another approach you can consider which does not need any changes to Nova. The approach works as follows: - Save the container image tar in Swift - Generate a Swift tempURL for the container file - Boot Nova vm and pass instructions for following steps through cloud init / user data - download the container file from Swift (wget) I believe this has to be carried out for every docker image? That is if i have a nginx image and it's provisioned twice, a fresh copy of such has to be wget'ed every time? IF the nova workflow is acceptable, then there can be optimizations thought around this. At this moment, my implementation copies the cached image for each of the containers - atleast making further boots faster. Also, how do you tackle the problem with snapshoting a container? - load it (docker load) - run it (docker run) Do you run the docker native commands inside the virtual machine? In such a case, do you actually install docker as a part of the cloud-init scripts? Do you have numbers w.r.t the boot time of the container image in this case? We have implemented this approach in Solum (where we use Heat for deploying a VM and then run application container on it by providing above instructions through user_data of the HOT). Thanks, Devdatta - From: Sudipta Biswas <sbisw...@linux.vnet.ibm.com> Sent: Wednesday, July 27, 2016 9:17 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Premise: While working with customers, we have realized: - They want to use containers but are wary of using the same host kernel for multiple containers. - They already have a significant investment (including skills) in OpenStack's Virtual Machine workflow and would like to re-use it as much as possible. - They are very interested in using docker images. There are some existing approaches like Hyper, Secure Containers workflows which already tries to address the first point. But we wanted to arrive at an approach that addresses all the above three in context of OpenStack Nova with minimalist changes. Design Considerations: We tried a few experiments with the present libvirt driver in nova to accomplish a work flow to deploy containers inside virtual machines in OpenStack via Nova. The fundamental premise of our approach is to run a single container encapsulated in a single VM. This VM image just has a bare minimum operating system required to run it. The container filesystem comes from the docker image. We would like to get the feedback on the below approaches from the community before proposing this as a spec or blueprint. Approach 1 User workflow: 1. The docker image is obtained in the form of a tar file. 2. Upload this tar file in glance. This support is already there in glance were a container-type of docker is supported. 3. Use this image along with nova libvirt driver to deploy a virtual machine. Following are some of the changes to the OpenStack code that implements this approach: 1. Define a new conf parameter in nova called – base_vm_image=/var/lib/libvirt/images/baseimage.qcow2 This option is used to specify the base VM image. 2. define a new sub_virt_type = container in nova conf. Setting this parameter will ensure mounting of the container filesystem inside the VM.
Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt
+1 on this, Still you loose all the great stuff about the containers but it is a first step towards native container orchestration platform. From: Devdatta Kulkarni <devdatta.kulka...@rackspace.com> Sent: July 27, 2016 12:21:30 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Hi Sudipta, There is another approach you can consider which does not need any changes to Nova. The approach works as follows: - Save the container image tar in Swift - Generate a Swift tempURL for the container file - Boot Nova vm and pass instructions for following steps through cloud init / user data - download the container file from Swift (wget) - load it (docker load) - run it (docker run) We have implemented this approach in Solum (where we use Heat for deploying a VM and then run application container on it by providing above instructions through user_data of the HOT). Thanks, Devdatta - From: Sudipta Biswas <sbisw...@linux.vnet.ibm.com> Sent: Wednesday, July 27, 2016 9:17 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Premise: While working with customers, we have realized: - They want to use containers but are wary of using the same host kernel for multiple containers. - They already have a significant investment (including skills) in OpenStack's Virtual Machine workflow and would like to re-use it as much as possible. - They are very interested in using docker images. There are some existing approaches like Hyper, Secure Containers workflows which already tries to address the first point. But we wanted to arrive at an approach that addresses all the above three in context of OpenStack Nova with minimalist changes. Design Considerations: We tried a few experiments with the present libvirt driver in nova to accomplish a work flow to deploy containers inside virtual machines in OpenStack via Nova. The fundamental premise of our approach is to run a single container encapsulated in a single VM. This VM image just has a bare minimum operating system required to run it. The container filesystem comes from the docker image. We would like to get the feedback on the below approaches from the community before proposing this as a spec or blueprint. Approach 1 User workflow: 1. The docker image is obtained in the form of a tar file. 2. Upload this tar file in glance. This support is already there in glance were a container-type of docker is supported. 3. Use this image along with nova libvirt driver to deploy a virtual machine. Following are some of the changes to the OpenStack code that implements this approach: 1. Define a new conf parameter in nova called – base_vm_image=/var/lib/libvirt/images/baseimage.qcow2 This option is used to specify the base VM image. 2. define a new sub_virt_type = container in nova conf. Setting this parameter will ensure mounting of the container filesystem inside the VM. Unless qemu and kvm are used as virt_type – this workflow will not work at this moment. 3. In the virt/libvirt/driver.py we do the following based on the sub_virt_type = container: - We create a qcow2 disk from the base_vm_image and expose that 'disk' as the boot disk for the virtual machine. Note – this is very similar to a regular virtual machine boot minus the fact that the image is not downloaded from glance but instead it is present on the host. - We download the docker image into the /var/lib/nova/instances/_base directory and then for each new virtual machine boot – we create a new directory /var/lib/nova/instances/ as it's and copy the docker filesystem to it. Note – there are subsequent improvements to this idea that could be performed around the lines of using a union filesystem approach. - The step above allows each virtual machine to have a different copy of the filesystem. - We create a 'passthrough' mount of the filesystem via libvirt. This code is also present in the nova libvirt driver and we just trigger it based on our sub_virt_type parameter. 4. A cloud init – userdata is provided that looks somewhat like this: runcmd: - mount -t 9p -o trans=virtio share_dir /mnt - chroot /mnt /bin/ The command_to_run is usually the entrypoint to for the docker image. There could be better approaches to determine the entrypoint as well (say from docker image metadata). Approach 2. In this approach, the workflow remains the same as the first one with the exception that the docker image is changed into a qcow2 image using a tool like virt-make-fs before uploading it to glance, instead of a tar file. A tool like virt-make-fs can convert a tar file to a qcow2 image very easily. This image is then downloaded on the compute node and a qcow2 disk is created/attached to the virtual machine that boots using the b
Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt
Hi Sudipta, There is another approach you can consider which does not need any changes to Nova. The approach works as follows: - Save the container image tar in Swift - Generate a Swift tempURL for the container file - Boot Nova vm and pass instructions for following steps through cloud init / user data - download the container file from Swift (wget) - load it (docker load) - run it (docker run) We have implemented this approach in Solum (where we use Heat for deploying a VM and then run application container on it by providing above instructions through user_data of the HOT). Thanks, Devdatta - From: Sudipta Biswas <sbisw...@linux.vnet.ibm.com> Sent: Wednesday, July 27, 2016 9:17 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Premise: While working with customers, we have realized: - They want to use containers but are wary of using the same host kernel for multiple containers. - They already have a significant investment (including skills) in OpenStack's Virtual Machine workflow and would like to re-use it as much as possible. - They are very interested in using docker images. There are some existing approaches like Hyper, Secure Containers workflows which already tries to address the first point. But we wanted to arrive at an approach that addresses all the above three in context of OpenStack Nova with minimalist changes. Design Considerations: We tried a few experiments with the present libvirt driver in nova to accomplish a work flow to deploy containers inside virtual machines in OpenStack via Nova. The fundamental premise of our approach is to run a single container encapsulated in a single VM. This VM image just has a bare minimum operating system required to run it. The container filesystem comes from the docker image. We would like to get the feedback on the below approaches from the community before proposing this as a spec or blueprint. Approach 1 User workflow: 1. The docker image is obtained in the form of a tar file. 2. Upload this tar file in glance. This support is already there in glance were a container-type of docker is supported. 3. Use this image along with nova libvirt driver to deploy a virtual machine. Following are some of the changes to the OpenStack code that implements this approach: 1. Define a new conf parameter in nova called – base_vm_image=/var/lib/libvirt/images/baseimage.qcow2 This option is used to specify the base VM image. 2. define a new sub_virt_type = container in nova conf. Setting this parameter will ensure mounting of the container filesystem inside the VM. Unless qemu and kvm are used as virt_type – this workflow will not work at this moment. 3. In the virt/libvirt/driver.py we do the following based on the sub_virt_type = container: - We create a qcow2 disk from the base_vm_image and expose that 'disk' as the boot disk for the virtual machine. Note – this is very similar to a regular virtual machine boot minus the fact that the image is not downloaded from glance but instead it is present on the host. - We download the docker image into the /var/lib/nova/instances/_base directory and then for each new virtual machine boot – we create a new directory /var/lib/nova/instances/ as it's and copy the docker filesystem to it. Note – there are subsequent improvements to this idea that could be performed around the lines of using a union filesystem approach. - The step above allows each virtual machine to have a different copy of the filesystem. - We create a 'passthrough' mount of the filesystem via libvirt. This code is also present in the nova libvirt driver and we just trigger it based on our sub_virt_type parameter. 4. A cloud init – userdata is provided that looks somewhat like this: runcmd: - mount -t 9p -o trans=virtio share_dir /mnt - chroot /mnt /bin/ The command_to_run is usually the entrypoint to for the docker image. There could be better approaches to determine the entrypoint as well (say from docker image metadata). Approach 2. In this approach, the workflow remains the same as the first one with the exception that the docker image is changed into a qcow2 image using a tool like virt-make-fs before uploading it to glance, instead of a tar file. A tool like virt-make-fs can convert a tar file to a qcow2 image very easily. This image is then downloaded on the compute node and a qcow2 disk is created/attached to the virtual machine that boots using the base_vm_image. Approach 3 A custom qcow2 image is created using kernel, initramfs and the docker image and uploaded to glance. No changes are needed in openstack nova. It boots as a regular VM. Changes will be needed in image generation tools and will involve few additional tasks from an operator point of view. I look forward to your comments/suggestions on the above. Thanks, S
Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt
Unfortunately, this doesn't seem to fit into Magnum as well. According to the current mission statement [1], Magnum is for provisioning, scaling, and managing Container Orchestration Engines (i.e. Kubernetes), so managing containers in VM is not consistent with Magnum's mission. It seems there is a misconception that Magnum is the home for containers in OpenStack, so all the container use cases were pushed to Magnum. However, this is not true because Magnum also has its scope and cannot support anything that goes beyond. [1] https://github.com/openstack/governance/blob/master/reference/projects.yaml#L2128 Best regards, Hongbin From: Maxime Belanger [mailto:mbelan...@internap.com] Sent: July-27-16 10:40 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt In my opinion, You are loosing so much advantages of the what a container platform already offer. Example (not exhaustive): 1. Glance is becoming your "Image Registry" * No incremental pull * No image layer caching * Decrease in speed * You have to convert from a Container image to a qcow2 image format (loosing time here and not imcremental) 1. One container per VM is exactly identical as having one service per VM * Only advantage is that your deployment recipes are less complicated 1. Scaling the app * Having to use Heat to scale a container (actually a vm). I understand why your client are asking for this as we are (as a first step) doing one container per VM because our deployment architecture is not yet ready for full container stack. But There is other way of doing what your client is asking without implementing anything in Nova. That said, there is Magnum project to support containers in an Openstack environment. Quite frankly, I am not sure nova should deal with containers as I do not see the link with current Nova responsibilities. Regards, Max From: Sudipta Biswas <sbisw...@linux.vnet.ibm.com<mailto:sbisw...@linux.vnet.ibm.com>> Sent: July 27, 2016 10:17:00 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Premise: While working with customers, we have realized: - They want to use containers but are wary of using the same host kernel for multiple containers. - They already have a significant investment (including skills) in OpenStack's Virtual Machine workflow and would like to re-use it as much as possible. - They are very interested in using docker images. There are some existing approaches like Hyper, Secure Containers workflows which already tries to address the first point. But we wanted to arrive at an approach that addresses all the above three in context of OpenStack Nova with minimalist changes. Design Considerations: We tried a few experiments with the present libvirt driver in nova to accomplish a work flow to deploy containers inside virtual machines in OpenStack via Nova. The fundamental premise of our approach is to run a single container encapsulated in a single VM. This VM image just has a bare minimum operating system required to run it. The container filesystem comes from the docker image. We would like to get the feedback on the below approaches from the community before proposing this as a spec or blueprint. Approach 1 User workflow: 1. The docker image is obtained in the form of a tar file. 2. Upload this tar file in glance. This support is already there in glance were a container-type of docker is supported. 3. Use this image along with nova libvirt driver to deploy a virtual machine. Following are some of the changes to the OpenStack code that implements this approach: 1. Define a new conf parameter in nova called - base_vm_image=/var/lib/libvirt/images/baseimage.qcow2 This option is used to specify the base VM image. 2. define a new sub_virt_type = container in nova conf. Setting this parameter will ensure mounting of the container filesystem inside the VM. Unless qemu and kvm are used as virt_type - this workflow will not work at this moment. 3. In the virt/libvirt/driver.py we do the following based on the sub_virt_type = container: - We create a qcow2 disk from the base_vm_image and expose that 'disk' as the boot disk for the virtual machine. Note - this is very similar to a regular virtual machine boot minus the fact that the image is not downloaded from glance but instead it is present on the host. - We download the docker image into the /var/lib/nova/instances/_base directory and then for each new virtual machine boot - we create a new directory /var/lib/nova/instances/ as it's and copy the docker filesystem to it. Note - there are subsequent improvements to this idea that could be performed around the lines of using a union filesystem appro
Re: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt
In my opinion, You are loosing so much advantages of the what a container platform already offer. Example (not exhaustive): 1. Glance is becoming your "Image Registry" * No incremental pull * No image layer caching * Decrease in speed * You have to convert from a Container image to a qcow2 image format (loosing time here and not imcremental) 2. One container per VM is exactly identical as having one service per VM * Only advantage is that your deployment recipes are less complicated 3. Scaling the app * Having to use Heat to scale a container (actually a vm). I understand why your client are asking for this as we are (as a first step) doing one container per VM because our deployment architecture is not yet ready for full container stack. But There is other way of doing what your client is asking without implementing anything in Nova. That said, there is Magnum project to support containers in an Openstack environment. Quite frankly, I am not sure nova should deal with containers as I do not see the link with current Nova responsibilities. Regards, Max From: Sudipta Biswas <sbisw...@linux.vnet.ibm.com> Sent: July 27, 2016 10:17:00 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][rfc] Booting docker images using nova libvirt Premise: While working with customers, we have realized: - They want to use containers but are wary of using the same host kernel for multiple containers. - They already have a significant investment (including skills) in OpenStack's Virtual Machine workflow and would like to re-use it as much as possible. - They are very interested in using docker images. There are some existing approaches like Hyper, Secure Containers workflows which already tries to address the first point. But we wanted to arrive at an approach that addresses all the above three in context of OpenStack Nova with minimalist changes. Design Considerations: We tried a few experiments with the present libvirt driver in nova to accomplish a work flow to deploy containers inside virtual machines in OpenStack via Nova. The fundamental premise of our approach is to run a single container encapsulated in a single VM. This VM image just has a bare minimum operating system required to run it. The container filesystem comes from the docker image. We would like to get the feedback on the below approaches from the community before proposing this as a spec or blueprint. Approach 1 User workflow: 1. The docker image is obtained in the form of a tar file. 2. Upload this tar file in glance. This support is already there in glance were a container-type of docker is supported. 3. Use this image along with nova libvirt driver to deploy a virtual machine. Following are some of the changes to the OpenStack code that implements this approach: 1. Define a new conf parameter in nova called – base_vm_image=/var/lib/libvirt/images/baseimage.qcow2 This option is used to specify the base VM image. 2. define a new sub_virt_type = container in nova conf. Setting this parameter will ensure mounting of the container filesystem inside the VM. Unless qemu and kvm are used as virt_type – this workflow will not work at this moment. 3. In the virt/libvirt/driver.py we do the following based on the sub_virt_type = container: - We create a qcow2 disk from the base_vm_image and expose that 'disk' as the boot disk for the virtual machine. Note – this is very similar to a regular virtual machine boot minus the fact that the image is not downloaded from glance but instead it is present on the host. - We download the docker image into the /var/lib/nova/instances/_base directory and then for each new virtual machine boot – we create a new directory /var/lib/nova/instances/ as it's and copy the docker filesystem to it. Note – there are subsequent improvements to this idea that could be performed around the lines of using a union filesystem approach. - The step above allows each virtual machine to have a different copy of the filesystem. - We create a 'passthrough' mount of the filesystem via libvirt. This code is also present in the nova libvirt driver and we just trigger it based on our sub_virt_type parameter. 4. A cloud init – userdata is provided that looks somewhat like this: runcmd: - mount -t 9p -o trans=virtio share_dir /mnt - chroot /mnt /bin/ The command_to_run is usually the entrypoint to for the docker image. There could be better approaches to determine the entrypoint as well (say from docker image metadata). Approach 2. In this approach, the workflow remains the same as the first one with the exception that the docker image is changed into a qcow2 image using a tool like virt-make-fs before uploading it to glance, instead of a tar file. A tool like virt-make-fs can convert a tar file to a qcow2 image very eas
[openstack-dev] [nova][rfc] Booting docker images using nova libvirt
*Premise:** * While working with customers, we have realized: - They want to use containers but are wary of using the same host kernel for multiple containers. - They already have a significant investment (including skills) in OpenStack's Virtual Machine workflow and would like to re-use it as much as possible. - They are very interested in using docker images. There are some existing approaches like Hyper, Secure Containers workflows which already tries to address the first point. But we wanted to arrive at an approach that addresses all the above three in context of OpenStack Nova with minimalist changes. * **Design Considerations:* We tried a few experiments with the present libvirt driver in nova to accomplish a work flow to deploy containers inside virtual machines in OpenStack via Nova. The fundamental premise of our approach is to run a single container encapsulated in a single VM. This VM image just has a bare minimum operating system required to run it. The container filesystem comes from the docker image. We would like to get the feedback on the below approaches from the community before proposing this as a spec or blueprint. *Approach 1* User workflow: 1. The docker image is obtained in the form of a tar file. 2. Upload this tar file in glance. This support is already there in glance were a container-type of docker is supported. 3. Use this image along with nova libvirt driver to deploy a virtual machine. Following are some of the changes to the OpenStack code that implements this approach: 1. Define a new conf parameter in nova called – /base_vm_image/=/var/lib/libvirt/images/baseimage.qcow2 This option is used to specify the base VM image. 2. define a new /sub_virt_type/ = container in nova conf. Setting this parameter will ensure mounting of the container filesystem inside the VM. Unless qemu and kvm are used as virt_type – this workflow will not work at this moment. 3. In the virt/libvirt/driver.py we do the following based on the sub_virt_type = container: - We create a qcow2 disk from the /base_vm_image/ and expose that 'disk' as the boot disk for the virtual machine. Note – this is very similar to a regular virtual machine boot minus the fact that the image is not downloaded from glance but instead it is present on the host. - We download the docker image into the //var/lib/nova/instances/_base directory/ and then for each new virtual machine boot – we create a new directory //var/lib/nova/instances// as it's and copy the docker filesystem to it. Note – there are subsequent improvements to this idea that could be performed around the lines of using a union filesystem approach. - The step above allows each virtual machine to have a different copy of the filesystem. - We create a '/passthrough/' mount of the filesystem via libvirt. This code is also present in the nova libvirt driver and we just trigger it based on our sub_virt_type parameter. 4. A cloud init – userdata is provided that looks somewhat like this: / //runcmd:// // - mount -t 9p -o trans=virtio share_dir /mnt// // - chroot /mnt /bin// The /command_to_run /is usually the entrypoint to for the docker image. There could be better approaches to determine the entrypoint as well (say from docker image metadata). * **Approach 2.* In this approach, the workflow remains the same as the first one with the exception that the docker image is changed into a qcow2 image using a tool like virt-make-fs before uploading it to glance, instead of a tar file. A tool like virt-make-fs can convert a tar file to a qcow2 image very easily. This image is then downloaded on the compute node and a qcow2 disk is created/attached to the virtual machine that boots using the /base_vm_image/. *Approach 3* A custom qcow2 image is created using kernel, initramfs and the docker image and uploaded to glance. No changes are needed in openstack nova. It boots as a regular VM. Changes will be needed in image generation tools and will involve few additional tasks from an operator point of view. I look forward to your comments/suggestions on the above. Thanks, Sudipto __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
