Re: [one-users] Storage subsystem: which one?
Hello Humberto Sorry for the delay. On 18.10.2011 10:35, Humberto N. Castejon Martinez wrote: Thank you very much, Fabian and Carlos, for your help. Things are much more clear now, I think. You're welcome. *Sharing the image repository. If I understood right, the aim with sharing the image repository between the front-end and the workers is to increase performance by reducing (or eliminating) the time needed to transfer an image from the repository to the worker that will run an instance of such image. I have, however, a With the shared images folder you are able to distribute the transfer over time, as the VM does only read (eg. transfer over NFS and those over the network) the parts of the image on an as needed basis. But from the performance point of view, NFS is most often slower then access to the image on the local disk. This may be different with other storage solutions, eg. with a distributed FS over all the cluster nodes, or eg. an other backend storage solution with iSCSI and 10 GBit/s Ethernet to the cluster nodes. This stuff most often depends on the complete setup and network infrastructure you have available. The best would be, if you can do performance testing by yourself on your own site and infrastructure to find the best solution depending on your expectation and needs of the VM cluster. question/remark here. To really reduce or eliminate the transfer time, the image should already reside on the worker node or close to it. If the image resides on a central server (case of NFS, if I am not wrong) or on an external shared distributed storage space (case of MooseFS, GlusterFS, Lustre, and the like), there is still a need to transfer the image to the worker, right? In the case of a distributed storage solution like MooseFs, etc., the worker could itself be part of the distributed storage space. In that case, the image may already reside on the worker, although not necessarily, right? But using the worker as both a storage server and client may actually compromise performance, for what I have read. With a distributed file system, it depends how this is working with such stuff. An example (I do not have any experience with it, but this is how I would expect the work of such a distributed file system to be done): In the example we would have cluster node 1 to 10, all set up with eg. MooseFS. We also would have a permanent image, which is located on the MooseFS storage (which for redundancy is physically distributed over several cluster nodes, probably also in parts). For the example, we assume that the image is physically on node 3, 5 and 7. Now when you start a VM which will use this image, the VM will be started on node 1, in the beginning, it will read the image through MooseFS over the network from one or more of the nodes 3, 5 or 7 and it can be used immediately. Now I expect from MooseFS, that it will modify the distributed file system in such a way, that over time the image physically will be stored on node 1 and to do this in the background. After a while the whole image should be available from the local disk of node 1, and those have the same performance as from a normal local disk. If somebody has experience with such stuff, please tell if my idea is right or wrong. Am I totally wrong with my thoughts here? If not, do we really increase transfer performance by sharing the image repository using, e.g. NFS? Are there any performance numbers for the different cases that could be shared? * Sharing theVM_dir. Sharing theVM_dir between the front-end and the workers is not really needed, but it is more of a convenient solution, right? Sharing theVM_dir between the workers themselves might be needed for live migration. I say might because i have just seen that, for example, with KVM we may perform live migrations without a shared storage [2]. Has anyone experimented with this? I'm not sure, but I guess OpenNebula depends on a shared file system for live migration, independent of the used Hypervisor. Probably you could do live migration with KVM and local storage when you are using KVM without OpenNebula. Regarding the documentation, Carlos, it looks fine. I would only suggest the possibility of documenting the 3rd case where the image repository is not shared but theVM_dir is shared. I am not sure, but I think OpenNebula is currently not able to handle this two differently, as it is defined per cluster node with the 'onehost create ...' command where you define to use tm_nfs or tm_ssh. bye Fabian ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Storage subsystem: which one?
Hi, Thank you very much, Fabian and Carlos, for your help. Things are much more clear now, I think. *Sharing the image repository. If I understood right, the aim with sharing the image repository between the front-end and the workers is to increase performance by reducing (or eliminating) the time needed to transfer an image from the repository to the worker that will run an instance of such image. I have, however, a question/remark here. To really reduce or eliminate the transfer time, the image should already reside on the worker node or close to it. If the image resides on a central server (case of NFS, if I am not wrong) or on an external shared distributed storage space (case of MooseFS, GlusterFS, Lustre, and the like), there is still a need to transfer the image to the worker, right? In the case of a distributed storage solution like MooseFs, etc., the worker could itself be part of the distributed storage space. In that case, the image may already reside on the worker, although not necessarily, right? But using the worker as both a storage server and client may actually compromise performance, for what I have read. Am I totally wrong with my thoughts here? If not, do we really increase transfer performance by sharing the image repository using, e.g. NFS? Are there any performance numbers for the different cases that could be shared? * Sharing the VM_dir. Sharing the VM_dir between the front-end and the workers is not really needed, but it is more of a convenient solution, right? Sharing the VM_dir between the workers themselves might be needed for live migration. I say might because i have just seen that, for example, with KVM we may perform live migrations without a shared storage [2]. Has anyone experimented with this? Regarding the documentation, Carlos, it looks fine. I would only suggest the possibility of documenting the 3rd case where the image repository is not shared but the VM_dir is shared. Thanks! Cheers, Humberto [1] http://opennebula.org/documentation:rel3.0:sfs#considerations_limitations [2] http://www.mail-archive.com/libvir-list@redhat.com/msg23674.html Date: Mon, 17 Oct 2011 16:30:13 +0200 From: Fabian Wenk fab...@wenks.ch To: users@lists.opennebula.org Subject: Re: [one-users] Storage subsystem: which one? Message-ID: 4e9c3bf5.1060...@wenks.ch Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hello Carlos On 17.10.2011 11:34, Carlos Mart?n S?nchez wrote: Thank you for your great contributions to the list! You're welcome. I'd like to add that we tried to summarize the implications of the shared [1] and non-shared [2] approaches in the documentation, let us know if there are any big gaps we forgot about. Thank you for documenting it on the website. I think it is complete and mentions all important facts about the two storage possibilities. bye Fabian ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Storage subsystem: which one?
Hi, On Thu, Oct 13, 2011 at 5:54 PM, Fabian Wenk fab...@wenks.ch wrote: I hope this helps and my information are correct, if not, could somebody from OpenNebula please correct me. Thank you for your great contributions to the list! I'd like to add that we tried to summarize the implications of the shared [1] and non-shared [2] approaches in the documentation, let us know if there are any big gaps we forgot about. Regards. [1] http://opennebula.org/documentation:rel3.0:sfs#considerations_limitations [2] http://opennebula.org/documentation:rel3.0:nsfs#considerations_limitations -- Carlos Martín, MSc Project Engineer OpenNebula - The Open Source Toolkit for Cloud Computing www.OpenNebula.org http://www.opennebula.org/ | cmar...@opennebula.org On Thu, Oct 13, 2011 at 5:54 PM, Fabian Wenk fab...@wenks.ch wrote: Hello Humberto On 13.10.2011 11:03, Humberto N. Castejon Martinez wrote: Reading the Opennebula documentation, I believe there are two things I have to deal with: 1) The image repository, and whether it is shared or not between the front-end and the workers I have some persistent Images which are used for persistent VMs. I have the images folder shared with NFS and I use tm_nfs. When starting a VM with a persistent images, it creates only a soft link in VM_DIR which points to the image in the images folder. If you want to have copied the persistent image into the VM_DIR on the cluster node, you would need tm_ssh. But then on startup the whole image will be copied to the cluster node into VM_DIR and on shutdown it will be copied back to the images folder. 2) TheVM_DIR, that contains deployment files, etc for the VMs running on a worker. This directory may or not be shared between the front-end and the workers, but it should always be shared between the workers if we want live migration, right? If you use tm_ssh you do not need to share this, if you use tm_nfs, you need to have it shared. The same with the images folder. For live migration you need a shared images folder and VM_DIR. Some of the questions I have are these (sorry if some of them seem stupid :-)): They are not stupid, It took me some time to try out and see through how this things works, and so I had to change my setup a few times until OpenNebula an I were happy. - What are the implications of sharing or not the image repository between the front-end and the workers (apart from the need to transfer images to the worker nodes in the latter case)? See above. - What are the implications of sharing or not theVM_DIR between the front-end and the workers? Also above. - Can I use ZFS or MooseFs and still be able to live migrate VMs? MooseFS [1] is a fault tolerant, network distributed file system (eg. over several servers). I do not know if ZFS can do this. MooseFS is similar like NFS, only that the data are distributed over several servers (including your local server). I guess live migration should work. [1] http://www.moosefs.org/ - Will theVM_DIR always hold a (copy of a) VM image for every VM running on a worker? Must theVM-DIR be in the local hard drive of a worker or may it reside on an external shared storage? See above, if used with tm_nfs, it is on a external shared storage (NFS server). When used with tm_ssh, it is on the local disk and all images (public or persistent) will be copied in full through ssh. I guess two factors i should also consider when choosing a solution are the following, right? - The speed of transferring VM images - The speed of cloning VM images Yes. Access to a persistent image through NFS only transfers the data needed, cloning always creates a full copy. When you use tm_ssh, the image will always be copied in full through ssh (which also gives some CPU overhead on the front end and cluster node). I hope this helps and my information are correct, if not, could somebody from OpenNebula please correct me. bye Fabian __**_ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**orghttp://lists.opennebula.org/listinfo.cgi/users-opennebula.org ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Storage subsystem: which one?
Hello Carlos On 17.10.2011 11:34, Carlos Martín Sánchez wrote: Thank you for your great contributions to the list! You're welcome. I'd like to add that we tried to summarize the implications of the shared [1] and non-shared [2] approaches in the documentation, let us know if there are any big gaps we forgot about. Thank you for documenting it on the website. I think it is complete and mentions all important facts about the two storage possibilities. bye Fabian ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] Storage subsystem: which one?
Hello Humberto On 13.10.2011 11:03, Humberto N. Castejon Martinez wrote: Reading the Opennebula documentation, I believe there are two things I have to deal with: 1) The image repository, and whether it is shared or not between the front-end and the workers I have some persistent Images which are used for persistent VMs. I have the images folder shared with NFS and I use tm_nfs. When starting a VM with a persistent images, it creates only a soft link in VM_DIR which points to the image in the images folder. If you want to have copied the persistent image into the VM_DIR on the cluster node, you would need tm_ssh. But then on startup the whole image will be copied to the cluster node into VM_DIR and on shutdown it will be copied back to the images folder. 2) TheVM_DIR, that contains deployment files, etc for the VMs running on a worker. This directory may or not be shared between the front-end and the workers, but it should always be shared between the workers if we want live migration, right? If you use tm_ssh you do not need to share this, if you use tm_nfs, you need to have it shared. The same with the images folder. For live migration you need a shared images folder and VM_DIR. Some of the questions I have are these (sorry if some of them seem stupid :-)): They are not stupid, It took me some time to try out and see through how this things works, and so I had to change my setup a few times until OpenNebula an I were happy. - What are the implications of sharing or not the image repository between the front-end and the workers (apart from the need to transfer images to the worker nodes in the latter case)? See above. - What are the implications of sharing or not theVM_DIR between the front-end and the workers? Also above. - Can I use ZFS or MooseFs and still be able to live migrate VMs? MooseFS [1] is a fault tolerant, network distributed file system (eg. over several servers). I do not know if ZFS can do this. MooseFS is similar like NFS, only that the data are distributed over several servers (including your local server). I guess live migration should work. [1] http://www.moosefs.org/ - Will theVM_DIR always hold a (copy of a) VM image for every VM running on a worker? Must theVM-DIR be in the local hard drive of a worker or may it reside on an external shared storage? See above, if used with tm_nfs, it is on a external shared storage (NFS server). When used with tm_ssh, it is on the local disk and all images (public or persistent) will be copied in full through ssh. I guess two factors i should also consider when choosing a solution are the following, right? - The speed of transferring VM images - The speed of cloning VM images Yes. Access to a persistent image through NFS only transfers the data needed, cloning always creates a full copy. When you use tm_ssh, the image will always be copied in full through ssh (which also gives some CPU overhead on the front end and cluster node). I hope this helps and my information are correct, if not, could somebody from OpenNebula please correct me. bye Fabian ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
