[Gluster-users] No performance difference using libgfapi?
Good Morning, In my earlier experience invoking a VM using qemu/libgfapi, I reported that it was noticeably faster than the same VM invoked from libvirt using a FUSE mount; however, this was erroneous as the qemu/libgfapi-invoked image was created using 2x the RAM and cpu's... So, invoking the image using both methods using consistent specs of 2GB RAM and 2 cpu's, I attempted to check drive performance using the following commands: (For regular FUSE mount I have the gluster volume mounted at /var/lib/libvirt/images.) (For libgfapi I specify -disk file=gluster://gfs-00/gfsvol/tester1/img.) Using libvirt/FUSE mount: [root@tester1 ~]# hdparm -Tt /dev/vda1 /dev/vda1: Timing cached reads:11346 MB in 2.00 seconds = 5681.63 MB/sec Timing buffered disk reads:36 MB in 3.05 seconds = 11.80 MB/sec [root@tester1 ~]# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 41943040 bytes (42MB) copied, 0.0646241 s, 649 MB/sec Using qemu/libgfapi: [root@tester1 ~]# hdparm -Tt /dev/vda1 /dev/vda1: Timing cached reads:11998 MB in 2.00 seconds = 6008.57 MB/sec Timing buffered disk reads:36 MB in 3.03 seconds = 11.89 MB/sec [root@tester1 ~]# dd if=/dev/zero of=/tmp/output bs=8k count=10k; rm -f /tmp/output 10240+0 records in 10240+0 records out 41943040 bytes (42MB) copied, 0.0621412 s, 675 MB/sec Should I be seeing a bigger difference, or am I doing something wrong? I'm also curious whether people have found that the performance difference is greater as the size of the gluster cluster scales up. Thanks, David ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] glusterfs via libgfapi as local block device?
I see many posts regarding using libgfapi for qemu guest voluimes, but how about as a mounted volume at the OS level? Is there a way to mount a glusterfs share as a local block device in Linux, bypassing the FUSE? Instead of doing a mount -t glusterfs server:/volume, which appears as type fuse.glusterfs, can libgfapi be used to allow direct access and eliminate the fuse overhead? Do i need to set this up as iSCSI target in order for this to work? Thanks, Dave ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?
I've come across individual posts from people who supposedly have done this in CentOS6.5. Basically, all that is shown in the posts is the XML file generated, no mention of *how* that file is generated. Virt-manager has no provision for attaching directly to the gluster volume except as a mount. Neither virt-manager nor virt-install recognize the gluster:// type. Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm also doesn't recognize the gluster:// type. Debian and Ubuntu supposedly have the newer versions of gluster and qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy... Red Hat seems content to do their own thing. Although the verisons of libvirt and qemu are older, libgfapi is supposed to have been backported. It's a shame that full functionality is not included. It's mindboggling seeing that Red Hat owns glusterfs, you would think full support for the backend would have been included in their product. If it is, as you say, that RH includes this functionality only to RHN subscribers and is not made available downstream to CentOS/SL, and unless I can find a repository with the latest full versions of qemu libvirt, then CentOS simply will not work. On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana har...@harshavardhana.netwrote: Virt-manager / libvirt is yet to expose perhaps this functionality - but as far as i remember libvirt should be doing this as a pass-through for the URL's which have been passed as schema://server/volname Does libvirt 'invoke' fuse when passed gluster:// schema? On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson davidchristians...@gmail.com wrote: Good Evening, I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5 (and by virtue CentOS and SL). However I am unable to figure out how to actually make it work as described. Virt-manager still only seems to support glusterfs volumes via fuse. I can use qemu-img to create a disk image on gluster://server/Volume. But virt-manager can only use it from a fuse mounted fileshare. There seems to be no ability to attach in virt-manager directly to the image on glusterfs using libgfapi. All documents I've found describe the use of the command qemu-system-x86_64, however that command does not exist in CentOS 6.5. That appears to be the only way to start the domain using libgfapi. So basically, I can create an image via libgfapi but cannot do anything useful with it. Should I be able to do this? If so, what's the procedure? Or is CentOS/RHEL 6.5 just not fully integrated? I really want to be able to use libgfapi and avoid the performance penalty of fuse. Should I just grab compile the latest verisons of libvirt and qemu-kvm? ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Religious confuse piety with mere ritual, the virtuous confuse regulation with outcomes ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?
/usr/libexec/qemu-kvm Actually, that did not work for me either. I issued the command in the form of: /usr/libexec/qemu-kvm -drive file=*gluster*:// *host/vloumelimage_file*,if=virtio. I got errors regarding inability to open the image, and glusterfs not being a valid protocol. However, I could create the image via libgfapi using qemu-img, I could see the file on the filesystem, and the permissions are correct. Also, I do have option rpc-auth-allow-insecure on in /etc/glusterfs/glusterd.vol. Is there something else that's missing? Thanks, Dave On Fri, Mar 28, 2014 at 9:56 AM, SATHEESARAN sasun...@redhat.com wrote: On 03/28/2014 05:37 PM, Dave Christianson wrote: I've come across individual posts from people who supposedly have done this in CentOS6.5. Basically, all that is shown in the posts is the XML file generated, no mention of *how* that file is generated. Virt-manager has no provision for attaching directly to the gluster volume except as a mount. Neither virt-manager nor virt-install recognize the gluster:// type. Yes, even I was frustrated at this. There seems no way to utilize the image file using libgfapi. As Harsha mentioned in his earlier thread, libvirt is yet to expose this. I have already raised a bug regarding this, https://bugzilla.redhat.com/show_bug.cgi?id=1017308 Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm also doesn't recognize the gluster:// type. With RHEL 6.5 / Centos 6.5 you can use, /usr/libexec/qemu-kvm This was the command that I used to make use of VM Images using libgfapi in RHEL 6.5 /usr/libexec/qemu-kvm -drive file=gluster:// 10.70.37.87/testvol/test.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 The above worked for me. If I am not wrong, the libvirt in RHEL 7 supports glusterfs's libgfapi way of access mechanism. -- Satheesaran Debian and Ubuntu supposedly have the newer versions of gluster and qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy... Red Hat seems content to do their own thing. Although the verisons of libvirt and qemu are older, libgfapi is supposed to have been backported. It's a shame that full functionality is not included. It's mindboggling seeing that Red Hat owns glusterfs, you would think full support for the backend would have been included in their product. If it is, as you say, that RH includes this functionality only to RHN subscribers and is not made available downstream to CentOS/SL, and unless I can find a repository with the latest full versions of qemu libvirt, then CentOS simply will not work. On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana har...@harshavardhana.net wrote: Virt-manager / libvirt is yet to expose perhaps this functionality - but as far as i remember libvirt should be doing this as a pass-through for the URL's which have been passed as schema://server/volname Does libvirt 'invoke' fuse when passed gluster:// schema? On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson davidchristians...@gmail.com wrote: Good Evening, I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5 (and by virtue CentOS and SL). However I am unable to figure out how to actually make it work as described. Virt-manager still only seems to support glusterfs volumes via fuse. I can use qemu-img to create a disk image on gluster://server/Volume. But virt-manager can only use it from a fuse mounted fileshare. There seems to be no ability to attach in virt-manager directly to the image on glusterfs using libgfapi. All documents I've found describe the use of the command qemu-system-x86_64, however that command does not exist in CentOS 6.5. That appears to be the only way to start the domain using libgfapi. So basically, I can create an image via libgfapi but cannot do anything useful with it. Should I be able to do this? If so, what's the procedure? Or is CentOS/RHEL 6.5 just not fully integrated? I really want to be able to use libgfapi and avoid the performance penalty of fuse. Should I just grab compile the latest verisons of libvirt and qemu-kvm? ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Religious confuse piety with mere ritual, the virtuous confuse regulation with outcomes ___ Gluster-users mailing listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?
Fair enough, Kaleb. I shouldn't have let my frustration do the talking. It's not a show-stopper for me - yet. At this stage I have enough freedom with this project to keep tinkering and testing. Meanwhile I plan to follow the SIG team's progress closely. Thanks everyone for your valuable insight1 Dave On Fri, Mar 28, 2014 at 10:15 AM, Kaleb S. KEITHLEY kkeit...@redhat.comwrote: On 03/28/2014 08:07 AM, Dave Christianson wrote: Red Hat seems content to do their own thing. Although the versions of libvirt and qemu are older, libgfapi is supposed to have been backported. It's a shame that full functionality is not included. It's mindboggling seeing that Red Hat owns glusterfs, you would think full support for the backend would have been included in their product. If it is, as you say, that RH includes this functionality only to RHN subscribers and is not made available downstream to CentOS/SL, and unless I can find a repository with the latest full versions of qemu libvirt, then CentOS simply will not work. I'm not sure that's a fair expectation. (As a side note, GlusterFS is a community project -- it's owned by the community. Red Hat has a product that it owns -- RHS or RHSS -- that is based on GlusterFS.) Core 'vanilla' RHEL is pretty conservative. Just because ... Red Hat owns GlusterFS... (sic) doesn't mean we get to randomly update the libvirt that ships in RHEL. RHEL has its own QA cycle which gates when things like libvirt can be updated. As a result the newer libvirt has to be delivered in a separate channel. The fact that CentOS -- which is a clone of vanilla RHEL -- doesn't distribute some or all of the things that are in channels is not Red Hat's fault. If there are no RPMs of the newer libvirt available the fault lies with the community. The source for the newer libvirt is certainly available, as it always is -- it needs is someone in the community to package it for general consumption. Someone has done that for Ubuntu! And as John Mark has already indicated, that's something that the CentOS Storage SIG is intended to address. And finally, although I can't promise that it will be, it's entirely possible that the newer libvirt could be in RHEL 6.6, and then CentOS would have it too. -- Kaleb ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?
Hi all, I just installed a Fedora 20 test host, and after struggling with dependencies I'm happy to report that everything works perfectly. I'm not relying on the virt-manager gui at all, and from what I'm reading, currently libvirt cannot start an image from gluster via libgfapi anyway(?). For various reasons I don't wish to use Fedora in my production environment, so until the SIG group works their magic, if there is a way to compile the latest sources for qemu and libvirt on CentOS or Debian, I'll be in business. The performance difference between libgfapi and fuse is like night and day, very impressive. I never want to have to go back to running VM's over fuse again. Thanks again to everyone, David On Fri, Mar 28, 2014 at 10:47 AM, Lalatendu Mohanty lmoha...@redhat.comwrote: On 03/28/2014 07:26 PM, SATHEESARAN wrote: On 03/28/2014 05:37 PM, Dave Christianson wrote: I've come across individual posts from people who supposedly have done this in CentOS6.5. Basically, all that is shown in the posts is the XML file generated, no mention of *how* that file is generated. Virt-manager has no provision for attaching directly to the gluster volume except as a mount. Neither virt-manager nor virt-install recognize the gluster:// type. Yes, even I was frustrated at this. There seems no way to utilize the image file using libgfapi. As Harsha mentioned in his earlier thread, libvirt is yet to expose this. I have already raised a bug regarding this, https://bugzilla.redhat.com/show_bug.cgi?id=1017308 Supposedly earlier versions of RHEL used qemu-kvm as a wrapper for qemu-system-x86_64, however in 6.5 qemu-kvm is its own binary. Qemu-kvm also doesn't recognize the gluster:// type. With RHEL 6.5 / Centos 6.5 you can use, /usr/libexec/qemu-kvm This was the command that I used to make use of VM Images using libgfapi in RHEL 6.5 /usr/libexec/qemu-kvm -drive file=gluster:// 10.70.37.87/testvol/test.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 The above worked for me. If I am not wrong, the libvirt in RHEL 7 supports glusterfs's libgfapi way of access mechanism. Sas, What about Fedora 20? Fedora 20 should have all the latest RPMs , so it should have required patches in libvirt too. -Lala Debian and Ubuntu supposedly have the newer versions of gluster and qemu/libvirt availabe (ppa's?). Maybe I'll test Wheezy... Red Hat seems content to do their own thing. Although the verisons of libvirt and qemu are older, libgfapi is supposed to have been backported. It's a shame that full functionality is not included. It's mindboggling seeing that Red Hat owns glusterfs, you would think full support for the backend would have been included in their product. If it is, as you say, that RH includes this functionality only to RHN subscribers and is not made available downstream to CentOS/SL, and unless I can find a repository with the latest full versions of qemu libvirt, then CentOS simply will not work. On Fri, Mar 28, 2014 at 3:13 AM, Harshavardhana har...@harshavardhana.net wrote: Virt-manager / libvirt is yet to expose perhaps this functionality - but as far as i remember libvirt should be doing this as a pass-through for the URL's which have been passed as schema://server/volname Does libvirt 'invoke' fuse when passed gluster:// schema? On Thu, Mar 27, 2014 at 5:34 PM, Dave Christianson davidchristians...@gmail.com wrote: Good Evening, I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5 (and by virtue CentOS and SL). However I am unable to figure out how to actually make it work as described. Virt-manager still only seems to support glusterfs volumes via fuse. I can use qemu-img to create a disk image on gluster://server/Volume. But virt-manager can only use it from a fuse mounted fileshare. There seems to be no ability to attach in virt-manager directly to the image on glusterfs using libgfapi. All documents I've found describe the use of the command qemu-system-x86_64, however that command does not exist in CentOS 6.5. That appears to be the only way to start the domain using libgfapi. So basically, I can create an image via libgfapi but cannot do anything useful with it. Should I be able to do this? If so, what's the procedure? Or is CentOS/RHEL 6.5 just not fully integrated? I really want to be able to use libgfapi and avoid the performance penalty of fuse. Should I just grab compile the latest verisons of libvirt and qemu-kvm? ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Religious confuse piety with mere ritual, the virtuous confuse regulation with outcomes
[Gluster-users] Possible to use libgfapi with libvirt in CentOS 6.5?
Good Evening, I have read that libgfapi has been backported to qemu-kvm in RHEL 6.5 (and by virtue CentOS and SL). However I am unable to figure out how to actually make it work as described. Virt-manager still only seems to support glusterfs volumes via fuse. I can use qemu-img to create a disk image on gluster://server/Volume. But virt-manager can only use it from a fuse mounted fileshare. There seems to be no ability to attach in virt-manager directly to the image on glusterfs using libgfapi. All documents I've found describe the use of the command qemu-system-x86_64, however that command does not exist in CentOS 6.5. That appears to be the only way to start the domain using libgfapi. So basically, I can create an image via libgfapi but cannot do anything useful with it. Should I be able to do this? If so, what's the procedure? Or is CentOS/RHEL 6.5 just not fully integrated? I really want to be able to use libgfapi and avoid the performance penalty of fuse. Should I just grab compile the latest verisons of libvirt and qemu-kvm? ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
