Re: [Gluster-users] What should I do to improve performance ?
Just to clarify: the Cinder Gluster driver, which is comparable to the Cinder NFS driver, uses FUSE: https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/glusterfs.py#L489-L499 Unless I'm mistaken, bootable volumes with Nova will use libgfapi (if configured with qemu_allowed_storage_drivers), but the best ephemeral instances can do is store the instances on a GlusterFS mounted /var/lib/nova/instances. We've also seen performance issues when running ephemeral instances on a glusterfs mounted directory. However, we're using the Gluster Cinder driver for volumes and have been mostly happy with it. On Tue, Mar 24, 2015 at 9:22 AM, Joe Julian j...@julianfamily.org wrote: Nova ephemeral disks don't use libgfapi, only Cinder volumes. On March 24, 2015 7:27:52 AM PDT, marianna cattani marianna.catt...@gmail.com wrote: Openstack doesn't have vsdm, it should be a configuration option: qemu_allowed_storage_drivers=gluster But, however , the machine is generated with the xml that you see. Now I try to write to the OpenStack's mailing list . tnx M 2015-03-24 15:14 GMT+01:00 noc n...@nieuwland.nl: On 24-3-2015 14:39, marianna cattani wrote: Many many thanks Mine is different :'( root@nodo-4:~# virsh -r dumpxml instance-002c | grep disk disk type='file' device='disk' source file='/var/lib/nova/instances/ef84920d-2009-42a2-90de-29d9bd5e8512/disk'/ alias name='virtio-disk0'/ /disk Thats a fuse connection. I'm running ovirt + glusterfs where vdsm is a special version with libgfapi support. Don't know if Openstack has that too? Joop -- Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] libgfapi on ubuntu
Hi Bernhard (dropping -devel as I'm not subscribed) I've been able to compile qemu using Ubuntu 12.04 and the OpenStack Havana cloud archive (https://wiki.ubuntu.com/ServerTeam/CloudArchive) which comes with its own version of qemu. Maybe you can just pull qemu from that repo without having to use 12.04. If you do get qemu to compile, though, I'd be curious if you run into the issue described here: http://www.gluster.org/pipermail/gluster-users/2013-December/thread.html#38302 I've been trying for weeks to clear time out to debug more, but just haven't been able to. Thanks, Joe On Tue, Jan 14, 2014 at 1:59 AM, Bernhard Glomm bernhard.gl...@ecologic.euwrote: I'm running ubuntu 13.10 and try recompile qemu as described here https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517 in order to test libgfapi. unfortunately compiling failed with messages like this: In file included from /usr/include/libfdt.h:55:0, from /usr/src/rebuild-qemu/qemu-1.5.0+dfsg/device_tree.c:28: /usr/include/fdt.h:58:2: error: unknown type name âfdt32_tâ fdt32_t magic; /* magic word FDT_MAGIC */ ^ . this both files are causing the trouble: /usr/include/fdt.h and /usr/include/libfdt.h I tried the qemu sources from trusty (ubuntu 14.04) with them make failes with: ... dpkg-shlibdeps: error: no dependency information found for /usr/lib/libgfapi.so.0 (used by debian/qemu-system-misc/usr/bin/qemu-system-s390x) dh_shlibdeps: dpkg-shlibdeps -Tdebian/qemu-system-misc.substvars debian/qemu-system-misc/usr/bin/qemu-system-microblazeel debian/qemu-system-misc/usr/bin/qemu-system-or32 debian/qemu-system-misc/usr/bin/qemu-system-xtensaeb debian/qemu-system-misc/usr/bin/qemu-system-alpha debian/qemu-system-misc/usr/bin/qemu-system-sh4eb debian/qemu-system-misc/usr/bin/qemu-system-unicore32 debian/qemu-system-misc/usr/bin/qemu-system-m68k debian/qemu-system-misc/usr/bin/qemu-system-cris debian/qemu-system-misc/usr/bin/qemu-system-lm32 debian/qemu-system-misc/usr/bin/qemu-system-moxie debian/qemu-system-misc/usr/bin/qemu-system-microblaze debian/qemu-system-misc/usr/bin/qemu-system-s390x debian/qemu-system-misc/usr/bin/qemu-system-sh4 debian/qemu-system-misc/usr/bin/qemu-system-xtensa returned exit code 2 make: *** [install] Error 2 dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2 debuild: fatal error at line 1361: dpkg-buildpackage -rfakeroot -D -us -uc -j10 failed anyone an idea how to proceed? Since the above mentioned bug report is closed with won't fix until there is a MIR filed for glusterfs I may ask will that be done? Otherwise how can I use libgfapi on an ubuntu system? TIA Bernhard ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] qemu remote insecure connections
Hi Vijay, Thank you for the reply. It does look like a connection has been established but qemu-img is blocked on something. Can you please start qemu-img with strace -f and capture the output? Sure. I have added two new files to the gist: https://gist.github.com/jtopjian/7981763 One is strace using the root user, so the command is successful. The other is using the libvirt user. I eventually ctrl-c'd that strace. Let me know if I can collect any more info for you. Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] qemu remote insecure connections
Hello, I apologize for the delayed reply. I've collected some logs and posted them here: https://gist.github.com/jtopjian/7981763 I stopped the Gluster service on 192.168.1.11, moved /var/log/glusterfs to a backup, then started Gluster so that the log files were more succinct. I then used the qemu-img command as mentioned before as root, which was successful. Then I ran the command as libvirt-qemu and let the command hang for 2 minutes before I killed it. It might also be helpful to note that this is on an Ubuntu 12.04 server using the packages found here: https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.4 Thanks, Joe On Sat, Dec 14, 2013 at 12:41 AM, Vijay Bellur vbel...@redhat.com wrote: On 12/13/2013 10:58 AM, Joe Topjian wrote: Hello, I'm having a problem getting remote servers to connect to Gluster with qemu. I have 5 servers, 4 of which run Gluster and host a volume. The qemu user on all 5 servers has the same uid. storage.owner-uid and storage.owner-gid is set to that user. In addition, server.allow-insecure is on and is also set in the glusterd.vol file. glusterd has also been restarted (numerous times). When attempting to create a qemu file by connecting to the same server, everything works: qemu@192.168.1.11 mailto:qemu@192.168.1.11 qemu-img create gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img 1M Formatting 'gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img', fmt=raw size=1048576 qemu@192.168.1.11 mailto:qemu@192.168.1.11 But when trying to do it remotely, the command hangs indefinitely: qemu@192.168.1.12 mailto:qemu@192.168.1.12 qemu-img create gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img 1M Formatting 'gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img', fmt=raw size=1048576 ^C Yet when 192.168.1.12 connects to gluster://192.168.1.12 http://192.168.1.12, the command works and the file shows up in the distributed volume. Further, when turning server.allow-insecure off, I get an immediate error no matter what the source and destination connection is: qemu@192.168.1.12 mailto:qemu@192.168.1.12 qemu-img create gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img 1M Formatting 'gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img', fmt=raw size=1048576 qemu-img: Gluster connection failed for server=192.168.1.11 port=0 volume=volumes image=v.img transport=tcp qemu-img: gluster://192.168.1.11/volumes/v.img http://192.168.1.11/volumes/v.img: error while creating raw: No data available Does anyone have any ideas how I can have an unprivileged user connect to remote gluster servers? Can you please provide glusterd and glusterfsd logs from 192.168.1.11? -Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] qemu remote insecure connections
Hello, I'm having a problem getting remote servers to connect to Gluster with qemu. I have 5 servers, 4 of which run Gluster and host a volume. The qemu user on all 5 servers has the same uid. storage.owner-uid and storage.owner-gid is set to that user. In addition, server.allow-insecure is on and is also set in the glusterd.vol file. glusterd has also been restarted (numerous times). When attempting to create a qemu file by connecting to the same server, everything works: qemu@192.168.1.11 qemu-img create gluster://192.168.1.11/volumes/v.img 1M Formatting 'gluster://192.168.1.11/volumes/v.img', fmt=raw size=1048576 qemu@192.168.1.11 But when trying to do it remotely, the command hangs indefinitely: qemu@192.168.1.12 qemu-img create gluster://192.168.1.11/volumes/v.img 1M Formatting 'gluster://192.168.1.11/volumes/v.img', fmt=raw size=1048576 ^C Yet when 192.168.1.12 connects to gluster://192.168.1.12, the command works and the file shows up in the distributed volume. Further, when turning server.allow-insecure off, I get an immediate error no matter what the source and destination connection is: qemu@192.168.1.12 qemu-img create gluster://192.168.1.11/volumes/v.img 1M Formatting 'gluster://192.168.1.11/volumes/v.img', fmt=raw size=1048576 qemu-img: Gluster connection failed for server=192.168.1.11 port=0 volume=volumes image=v.img transport=tcp qemu-img: gluster://192.168.1.11/volumes/v.img: error while creating raw: No data available Does anyone have any ideas how I can have an unprivileged user connect to remote gluster servers? Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster / KVM Filesystem Benchmarks
Hello, I did a few filesystem benchmarks with Gluster (3.3) and KVM using iozone and have compiled a spreadsheet with the results: https://docs.google.com/open?id=0B6GzZufzohwFZmozTFRSSHk5T0E Just a heads up: It is an Excel spreadsheet. All of the details that were used to generate the results are described in the spreadsheet. Of most interest would be the second tab titled Gluster. The results that do not have vm in the description were iozone procs running directly on a mounted replicated Gluster volume (2 bricks). The vm results are iozone procs running in KVM virtual machines stored in qcow2 files. The first tab, General, is just some simple non-Gluster benchmarks that I ran for comparison. The third tab, Gluster old, was me doing iozone benchmarks on files with sizes ranging from 8mb to 512mb. I noticed that there was very little difference in the results so I decided to work with only 128mb and 256mb sized files. If you do not have access to Excel or something compatible, you can still view most of the information in the Google Doc. Here is a jpeg image of the main graph that was generated: https://docs.google.com/open?id=0B6GzZufzohwFWGtFS3I5UEllTkU Questions I have: * The optimized settings that I used were pulled from a Gluster Performance Tuning presentation. It doesn't look like the settings did very much in terms of optimization. Can someone comment on these settings? Are there better settings to use? * I'm a bit confused at how the KVM / qcow2 reads are much higher than the reads directly on the Gluster volume. Any idea why that is? * I ran all tests with the cache-io translator on and off. Like the optimized settings, it wasn't of much use. Did I use this incorrectly? * The reason I did all tests with 128mb and 256mb sized files was to highlight the very bizarre trait where certain increments (16, 64, 256) gave very poor results while increments such as 8, 32, and 128 had good results. Any idea why that is? * Can anyone comment on if these results are of any use? Or are the stats I collect and the way I collected them incorrect? Please let me know if anyone has any questions or needs anything clarified. Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] FeedBack Requested : Changes to CLI output of 'peer status'
Hi Pranith, On Tue, Aug 28, 2012 at 6:51 AM, Pranith Kumar Karampuri pkara...@redhat.com wrote: Why not use --xml at the end of the command? That will print the output in xml format. Would that make it easy to parse?. IMO, I can see --xml being useful for more sophisticated scripts that utilize Perl, Python, or Ruby, however, for quick shell scripting, I can see this being very difficult if not unusable. Having the output print in set columns, though might wrap when printed on =80 char terminals, would make it very easy to parse with awk. Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] FeedBack Requested : Changes to CLI output of 'peer status'
Hi James, FWIW: I tend to agree on this point. The other question that comes to mind is what quick bash scripts are you writing for managing gluster? quick might not have been the best word to use -- maybe small would have been better. For example, peer status + columns + grep + awk could be used for a small Nagios check command. Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Ownership changed to root
Hi Brian, This sounds extremely unlikely. mdadm and LVM both work at the block device layer - reading and writing 512-byte blocks. They have no understanding of filesystems and no understanding of user IDs. Agreed. I suspect there were other differences between the tests. For example, did you do one with an ext4 filesystem and one with xfs? Or did you have a failed drive in your RAID1, which meant that some writes were timing out? Both environments were xfs. The only difference is that the original environment was not using LVM or mdadm -- similar to what I ended up doing on the second environment in the end. I do plan on testing this out in more detail soon as I'd like to narrow the issue down further. FWIW, I've also seen the files owned by root occasionally in testing, but wasn't able to pin down the cause. Good to know - if anything to say I'm not going crazy :) Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Ownership changed to root
Hello, I'm seeing a weird issue with OpenStack and Gluster. I have /var/lib/nova/instances mounted as a glusterfs volume. The owner of /var/lib/nova/instances is nova:nova. When I launch a vm and watch it launching, I see the following: root@c01:/var/lib/nova/instances/instance-0012# ls -l total 8 -rw-rw 1 nova nova0 Aug 24 14:22 console.log -rw-rw-r-- 1 nova nova 1459 Aug 24 14:22 libvirt.xml This is correct. Then it changes ownership to libvirt-qemu: root@c01:/var/lib/nova/instances/instance-0012# ls -l total 22556 -rw-rw 1 libvirt-qemu kvm 0 Aug 24 14:22 console.log -rw-r--r-- 1 libvirt-qemu kvm 27262976 Aug 24 14:22 disk -rw-rw-r-- 1 nova nova 1459 Aug 24 14:22 libvirt.xm Again, this is correct. But then it changes to root: root@c01:/var/lib/nova/instances/instance-0012# ls -l total 22556 -rw-rw 1 root root0 Aug 24 14:22 console.log -rw-r--r-- 1 root root 27262976 Aug 24 14:22 disk -rw-rw-r-- 1 nova nova 1459 Aug 24 14:22 libvirt.xml OpenStack then errors out due to not being able to correctly access the files. If I remove the /var/lib/nova/instances mount and just use the normal filesystem, the root ownership part does not happen. I have successfully had Gluster working with OpenStack in this way on a different installation, so I'm not sure why I'm seeing this issue now. Any ideas? Thanks, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
