Re: [Gluster-users] glusterfs + cloudstack setup
Bryan, Thanks for your answers. Indeed, secondary storage is not used much if you do not have regular snapshotting taking place, as Nux suggested. In my case it will be the case so I was looking at using gluster for that. However, secondary storage vm seems to freeze the mountpoint if it tries to write something to it. It is very odd as I can see that ssvm created a bunch of folder and files, but all of them have 0 size. Also, I am unable to access that mountpoint for long time periods as any commands like du -h mount point or ls -lR mountpoint freeze for tens of minutes. Regarding your gluster + kvm setup, I would be very interested to learn how you've managed to achieve speeds of up to 150mb/s. Am i right in uderstanding that these are the vm disk speeds and not the glusterfs mountpoint speeds on the host side? When I was doing my testing I've faced a very interesting problem. The host mountpoint speeds (using dd and /dev/zero) were pretty impressive. I know that it's not the real data, but still. I was getting around 700-800MB/s from a single client. However, running the same command from the vm was giving me a maximum of 50MB/s. Do you mind sharing how you've got speeds of 150mb/s? Many thanks - Original Message - From: Bryan Whitehead dri...@megahappy.net To: Andrei Mikhailovsky and...@arhont.com Cc: gluster-users gluster-users@gluster.org Sent: Thursday, 16 May, 2013 10:07:59 PM Subject: Re: glusterfs + cloudstack setup I've been slowly working on a full-writeup/case-study. I've just been drowning in other work. Anyhow, for the secondary storage I've just used regular NFS export. Once a template gets deployed (copied to a primary storage system on a shared-mount) it doesn't need to be copied again. Secondary storage is hardly used except for the occasional snapshot and new template. On Thu, May 16, 2013 at 1:28 PM, Andrei Mikhailovsky and...@arhont.com wrote: Hi Bryan, I've read about your adventures with infiniband and cloudstack on http://www.marshut.com/kqss/cloudstack-glusterfs-over-infiniband.html. I was wondering if you could share how you've setup your secondary storage in cloudstack? Have you done this through nfs over glusterfs? If so, would you mind sharing a bit more information? I am setting up a similar infrastructure and is currently having issues with the secondary storage vm and nfs over glusterfs mountpoint. Many thanks Andrei ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs + cloudstack setup
I think you are referring to this statement: After adding a gluster layer (fuse mount) write speeds per process are at ~150MB/sec. So the raw fs write speed for 1 or more processes against a mountpoint on xfs I get ~300MB/sec. After the fuse layer is added (with replica=2 as shown in my config) I'm getting 150MB/sec for 1 process (still not inside a VM). If I add another process I get roughly another 150MB/sec on that process. Adding a third process - each process is only pushing 100MB/sec (so 3 processes writing as fast as they can are collectively getting 300MB/sec). In practice (outside the lab) I have more than 2 machines so it hard to go back and figure out exactly what the total bandwidth or write speed ends up being because replica=2 with 6 bricks and where a hash ends up hitting all make benchmarking difficult. Inside a VM I typically can push 50-70MB/sec when the qcow2 file is living on glusterfs mount. If I create a VM that lives on a local mountpoint (non-gluster non-shared filesystem) I get 70-90MB/sec (this is using the same hardware/raid as my gluster bricks). So using a mountpoint where I can easily push 300MB/sec with dd sucks once qcow2 adds its layer. Interestingly, if I run dd=/dev/zero of=/tmp/blah and let the disk fill up inside the VM, then delete the temp file the speed on a gluster mountpoint vs local proportionally shifts for the better. Basically once a qcow2 file is fully-grown (not expanding) I get 70-110MB/sec on a gluster mount and 90-130MB/sec on locally mounted qcow2 filesystem. I'm assuming the write speeds remain lame as the qcow2 file probably grew in some really retarded way as ext4 allocated non-contiguous blocks inside the VM while the qcow2 file grew in a contiguous manner. This means the qcow2 file is basically a scrambled mess of pointers to small out of order blocks. I can do the same 'per process' idea where I spin up 2, 3, 4, 5, 6, ... N VM's running on the glusterfs mountpoint (as apposed to simple dd processes) and have them all go crazy writing. I'll end up maxing out the underlying block devices. so roughly after 6 VM's are dd-ing like crazy (inside the vm) I've maxed the IO out. 50MB/sec * 6 VM's = 300MB/sec (well 600MB/sec because replica=2). 2 machines in the test cluster and they each have a brick that can do 300MB/sec. with replica=2, 6 VM's going full speed have used 100% of my IO resources available. So to be honest - I think that means everything is a-ok. It would be nice if 1 VM could push a full 300MB/sec. But that isn't even remotely happening even when using a native mount with qcow2. The nice part of many processes seeking for data non-contiguously in a qcow2 file is it is happening N times so overall I still get pretty decent speeds. Read performance seems to be pretty good (I say this non-empirically since my disk related issues have been write speed related - read benchmark blurb below). The growing/resizing of a qcow2 files suck. They suck more when on top of glusterfs (with replica=2 anyway). Growing a file across 2 filesystems on 2 different bricks on 2 different boxes is just going to cost some kind of time somewhere. Either brick a or brick b is going finish before the other and one will wait for the other. Knowing that worst case I'm going to get 50-70MB/sec inside a VM I focus on making sure scaling out horizontally is *really* working. My biggest regret on all these benchmarks I did: I pretty much skipped out on doing a full round of read benchmarks. Also, I just don't have the same hardware in my lab that matches anymore to avoid having to redo all my benchmarks (can't just do some read benchmarks since the hardware used is now in production). :( Once native gluster support in qemu hits the mainstream the qcow2 will go away and things should be ... better. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] glusterfs + cloudstack setup
Hi Bryan, I've read about your adventures with infiniband and cloudstack on http://www.marshut.com/kqss/cloudstack-glusterfs-over-infiniband.html. I was wondering if you could share how you've setup your secondary storage in cloudstack? Have you done this through nfs over glusterfs? If so, would you mind sharing a bit more information? I am setting up a similar infrastructure and is currently having issues with the secondary storage vm and nfs over glusterfs mountpoint. Many thanks Andrei ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs + cloudstack setup
On 16.05.2013 22:07, Bryan Whitehead wrote: Anyhow, for the secondary storage I've just used regular NFS export. Once a template gets deployed (copied to a primary storage system on a shared-mount) it doesn't need to be copied again. Secondary storage is hardly used except for the occasional snapshot and new template. Hi, Just throwing in my 2 pence. Depending on the nature of the deployment, the occasional snapshot can become the daily snapshot or even more frequent, depends how many customers you have and what need they have of the snapshots (i.e. for regular backups). In these cases the NFS backed secondary storage can quickly become a bottleneck. I'd either push for making the secondary storage VM glusterfs-aware (or at least NFS v3 via TCP) or enable Swift for the zone (and use the glusterfs-swift proxy - UFO). I'm not sure what Swift's role is in Cloudstack, but if it's what I think it is then it should scale much better. Lucian -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
