Re: [Gluster-users] gluster client performance
[..] I don't know why my writes are so slow compared to reads. Let me know if you're able to get better write speeds with the newer version of gluster and any of the configurations (if they apply) that I've posted. It might compel me to upgrade. From your documentation of nfsspeedtest, I see that the reads can happen either via dd or via perl's sysread. I'm not sure if one is better over the other. Secondly - Are you doing direct IO on the backend XFS ? If not, try it with direct IO so that you are not misled by the memory situation in the system at the time of your test. It will give a clearer picture of what your backend is capable of. Your test is such that you write a file and immediately read the same file back. It is possible that a good chunk of it is cached on the backend. After the write, do a flush of the filesystem caches by using: echo 3 /proc/sys/vm/drop_caches. Sleep for a while. Then do the read. Or as suggested earlier, resort to direct IO while testing the backend FS. Pavan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] gluster's cpuload is too high on specific birck daemon
Hello, I'm gluster with distributed-replicated mode(4 brick server) And 10client server mount gluster volume brick1 server. ( mount -t glustefs brick1:/volume /mnt) And there's very strange thing. The brick1's cpu load is too high. From 'top' command, it's over 400% But other brick's load is too low. Is there any reason for this? Or Is there anyway tracking down this issue? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster's cpuload is too high on specific birck daemon
On Wednesday 27 July 2011 01:45 PM, 공용준(yongjoon kong)/Cloud Computing 기술담당/SKCC wrote: Hello, I'm gluster with distributed-replicated mode(4 brick server) And 10client server mount gluster volume brick1 server. ( mount -t glustefs brick1:/volume /mnt) And there's very strange thing. The brick1's cpu load is too high. From 'top' command, it's over 400% But other brick's load is too low. It is possible that an AFR self heal is getting triggered. On the brick, run the following command: strace -f -c -p glusterfs.pid and provide the output. Pavan Is there any reason for this? Or Is there anyway tracking down this issue? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] anyone else playing with gcollect yet?
I'm messing with it and had to do a few patches to get rid of warnings/errors on my system (it threw lots of warnings cause of my configured options on volumes and there was a traceback do to a typo), but now it just returns empty with a return code of 0. -greg ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] anyone else playing with gcollect yet?
gluster-users-boun...@gluster.org wrote on 07/27/2011 07:40:13 AM: I'm messing with it and had to do a few patches to get rid of warnings/errors on my system (it threw lots of warnings cause of my configured options on volumes and there was a traceback do to a typo), but now it just returns empty with a return code of 0. so for the sake of discussion... reasons it is returning blank: 1: iterator only goes through number of bricks-1 when evaluating the bricks, manytimes missing the local brick cause I happen to be testing on the last node in the list. patched here: https://github.com/gregswift/Gluster/commit/a16567b5149aea2ddbec1e61d6b9a8e8e3b10e76 2: hostname check doesn't work on my system because I don't use hostnames ;) yes yes.. i usually love dns... please don't fight me on this one. working on patch -greg ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] anyone else playing with gcollect yet?
This looks interesting, but is there any tie in with collectd (http://collectd.org/)? I'm currently using ganglia, but only on the cluster, want to run collectd and have it consolidate data to the central monitoring server. Wondering if gcollect could do this too...could call it gcollectd then :) Thanks P On Wed, Jul 27, 2011 at 9:40 AM, greg_sw...@aotx.uscourts.gov wrote: gluster-users-boun...@gluster.org wrote on 07/27/2011 07:40:13 AM: I'm messing with it and had to do a few patches to get rid of warnings/errors on my system (it threw lots of warnings cause of my configured options on volumes and there was a traceback do to a typo), but now it just returns empty with a return code of 0. so for the sake of discussion... reasons it is returning blank: 1: iterator only goes through number of bricks-1 when evaluating the bricks, manytimes missing the local brick cause I happen to be testing on the last node in the list. patched here: https://github.com/gregswift/Gluster/commit/a16567b5149aea2ddbec1e61d6b9a8e8e3b10e76 2: hostname check doesn't work on my system because I don't use hostnames ;) yes yes.. i usually love dns... please don't fight me on this one. working on patch -greg ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- http://philcryer.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] anyone else playing with gcollect yet?
as far as I'm aware its standalone and from scratch. the initial goal is that it is setup to populate the data directly into ganglia, but being flexible enough to support other monitoring solutions (i'm working on making it work with zabbix). -greg phil cryer p...@cryer.us wrote on 07/27/2011 10:39:28 AM: This looks interesting, but is there any tie in with collectd (http://collectd.org/)? I'm currently using ganglia, but only on the cluster, want to run collectd and have it consolidate data to the central monitoring server. Wondering if gcollect could do this too...could call it gcollectd then :) Thanks P On Wed, Jul 27, 2011 at 9:40 AM, greg_sw...@aotx.uscourts.gov wrote: gluster-users-boun...@gluster.org wrote on 07/27/2011 07:40:13 AM: I'm messing with it and had to do a few patches to get rid of warnings/errors on my system (it threw lots of warnings cause of my configured options on volumes and there was a traceback do to a typo), but now it just returns empty with a return code of 0. so for the sake of discussion... reasons it is returning blank: 1: iterator only goes through number of bricks-1 when evaluating the bricks, manytimes missing the local brick cause I happen to be testing on the last node in the list. patched here: https://github.com/gregswift/Gluster/commit/ a16567b5149aea2ddbec1e61d6b9a8e8e3b10e76 2: hostname check doesn't work on my system because I don't use hostnames ;) yes yes.. i usually love dns... please don't fight me on this one. working on patch -greg ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- http://philcryer.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Scaling Gluster
Use ucarp to share a virtual IP address between the Gluster servers, and mount using that IP address. Joshua Baker-LePain - Yes, step by step is here: http://www.misdivision.com/blog/setting-up-a-highly-available-storage-cluster- using-glusterfs-and-ucarp ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster client performance
But that still does not explain why you should get as low as 50 MB/s for a single stream single client write when the backend can support direct IO throughput of more than 700 MB/s. On the server, can you collect: # iostat -xcdh 2 iostat.log.brickXX for the duration of the dd command ? and # strace -f -o stracelog.server -tt -T -e trace=write,writev -p glusterfsd.pid (again for the duration of the dd command) Hi John, A small change in the request. I hope you have not already spent time on this. The strace command should be: strace -f -o stracelog.server -tt -T -e trace=pwrite -p glusterfsd.pid Thanks, Pavan With the above, I want to measure the delay between the writes coming in from the client. iostat will describe the IO scenario on the server. Once the exercise is done, please attach the iostat.log.brickXX and stracelog.server. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] 3.2.2
Who has upgraded? how's the water? I'd like to know before I jump in the pool naked... Has the GFID issue been fixed fixed? Has the invalid argument gone and stayed away? (my biggest pain right now..) Has the /cache coherency/ *feature* been improved? -luis Luis E. Cerezo http://www.luiscerezo.org http://twitter.com/luiscerezo http://flickr.com/photos/luiscerezo photos for sale: http://photos.luiscerezo.org Voice: 412 223 7396 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] performance expectations?
Hello, I'm keen to use glusterfs as scalable storage to match the rest of my horizontally-scalable, commodity platform. I've been testing on a machine with 4x SATA 7200 drives. I created four KVM virtual machines, gave each of them a dedicated disk, and installed gluster-3.2.2. I created a distributed replicated volume across four bricks, with a replication factor of 2. The virtual machines are networked together via software bridge (standard KVM). When transferring a 4GB filesystem of relatively small files (mean=795KB), I see write throughput of about 512 KB/s, with reads approaching 3 MB/s. As expected, large files show much better throughput: 20MB/s write and 50MB/s read for a 500MB file. In production, I plan to swap out magnetic drives for SSDs. Will this help with the write performance of small files on glusterfs, as I suspect? Are my results for small files consistent with other users? Cheers, -C- ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] performance expectations?
See bug# 2869 . It looks like there is lot of context switching that occurs when small files are involved since with small files you are doing more system calls. Feel free to ask for update on that bug. I haven't got any yet. Have you tried using Raid Controller with Cache? On Wed, Jul 27, 2011 at 12:56 PM, Chris Haumesser c...@luciddg.com wrote: Hello, I'm keen to use glusterfs as scalable storage to match the rest of my horizontally-scalable, commodity platform. I've been testing on a machine with 4x SATA 7200 drives. I created four KVM virtual machines, gave each of them a dedicated disk, and installed gluster-3.2.2. I created a distributed replicated volume across four bricks, with a replication factor of 2. The virtual machines are networked together via software bridge (standard KVM). When transferring a 4GB filesystem of relatively small files (mean=795KB), I see write throughput of about 512 KB/s, with reads approaching 3 MB/s. As expected, large files show much better throughput: 20MB/s write and 50MB/s read for a 500MB file. In production, I plan to swap out magnetic drives for SSDs. Will this help with the write performance of small files on glusterfs, as I suspect? Are my results for small files consistent with other users? Cheers, -C- ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster client performance
On 07/27/2011 12:53 AM, Pavan T C wrote: 2. What is the disk bandwidth you are getting on the local filesystem on a given storage node ? I mean, pick any of the 10 storage servers dedicated for Gluster Storage and perform a dd as below: Seeing an average of 740 MB/s write, 971 GB/s read. I presume you did this in one of the /data-brick*/export directories ? Command output with the command line would have been clearer, but thats fine. That is correct -- we used /data-brick1/export. 3. What is the IB bandwidth that you are getting between the compute node and the glusterfs storage node? You can run the tool rdma_bw to get the details: 30407: Bandwidth peak (#0 to #976): 2594.58 MB/sec 30407: Bandwidth average: 2593.62 MB/sec 30407: Service Demand peak (#0 to #976): 978 cycles/KB 30407: Service Demand Avg : 978 cycles/KB This looks like a DDR connection. ibv_devinfo -v will tell a better story about the line width and speed of your infiniband connection. QDR should have a much higher bandwidth. But that still does not explain why you should get as low as 50 MB/s for a single stream single client write when the backend can support direct IO throughput of more than 700 MB/s. ibv_devinfo shows 4x for active width and 10 Gbps for active speed. Not sure why we're not seeing better bandwidth with rdma_bw -- we'll have to troubleshoot that some more -- but I agree, it shouldn't be the limiting factor as far the Gluster client speed problems we're seeing. I'll send you the log files you requested off-list. John -- John Lalande University of Wisconsin-Madison Space Science Engineering Center 1225 W. Dayton Street, Room 439, Madison, WI 53706 608-263-2268 / john.lala...@ssec.wisc.edu smime.p7s Description: S/MIME Cryptographic Signature ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster's cpuload is too high on specific birck daemon
Here's my result of strace -f -c -p glusterPID Process 1993 detached Process 1994 detached Process 1996 detached Process 2004 detached Process 2006 detached Process 2013 detached Process 3469 detached % time seconds usecs/call callserrors syscall -- --- --- - - 38.39 23.261809 11899 195586 lgetxattr 28.31 17.155779 35667 48152 futex 16.58 10.046922 70258 143 epoll_wait 15.339.292713 92927110 nanosleep 1.360.826873 826873 1 restart_syscall 0.010.007801 51 154 getdents 0.000.002299 4 572 readv 0.000.002176 1 1787 lstat 0.000.001078 9012 read 0.000.000896 6 141 writev 0.000.000472 3 142 lseek 0.000.000207 2 126 clock_gettime 0.000.000194 49 4 munmap 0.000.79 422 fcntl 0.000.00 026 open 0.000.00 025 close 0.000.00 020 stat 0.000.00 0 4 fstat 0.000.00 0 4 mmap 0.000.00 0 4 statfs -- --- --- - - 100.00 60.599298 5633 138 total -Original Message- From: Pavan T C [mailto:t...@gluster.com] Sent: Wednesday, July 27, 2011 6:18 PM To: 공용준(yongjoon kong)/Cloud Computing 기술담당/SKCC Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] gluster's cpuload is too high on specific birck daemon On Wednesday 27 July 2011 01:45 PM, 공용준(yongjoon kong)/Cloud Computing 기술담당/SKCC wrote: Hello, I'm gluster with distributed-replicated mode(4 brick server) And 10client server mount gluster volume brick1 server. ( mount -t glustefs brick1:/volume /mnt) And there's very strange thing. The brick1's cpu load is too high. From 'top' command, it's over 400% But other brick's load is too low. It is possible that an AFR self heal is getting triggered. On the brick, run the following command: strace -f -c -p glusterfs.pid and provide the output. Pavan Is there any reason for this? Or Is there anyway tracking down this issue? Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
