[Gluster-users] IPv6 support
Hello I have found that glusterfs supports IPv6. However I have completely no idea how to force glusterfs to listen on IPv6 addresses. Could you please update your documentation? regards M. Galkiewicz ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
I'm working with Paul on this. We did take advice on XFS beforehand, and were given the impression that it would just be a performance issue rather than things not actually working. We've got quite fast hardware, and are more comfortable with XFS that ext4 from our own experience so we did our own tests and were happy with XFS performance. Likewise, we're aware of the very poor performance of gluster with small files. We serve a lot of large files, and we're now moved most of the small files off to a normal nfs server. Again small files aren't known to break gluster are they? David On 21 February 2011 14:42, Fabricio Cannini fcann...@gmail.com wrote: Em Sexta-feira 18 Fevereiro 2011, às 23:24:10, paul simpson escreveu: hello all, i have been testing gluster as a central file server for a small animation studio/post production company. my initial experiments were using the fuse glusterfs protocol - but that ran extremely slowly for home dirs and general file sharing. we have since switched to using NFS over glusterfs. NFS has certainly seemed more responsive re. stat and dir traversal. however, i'm now being plagued with three different types of errors: 1/ Stale NFS file handle 2/ input/output errors 3/ and a new one: $ l -l /n/auto/gv1/production/conan/hda/published/OLD/ ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot: Remote I/O error total 0 d? ? ? ? ?? shot ...so it's a bit all over the place. i've tried rebooting both servers and clients. these issues are very erratic - they come and go. some information on my setup: glusterfs 3.1.2 g1:~ # gluster volume info Volume Name: glustervol1 Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: g1:/mnt/glus1 Brick2: g2:/mnt/glus1 Brick3: g3:/mnt/glus1 Brick4: g4:/mnt/glus1 Brick5: g1:/mnt/glus2 Brick6: g2:/mnt/glus2 Brick7: g3:/mnt/glus2 Brick8: g4:/mnt/glus2 Options Reconfigured: performance.write-behind-window-size: 1mb performance.cache-size: 1gb performance.stat-prefetch: 1 network.ping-timeout: 20 diagnostics.latency-measurement: off diagnostics.dump-fd-stats: on that is 4 servers - serving ~30 clients - 95% linux, 5% mac. all NFS. other points: - i'm automounting using NFS via autofs (with ldap). ie: gus:/glustervol1 on /n/auto/gv1 type nfs (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13) gus is pointing to rr dns machines (g1,g2,g3,g4). that all seems to be working. - backend files system on g[1-4] is xfs. ie, g1:/var/log/glusterfs # xfs_info /mnt/glus1 meta-data=/dev/sdb1 isize=256agcount=7, agsize=268435200 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1627196928, imaxpct=5 = sunit=256swidth=2560 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=8 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 - sometimes root can stat/read the file in question while the user cannot! i can remount the same NFS share to another mount point - and i can then see that with the same user. - sample output of g1 nfs.log file: [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/entries [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 1414 bytes [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 001024b+ : 1 [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp [2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/log [2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHN O_DF.6084.exr [2011-02-18 15:27:07.211940] I [io-stats.c:343:io_stats_dump_fd] glustervol1: Lifetime : 8731secs, 610796usecs [2011-02-18 15:27:07.211951] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 2321370 bytes [2011-02-18 15:27:07.211962] I
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
hi fabricio, many thanks for your input. indeed i am using xfs - but that seems to be mentioned in the gluster docs without any mention of problems. we benchmarked xfs vs ext4 - and found that xfs to be much better at dealing with the bulk of our data - hi-def frames ~3-10M each - and large geometry/particle/volume files. 10M-200M. so, i'm keen to hear from anyone abotu xfs's suitability for gluster storage... as for file size; my understanding is that a distributed file system performance only really kicks in when your dealing with large 1M files. however, is dealing with small files meant to be unreliable with locking/access errors? again thanks - and i look forward to hearing if gluster is able to reliably serve svn working directories and cope with locks... regards, paul On Mon, Feb 21, 2011 at 2:42 PM, Fabricio Cannini fcann...@gmail.comwrote: Em Sexta-feira 18 Fevereiro 2011, às 23:24:10, paul simpson escreveu: hello all, i have been testing gluster as a central file server for a small animation studio/post production company. my initial experiments were using the fuse glusterfs protocol - but that ran extremely slowly for home dirs and general file sharing. we have since switched to using NFS over glusterfs. NFS has certainly seemed more responsive re. stat and dir traversal. however, i'm now being plagued with three different types of errors: 1/ Stale NFS file handle 2/ input/output errors 3/ and a new one: $ l -l /n/auto/gv1/production/conan/hda/published/OLD/ ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot: Remote I/O error total 0 d? ? ? ? ?? shot ...so it's a bit all over the place. i've tried rebooting both servers and clients. these issues are very erratic - they come and go. some information on my setup: glusterfs 3.1.2 g1:~ # gluster volume info Volume Name: glustervol1 Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: g1:/mnt/glus1 Brick2: g2:/mnt/glus1 Brick3: g3:/mnt/glus1 Brick4: g4:/mnt/glus1 Brick5: g1:/mnt/glus2 Brick6: g2:/mnt/glus2 Brick7: g3:/mnt/glus2 Brick8: g4:/mnt/glus2 Options Reconfigured: performance.write-behind-window-size: 1mb performance.cache-size: 1gb performance.stat-prefetch: 1 network.ping-timeout: 20 diagnostics.latency-measurement: off diagnostics.dump-fd-stats: on that is 4 servers - serving ~30 clients - 95% linux, 5% mac. all NFS. other points: - i'm automounting using NFS via autofs (with ldap). ie: gus:/glustervol1 on /n/auto/gv1 type nfs (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13) gus is pointing to rr dns machines (g1,g2,g3,g4). that all seems to be working. - backend files system on g[1-4] is xfs. ie, g1:/var/log/glusterfs # xfs_info /mnt/glus1 meta-data=/dev/sdb1 isize=256agcount=7, agsize=268435200 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1627196928, imaxpct=5 = sunit=256swidth=2560 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=8 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 - sometimes root can stat/read the file in question while the user cannot! i can remount the same NFS share to another mount point - and i can then see that with the same user. - sample output of g1 nfs.log file: [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/entries [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 1414 bytes [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 001024b+ : 1 [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp [2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/log [2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHN O_DF.6084.exr [2011-02-18 15:27:07.211940] I [io-stats.c:343:io_stats_dump_fd] glustervol1:
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 09:53 AM, David Lloyd wrote: I'm working with Paul on this. We did take advice on XFS beforehand, and were given the impression that it would just be a performance issue rather than things not actually working. Hi David XFS works fine as a backing store for GlusterFS. We've deployed this now to many customer sites, and have not run into issues with it. We've got quite fast hardware, and are more comfortable with XFS that ext4 from our own experience so we did our own tests and were happy with XFS performance. There are many reasons to choose XFS in general, and there are no issues with using it with GlusterFS. Especially on large file transfers. Likewise, we're aware of the very poor performance of gluster with small files. We serve a lot of large files, and we're now moved most of the small files off to a normal nfs server. Again small files aren't known to break gluster are they? Small files are the bane of every cluster file system. We recommend using NFS client with GlusterFS for smaller files, simply due to the additional caching you can get out of the NFS system. Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On Mon, 21 Feb 2011, Joe Landman wrote: On 02/21/2011 09:53 AM, David Lloyd wrote: Likewise, we're aware of the very poor performance of gluster with small files. We serve a lot of large files, and we're now moved most of the small files off to a normal nfs server. Again small files aren't known to break gluster are they? Small files are the bane of every cluster file system. We recommend using NFS client with GlusterFS for smaller files, simply due to the additional caching you can get out of the NFS system. Those who are looking for better metadata performance might want to see if MooseFS fits their needs. It uses a metadata server which caches the entire filesystem metadata in RAM, so it seems to be very responsive. I have my home directory on GlusterFS and I did a quick trial on MooseFS. du on GlusterFS (native FUSE, not NFS) was 1 minute 8 seconds, du on MooseFS was 2 seconds. Good metadata performance should equate to good small-file performance (but you'll want to run your own tests to be sure). Note that MooseFS stores data with a fixed ~64KB block size, so it will be somewhat wasteful of space on very small files. Thanks, Brent Nelson Director of Computing Dept. of Physics University of Florida ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 12:05 PM, Brent A Nelson wrote: Those who are looking for better metadata performance might want to see if MooseFS fits their needs. It uses a metadata server which caches the entire filesystem metadata in RAM, so it seems to be very responsive. I have my home directory on GlusterFS and I did a quick trial on MooseFS. du on GlusterFS (native FUSE, not NFS) was 1 minute 8 seconds, du on MooseFS was 2 seconds. du isn't metadata intensive. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
Em Segunda-feira 21 Fevereiro 2011, às 12:01:11, Joe Landman escreveu: On 02/21/2011 09:53 AM, David Lloyd wrote: I'm working with Paul on this. We did take advice on XFS beforehand, and were given the impression that it would just be a performance issue rather than things not actually working. Hi David XFS works fine as a backing store for GlusterFS. We've deployed this now to many customer sites, and have not run into issues with it. That's nice to hear. Next time i setup a gluster volume i'm going to take a look at xfs as a backend. BTW Joe, these deployments with xfs as backend fs, which version of gluster have you used ? We've got quite fast hardware, and are more comfortable with XFS that ext4 from our own experience so we did our own tests and were happy with XFS performance. There are many reasons to choose XFS in general, and there are no issues with using it with GlusterFS. Especially on large file transfers. Indeed it is. But i was with the 3.0x series docs on my mind still. Likewise, we're aware of the very poor performance of gluster with small files. We serve a lot of large files, and we're now moved most of the small files off to a normal nfs server. Again small files aren't known to break gluster are they? Small files are the bane of every cluster file system. We recommend using NFS client with GlusterFS for smaller files, simply due to the additional caching you can get out of the NFS system. Good to know. Thanks for the tip. Regards, Joe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 12:17 PM, Fabricio Cannini wrote: Em Segunda-feira 21 Fevereiro 2011, às 12:01:11, Joe Landman escreveu: On 02/21/2011 09:53 AM, David Lloyd wrote: I'm working with Paul on this. We did take advice on XFS beforehand, and were given the impression that it would just be a performance issue rather than things not actually working. Hi David XFS works fine as a backing store for GlusterFS. We've deployed this now to many customer sites, and have not run into issues with it. That's nice to hear. Next time i setup a gluster volume i'm going to take a look at xfs as a backend. BTW Joe, these deployments with xfs as backend fs, which version of gluster have you used ? Everything from 1.3.x onward to 3.1.2 -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
so, while your all about - my big question is can/does gluster (with NFS/fuse client) properly lock files? ie, a simple test is to checkout a svn tree to a gluster, modify, checkin, list, alter, revert. everytime i do this with 3.1.2 i get input/output errors from my client machine within a few minutes. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 09:54 AM, paul simpson wrote: hi fabricio, many thanks for your input. indeed i am using xfs - but that seems to be mentioned in the gluster docs without any mention of problems. we benchmarked xfs vs ext4 - and found that xfs to be much better at dealing with the bulk of our data - hi-def frames ~3-10M each - and large geometry/particle/volume files. 10M-200M. so, i'm keen to hear from anyone abotu xfs's suitability for gluster storage... as for file size; my understanding is that a distributed file system performance only really kicks in when your dealing with large1M files. however, is dealing with small files meant to be unreliable with locking/access errors? We had trouble with reliability for small, actively-accessed files on a distribute-replicate volume in both GlusterFS 3.11 and 3.12. It seems that the replicated servers would eventually get out of sync with each other on these kinds of files. For a while, we dropped replication and only ran the volume as distributed. This has worked reliably for the past week or so without any errors that we were seeing before: no such file, invalid argument, etc. Steve again thanks - and i look forward to hearing if gluster is able to reliably serve svn working directories and cope with locks... regards, paul ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 12:45 PM, Steve Wilson wrote: On 02/21/2011 09:54 AM, paul simpson wrote: hi fabricio, many thanks for your input. indeed i am using xfs - but that seems to be mentioned in the gluster docs without any mention of problems. we benchmarked xfs vs ext4 - and found that xfs to be much better at dealing with the bulk of our data - hi-def frames ~3-10M each - and large geometry/particle/volume files. 10M-200M. so, i'm keen to hear from anyone abotu xfs's suitability for gluster storage... as for file size; my understanding is that a distributed file system performance only really kicks in when your dealing with large1M files. however, is dealing with small files meant to be unreliable with locking/access errors? We had trouble with reliability for small, actively-accessed files on a distribute-replicate volume in both GlusterFS 3.11 and 3.12. It seems that the replicated servers would eventually get out of sync with each other on these kinds of files. For a while, we dropped replication and only ran the volume as distributed. This has worked reliably for the past week or so without any errors that we were seeing before: no such file, invalid argument, etc. Steve: As a sanity check, do test your date stamps across the servers. We found *significant* issues when they drifted. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 12:47 PM, Joe Landman wrote: On 02/21/2011 12:45 PM, Steve Wilson wrote: On 02/21/2011 09:54 AM, paul simpson wrote: hi fabricio, many thanks for your input. indeed i am using xfs - but that seems to be mentioned in the gluster docs without any mention of problems. we benchmarked xfs vs ext4 - and found that xfs to be much better at dealing with the bulk of our data - hi-def frames ~3-10M each - and large geometry/particle/volume files. 10M-200M. so, i'm keen to hear from anyone abotu xfs's suitability for gluster storage... as for file size; my understanding is that a distributed file system performance only really kicks in when your dealing with large1M files. however, is dealing with small files meant to be unreliable with locking/access errors? We had trouble with reliability for small, actively-accessed files on a distribute-replicate volume in both GlusterFS 3.11 and 3.12. It seems that the replicated servers would eventually get out of sync with each other on these kinds of files. For a while, we dropped replication and only ran the volume as distributed. This has worked reliably for the past week or so without any errors that we were seeing before: no such file, invalid argument, etc. Steve: As a sanity check, do test your date stamps across the servers. We found *significant* issues when they drifted. Thanks, Joe. Both servers use NTP against the same subnet router so it's unlikely that they had a time discrepancy. I just checked the two servers and their times are consistent with each other at the moment. Steve ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
Thanks, Joe. Both servers use NTP against the same subnet router so it's unlikely that they had a time discrepancy. I just checked the two servers and their times are consistent with each other at the moment. ...same with our machines here. all running ntpd. it's good to hear i'm not alone with this issue - it's been making me tear my hair out. it's a show-stopper for a file system... joe - i really hope your correct with 3.1.3 fixing this. i'd dearly love to hear from any gluster people - anonymous or otherwise... ;) these kind of errors really undermine that cast iron feeling you need when thinking about using a filesystem in a live production environment. regards to all, paul ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On Mon, Feb 21, 2011 at 9:45 AM, Steve Wilson ste...@purdue.edu wrote: We had trouble with reliability for small, actively-accessed files on a distribute-replicate volume in both GlusterFS 3.11 and 3.12. It seems that the replicated servers would eventually get out of sync with each other on these kinds of files. For a while, we dropped replication and only ran the volume as distributed. This has worked reliably for the past week or so without any errors that we were seeing before: no such file, invalid argument, etc. I'm running thousands of small files over NFSv3 through NGINX with distribute and have had the opposite experience. Unfortunately when NGINX can't access a file over NFS it means a customer calling us, so right now gluster is basically sitting idle (posted my output to the list a while back with no response). Cheers Kon ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 01:39 PM, Kon Wilms wrote: On Mon, Feb 21, 2011 at 9:45 AM, Steve Wilsonste...@purdue.edu wrote: We had trouble with reliability for small, actively-accessed files on a distribute-replicate volume in both GlusterFS 3.11 and 3.12. It seems that the replicated servers would eventually get out of sync with each other on these kinds of files. For a while, we dropped replication and only ran the volume as distributed. This has worked reliably for the past week or so without any errors that we were seeing before: no such file, invalid argument, etc. I'm running thousands of small files over NFSv3 through NGINX with distribute and have had the opposite experience. Unfortunately when NGINX can't access a file over NFS it means a customer calling us, so right now gluster is basically sitting idle (posted my output to the list a while back with no response). We've had lots of issues with files disappearing or being inaccessible prior to 3.1.2 with the NFS client and server translator. After 3.1.2, many of these problems *seem* to have been resolved, though all this means in this instance is that the customer hasn't submitted a ticket yet. I had thought it was originally a timebase issue ... as we had a minute or two drift on some of the nodes (since fixed). But we had a pretty consistent error in this regard. We did open problem reports. Unfortunately, no action so far (they just closed them this morning, though nothing has been solved per se, the issue simply has not yet resurfaced). I'll leave those reports closed for now. This said, this error, or one with a very similar signature, has been in the code since the 2.x series. I really ... really want to track it down, but I can't create a simple replicator for it to present to the team. If you have what you think is a simple replicator, please, email me offline. We'll try it here, and if we can get it down to a very simple replication case and test, we'll re-open the bugs. I'd hate to think its a heisenbug, but that is where I am leaning now. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
thanks for all responding - at least i know i'm not all alone. i am shocked to think that so many on this list are having serious fundamental issues with glusterfs - and seemingly for a long time. so, without wanting to troll - my question is: is gluster a serious stable general purpose file system? or, is it more a good caching system for a specific narrow domain? i'd really like to hear from any official gluster people out there - right now the silence is deafening. is this issue know? is it viewed a serious? it is being worked on? i'm all up for volunteering to help by sending in a test case, sending logs - whatever is asked of me. i want to believe gluster is going to work - as do many other sys-admins i know of in the post/film industry. however, i'm rapidly loosing confidence in gluster with each passing day of silence... in hope - paul On Mon, Feb 21, 2011 at 6:47 PM, Joe Landman land...@scalableinformatics.com wrote: On 02/21/2011 01:39 PM, Kon Wilms wrote: On Mon, Feb 21, 2011 at 9:45 AM, Steve Wilsonste...@purdue.edu wrote: We had trouble with reliability for small, actively-accessed files on a distribute-replicate volume in both GlusterFS 3.11 and 3.12. It seems that the replicated servers would eventually get out of sync with each other on these kinds of files. For a while, we dropped replication and only ran the volume as distributed. This has worked reliably for the past week or so without any errors that we were seeing before: no such file, invalid argument, etc. I'm running thousands of small files over NFSv3 through NGINX with distribute and have had the opposite experience. Unfortunately when NGINX can't access a file over NFS it means a customer calling us, so right now gluster is basically sitting idle (posted my output to the list a while back with no response). We've had lots of issues with files disappearing or being inaccessible prior to 3.1.2 with the NFS client and server translator. After 3.1.2, many of these problems *seem* to have been resolved, though all this means in this instance is that the customer hasn't submitted a ticket yet. I had thought it was originally a timebase issue ... as we had a minute or two drift on some of the nodes (since fixed). But we had a pretty consistent error in this regard. We did open problem reports. Unfortunately, no action so far (they just closed them this morning, though nothing has been solved per se, the issue simply has not yet resurfaced). I'll leave those reports closed for now. This said, this error, or one with a very similar signature, has been in the code since the 2.x series. I really ... really want to track it down, but I can't create a simple replicator for it to present to the team. If you have what you think is a simple replicator, please, email me offline. We'll try it here, and if we can get it down to a very simple replication case and test, we'll re-open the bugs. I'd hate to think its a heisenbug, but that is where I am leaning now. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
On 02/21/2011 02:06 PM, paul simpson wrote: thanks for all responding - at least i know i'm not all alone. i am shocked to think that so many on this list are having serious fundamental issues with glusterfs - and seemingly for a long time. so, without wanting to troll - my question is: is gluster a serious stable general purpose file system? or, is it more a good caching system for a specific narrow domain? I do believe it is a serious and stable system. We are running into specific bugs. I'm ok with workarounds, but I really want to find a way to cause the problems. No one really questions other parallel FS, and from some of our experiences with them, they have far worse issues. Even with the issues, Gluster is one of the best on the market. Gluster is evolving, but the issue is, without the replicator, all we can do is say we are experiencing an issue and hope it can be tracked. i'd really like to hear from any official gluster people out there - right now the silence is deafening. is this issue know? is it viewed a I don't know if you have a support contract with them. If you do, you should be speaking with them directly. If not, their paying customers come first. serious? it is being worked on? i'm all up for volunteering to help by sending in a test case, sending logs - whatever is asked of me. i want to believe gluster is going to work - as do many other sys-admins i know of in the post/film industry. however, i'm rapidly loosing confidence in gluster with each passing day of silence... We have a growing number of customers in this space (and several related). We are highly motivated to find answers. And we are opening bugs as we find replicators that they can use. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
hi joe, it's very reassuring to read your post. answers in-line. I do believe it is a serious and stable system. We are running into specific bugs. I'm ok with workarounds, but I really want to find a way to cause the problems. same with both. right now i'm putting more of my production tree back onto an NFS server. i think this is quite a reproducible bug: just check out a subversion repo onto a distributed replicated volume - alter some files, check them back in and repeat a few times. within 20 mins we've been getting input/output errors and stale NFS handles. this is with both NFS/fuse clients on multiple clients. No one really questions other parallel FS, and from some of our experiences with them, they have far worse issues. Even with the issues, Gluster is one of the best on the market. Gluster is evolving, but the issue is, without the replicator, all we can do is say we are experiencing an issue and hope it can be tracked. i hear you - and very much understand the value of reproducible bug reports :) i hope above example helps. i've already sent a detailed report to the gluster people last week. i'd be happy to give them a ssh account to see it first hand. i want to help/contribute... i'd really like to hear from any official gluster people out there - right now the silence is deafening. is this issue know? is it viewed a I don't know if you have a support contract with them. If you do, you should be speaking with them directly. If not, their paying customers come first. i have chatted with them - and certainly would expect that to service paying customers (which we are not yet). i've had couple of chats - and they seem like a very cool helpful switched on bunch... again, thanks for responding. good to know i'm not alone.. -paul ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
Hi Paul, Locking is part of the core GlusterFS protocol but the NFS server module does not have NLM support yet(NLM is the locking protocol associated with NFSv3). On linux, the workaround is generally to mount with the -o nolock option although I dont see why excluding this option results in stale file handle and other errors. let me go through the complete thread, I'll reply elsewhere. Today, if locking among multiple client machines is a must-have, you'll have to use FUSE. paul simpson wrote: so, while your all about - my big question is can/does gluster (with NFS/fuse client) properly lock files? ie, a simple test is to checkout a svn tree to a gluster, modify, checkin, list, alter, revert. everytime i do this with 3.1.2 i get input/output errors from my client machine within a few minutes. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
paul simpson wrote: hello all, i have been testing gluster as a central file server for a small animation studio/post production company. my initial experiments were using the fuse glusterfs protocol - but that ran extremely slowly for home dirs and general file sharing. we have since switched to using NFS over glusterfs. NFS has certainly seemed more responsive re. stat and dir traversal. however, i'm now being plagued with three different types of errors: 1/ Stale NFS file handle 2/ input/output errors 3/ and a new one: $ l -l /n/auto/gv1/production/conan/hda/published/OLD/ ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot: Remote I/O error total 0 d? ? ? ? ?? shot ...so it's a bit all over the place. i've tried rebooting both servers and clients. these issues are very erratic - they come and go. some information on my setup: glusterfs 3.1.2 g1:~ # gluster volume info Volume Name: glustervol1 Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: g1:/mnt/glus1 Brick2: g2:/mnt/glus1 Brick3: g3:/mnt/glus1 Brick4: g4:/mnt/glus1 Brick5: g1:/mnt/glus2 Brick6: g2:/mnt/glus2 Brick7: g3:/mnt/glus2 Brick8: g4:/mnt/glus2 Options Reconfigured: performance.write-behind-window-size: 1mb performance.cache-size: 1gb performance.stat-prefetch: 1 network.ping-timeout: 20 diagnostics.latency-measurement: off diagnostics.dump-fd-stats: on that is 4 servers - serving ~30 clients - 95% linux, 5% mac. all NFS. Mac OS as a nfs client remains untested against Gluster NFS. Do you see these errors on Mac or Linux clients? other points: - i'm automounting using NFS via autofs (with ldap). ie: gus:/glustervol1 on /n/auto/gv1 type nfs (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13) gus is pointing to rr dns machines (g1,g2,g3,g4). that all seems to be working. - backend files system on g[1-4] is xfs. ie, g1:/var/log/glusterfs # xfs_info /mnt/glus1 meta-data=/dev/sdb1 isize=256agcount=7, agsize=268435200 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1627196928, imaxpct=5 = sunit=256swidth=2560 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=8 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 - sometimes root can stat/read the file in question while the user cannot! i can remount the same NFS share to another mount point - and i can then see that with the same user. I think that may be occurring because NFS+LDAP requires a slightly different authentication scheme as compared to a NFS only setup. Please try the same test without LDAP in the middle. - sample output of g1 nfs.log file: [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/entries [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 1414 bytes [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 001024b+ : 1 [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp [2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/log [2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHNO_DF.6084.exr [2011-02-18 15:27:07.211940] I [io-stats.c:343:io_stats_dump_fd] glustervol1: Lifetime : 8731secs, 610796usecs [2011-02-18 15:27:07.211951] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 2321370 bytes [2011-02-18 15:27:07.211962] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 000512b+ : 1 [2011-02-18 15:27:07.211972] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 002048b+ : 1 [2011-02-18 15:27:07.211983] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 004096b+ : 4 [2011-02-18 15:27:07.212009] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 008192b+ : 4 [2011-02-18 15:27:07.212019] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 016384b+ : 20 [2011-02-18 15:27:07.212030] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 032768b+ : 54 [2011-02-18 15:27:07.228051] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd
Re: [Gluster-users] Fwd: files not syncing up with glusterfs 3.1.2
David Lloyd wrote: I'm working with Paul on this. We did take advice on XFS beforehand, and were given the impression that it would just be a performance issue rather than things not actually working. We've got quite fast hardware, and are more comfortable with XFS that ext4 from our own experience so we did our own tests and were happy with XFS performance. Likewise, we're aware of the very poor performance of gluster with small files. We serve a lot of large files, and we're now moved most of the small files off to a normal nfs server. Again small files aren't known to break gluster are they? No. Gluster will work fine but the usual caveats about small file performance over a network file system apply. -Shehjar David On 21 February 2011 14:42, Fabricio Cannini fcann...@gmail.com wrote: Em Sexta-feira 18 Fevereiro 2011, às 23:24:10, paul simpson escreveu: hello all, i have been testing gluster as a central file server for a small animation studio/post production company. my initial experiments were using the fuse glusterfs protocol - but that ran extremely slowly for home dirs and general file sharing. we have since switched to using NFS over glusterfs. NFS has certainly seemed more responsive re. stat and dir traversal. however, i'm now being plagued with three different types of errors: 1/ Stale NFS file handle 2/ input/output errors 3/ and a new one: $ l -l /n/auto/gv1/production/conan/hda/published/OLD/ ls: cannot access /n/auto/gv1/production/conan/hda/published/OLD/shot: Remote I/O error total 0 d? ? ? ? ?? shot ...so it's a bit all over the place. i've tried rebooting both servers and clients. these issues are very erratic - they come and go. some information on my setup: glusterfs 3.1.2 g1:~ # gluster volume info Volume Name: glustervol1 Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: g1:/mnt/glus1 Brick2: g2:/mnt/glus1 Brick3: g3:/mnt/glus1 Brick4: g4:/mnt/glus1 Brick5: g1:/mnt/glus2 Brick6: g2:/mnt/glus2 Brick7: g3:/mnt/glus2 Brick8: g4:/mnt/glus2 Options Reconfigured: performance.write-behind-window-size: 1mb performance.cache-size: 1gb performance.stat-prefetch: 1 network.ping-timeout: 20 diagnostics.latency-measurement: off diagnostics.dump-fd-stats: on that is 4 servers - serving ~30 clients - 95% linux, 5% mac. all NFS. other points: - i'm automounting using NFS via autofs (with ldap). ie: gus:/glustervol1 on /n/auto/gv1 type nfs (rw,vers=3,rsize=32768,wsize=32768,intr,sloppy,addr=10.0.0.13) gus is pointing to rr dns machines (g1,g2,g3,g4). that all seems to be working. - backend files system on g[1-4] is xfs. ie, g1:/var/log/glusterfs # xfs_info /mnt/glus1 meta-data=/dev/sdb1 isize=256agcount=7, agsize=268435200 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1627196928, imaxpct=5 = sunit=256swidth=2560 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=8 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 - sometimes root can stat/read the file in question while the user cannot! i can remount the same NFS share to another mount point - and i can then see that with the same user. - sample output of g1 nfs.log file: [2011-02-18 15:27:07.201433] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/entries [2011-02-18 15:27:07.201445] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 1414 bytes [2011-02-18 15:27:07.201455] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 001024b+ : 1 [2011-02-18 15:27:07.205999] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.206032] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/props/tempfile.tmp [2011-02-18 15:27:07.210799] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.210824] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /production/conan/hda/published/shot/backup/.svn/tmp/log [2011-02-18 15:27:07.211904] I [io-stats.c:333:io_stats_dump_fd] glustervol1: --- fd stats --- [2011-02-18 15:27:07.211928] I [io-stats.c:338:io_stats_dump_fd] glustervol1: Filename : /prod_data/xmas/lgl/pic/mr_all_PBR_HIGHNO_DF/035/1920x1080/mr_all_PBR_HIGHN O_DF.6084.exr [2011-02-18 15:27:07.211940] I [io-stats.c:343:io_stats_dump_fd] glustervol1: Lifetime : 8731secs, 610796usecs [2011-02-18 15:27:07.211951] I [io-stats.c:353:io_stats_dump_fd] glustervol1: BytesWritten : 2321370 bytes [2011-02-18 15:27:07.211962] I [io-stats.c:365:io_stats_dump_fd] glustervol1: Write 000512b+ : 1
