[Gluster-devel] filesystem full / unify
Hello ml, we have the following gluster setup for our backup storage. |-- u n i f y | | gluster1 <-- afr ---> gluster2 | | gluster3 <-- afr ---> gluster4 | | gluster5 <-- afr ---> gluster6 | | gluster7 <-- afr ---> gluster8 | |-- u n i f y | gluster1 / 2 has 8gb free diskspace gluster3 / 4 has 2 gb free diskspace (*) gluster5 / 6 has 2 gb free diskspace (*) gluster7 / 8 has 2 tb free diskspace (*) => no writes due to alu.limits.min-free-disk 6GB Now, when i try to copy a large file (40gb) into the gluster it starts writing to gluster1/2. after 8gb gluster1/2 has no more free diskspace and the copy process dies. Is there a configuration option for unify to write on an other gluster-server if one server run out of space during a write? Regards, jan ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] AFR between two bricks over 3000 miles
On Thu, 28 Feb 2008, LI Daobing wrote: On Thu, Feb 28, 2008 at 2:07 PM, Anand Avati <[EMAIL PROTECTED]> wrote: on the client. you also might want to put this write-behind + io-cache pair in the subvolume path of afr which leads towards the remote site alone. Also make sure afr does not have the remote site as the first subvolume, and has the option read-subvolume so that reads are not scheduled to the remote site. in 1.3.8pre1, the order of afr's subvolume should be same among clients, otherwise there is a race condition on create and other operations on inode. consider two client are creating a same file at a same time. So then it is not possible to have clients each write to their local part of the AFR since the order needs to be the same? -nathan ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] AFR between two bricks over 3000 miles
On Thu, Feb 28, 2008 at 11:33 PM, <[EMAIL PROTECTED]> wrote: > > > On Thu, 28 Feb 2008, LI Daobing wrote: > > > On Thu, Feb 28, 2008 at 2:07 PM, Anand Avati <[EMAIL PROTECTED]> wrote: > >> on the client. you also might want to put this write-behind + io-cache > pair > >> in the subvolume path of afr which leads towards the remote site alone. > Also > >> make sure afr does not have the remote site as the first subvolume, and > has > >> the option read-subvolume so that reads are not scheduled > to > >> the remote site. > > > > in 1.3.8pre1, the order of afr's subvolume should be same among > > clients, otherwise there is a race condition on create and other > > operations on inode. consider two client are creating a same file at a > > same time. > > So then it is not possible to have clients each write to their local part > of the AFR since the order needs to be the same? > afr need write to every sub volume, so you can't only write to the local part in any case. -- Best Regards, LI Daobing ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
[Gluster-devel] very strange in disk reads
Hello, I have AFR setup on 2 nodes. The nodes are on a wireless connection, so I can really see a performance lag when reading data. On each node, a glusterfs client is mounted pointing to the same glusterfs directory. Here is the strange part: node 1: read file1.txt read file2.txt node 2: read file1.txt read file2.txt We are seeing inconsistent performance from one of the nodes. Suddenly instead of reading over the gluster fs network, it seems to be reading the file only from the local node's gluster filesystem as if failover occurred. word count on file1.txt takes 5 seconds on node 1 and node 2 Then all of a sudden word count on file1.txt takes .2 seconds on node 1, but same 5 seconds on node 2, almost as if the client on node 1 saw a terminated connection of node 2 and only pulled locally. Now here is the stranger part. file2.txt takes 5 seconds to obtain a word count on node 1 and node 2 consistently. Even when file1.txt on node 1 takes .2 seconds to obtain a word count, file2.txt on that same node still takes 5 seconds. We are using round robin for writes (probably unrelated) No errors or messages of disconnect in glusterfs.log or glusterfsd.log. All seems healthy. Using posix-locks and AFR. No other features/etc Any ideas? Thanks. Billy ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] filesystem full / unify
Let me tell how it works currently, and why. When a file is created on filesystem, there is no way a for a filesystem to know about its future size. It may be just a 0byte file, may be 4k, may be 1GB, may be 100GB. Hence, while creating a file, GlusterFS just creates it on the best possible node (depending on the scheduler) at the time of creation. Hope that explains the reason why the 40GB file got created on the afr (gluster1/2). snip |-- u n i f y | | gluster1 <-- afr ---> gluster2 | | gluster3 <-- afr ---> gluster4 | | gluster5 <-- afr ---> gluster6 | | gluster7 <-- afr ---> gluster8 | |-- u n i f y | gluster1 / 2 has 8gb free diskspace gluster3 / 4 has 2 gb free diskspace (*) gluster5 / 6 has 2 gb free diskspace (*) gluster7 / 8 has 2 tb free diskspace (*) => no writes due to alu.limits.min-free-disk 6GB Now, when i try to copy a large file (40gb) into the gluster it starts writing to gluster1/2. after 8gb gluster1/2 has no more free diskspace and the copy process dies. Is there a configuration option for unify to write on an other gluster-server if one server run out of space during a write? end-snip Well, then it would be great if we could handle such cases too. (Its on the roadmap) currently there is a switch scheduler in 1.3.8pre1 or latest tla archives, where you can specify where the file should go if it matches a specific pattern. (this helps if you have a pattern for all the large files). Another way of tackling this problem is using stripe, where you can stripe your file across, but well, it may not suit everyone. It would be an ideal case if users had a way to tell about the possible filesize during the creation. There is no such option in any of the currently available kernels/system calls. Now, we can have discussion on how to handle such situation if anyone has idea. Please do think about all the parameters before suggesting one. For example I can say "well, when the file reaches 6GB on afr (gluster1/2) you are just left with 2GB there, hence, move it to afr (gluster7/8) where you have 2TB', but now, while a single 'write()' gets the error ENOSPC, I have to move whole of 6GB from one node to another internally, which stops the application for that long (by that time, 'transport-timeout' would be called too). Next problem is where will you go if the file size reaches 2TB ? Open for design ideas. Thanks and regards, Amar > Regards, > > jan > > > ___ > Gluster-devel mailing list > Gluster-devel@nongnu.org > http://lists.nongnu.org/mailman/listinfo/gluster-devel > -- Amar Tumballi Gluster/GlusterFS Hacker [bulde on #gluster/irc.gnu.org] http://www.zresearch.com - Commoditizing Supercomputing and Superstorage! ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
[Gluster-devel] what am i missing here? (fast transfer only through 1 client)
Any help is greatly appreciated: server 1 - 1Gbps & identical hardware server 2 - 1Gbps & identical hardware both have identical client & server conf files w/ performance enhancement and afr the clients are run to mount to the same file path on both servers. now, scping a file from local file system on server1 to mounted gluster fs on server 2 transfers at ~3MB/s scping a file from local file system on server2 to mounted gluster fs on server1 transfers at ~23MB/s ALSO, scping to non-glusterfs yields ~23MB/s from server1 to server2 and vise versa, so something with gluster that is bottlenecking at 3MB/s If I change the afr section of the client files to list "subvolume brick1 brick2" to "subvolume brick2 brick1", the transfer rates above switch - now server 1 to server 2 is fast and server 2 to server 1 is slow. volume afr type cluster/afr subvolumes brick1 brick2 option self-heal on end-volume -- to -- volume afr type cluster/afr subvolumes brick2 brick1 option self-heal on end-volume /var/lib/data is the mounted file system from glusterfs-client.vol [EMAIL PROTECTED]:/var/lib$ scp /tmp/2.mpg [EMAIL PROTECTED]:/var/lib/data/ 2.mpg 14% 62MB 2.4MB/s 02:34 ETA Now, the other way: [EMAIL PROTECTED]:/tmp$ sudo scp 3.mpg [EMAIL PROTECTED]:/var/lib/data/ 3.mpg 100% 71MB 23.6MB/s 00:03 ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] what am i missing here? (fast transfer only through 1 client)
Can you share your client config? It surely looks strange. On Thu, Feb 28, 2008 at 5:34 PM, billy cokalopolous < [EMAIL PROTECTED]> wrote: > Any help is greatly appreciated: > > server 1 - 1Gbps & identical hardware > server 2 - 1Gbps & identical hardware > > both have identical client & server conf files w/ performance enhancement > and afr > > the clients are run to mount to the same file path on both servers. > > now, scping a file from local file system on server1 to mounted gluster fs > on server 2 transfers at ~3MB/s > scping a file from local file system on server2 to mounted gluster fs on > server1 transfers at ~23MB/s > ALSO, scping to non-glusterfs yields ~23MB/s from server1 to server2 and > vise versa, so something with gluster that is bottlenecking at 3MB/s > > If I change the afr section of the client files to list "subvolume brick1 > brick2" to "subvolume brick2 brick1", the transfer rates above switch - > now > server 1 to server 2 is fast and server 2 to server 1 is slow. > > volume afr > type cluster/afr > subvolumes brick1 brick2 > option self-heal on > end-volume > -- > to > -- > volume afr > type cluster/afr > subvolumes brick2 brick1 > option self-heal on > end-volume > > /var/lib/data is the mounted file system from glusterfs-client.vol > > [EMAIL PROTECTED]:/var/lib$ scp /tmp/2.mpg [EMAIL PROTECTED]:/var/lib/data/ > 2.mpg 14% 62MB 2.4MB/s > 02:34 > ETA > > Now, the other way: > > [EMAIL PROTECTED]:/tmp$ sudo scp 3.mpg [EMAIL PROTECTED]:/var/lib/data/ > 3.mpg 100% 71MB 23.6MB/s > 00:03 > ___ > Gluster-devel mailing list > Gluster-devel@nongnu.org > http://lists.nongnu.org/mailman/listinfo/gluster-devel > -- Amar Tumballi Gluster/GlusterFS Hacker [bulde on #gluster/irc.gnu.org] http://www.zresearch.com - Commoditizing Supercomputing and Superstorage! ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
[Gluster-devel] failure to compile on OSX
Hi, The wiki suggests that glusterfs should compile on OSX. When I attempt to do so, I get the following error: gcc -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Wall -I../../../../libglusterfs/src -shared -nostartfiles -bundle -undefined suppress -flat_namespace -D__FreeBSD__=10 -o unify.so unify.o unify-self-heal.o -lpthread ld: can't insert lazy pointers, __dyld section not found collect2: ld returned 1 exit status make[5]: *** [unify.so] Error 1 make[4]: *** [all-recursive] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 This is with a checkout from arch just today, I am not really sure how to use arch, so I am unsure how to get a revision number. the last couple lines of the checkout were: * patching for revision: * [EMAIL PROTECTED]/glusterfs--mainline--2.5--patch-689 * making pristine copy * tree version set [EMAIL PROTECTED]/glusterfs--mainline--2.5 I hope this helps. Thanks!! Luke Schierer Accelerated Servers. ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
[Gluster-devel] Glusterfs on Mac OS X
I see on the Glusterfs wiki FAQ that Glusterfs has been tested on Mac OS X. Can I get a bit more info on this, ie which gluster revision was used, which OS version, config flags etc. I have access to 300 iMac (each with 250G drive) running 10.5 and I want to put the excess diskspace (some 60TB) to good use. I tla'ed the glusterfs--mainline--2.5 revision (which seemed to be the latest) and got it to compile. The server seems to works. The client mounts ok but the stats on the mount gives an io error: > ls -al /mnt > ls: glusterfs: Input/output error -- Dr. Rodney G. McDuff |Ex ignorantia ad sapientiam Manager, Strategic Technologies Group|Ex luce ad tenebras Information Technology Services | The University of Queensland | EMAIL: [EMAIL PROTECTED] | TELEPHONE: +61 7 3365 8220 | ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] failure to compile on OSX
Hi Luke, can you try to configure like below? CFLAGS='-D__FreeBSD__=10 -D__DARWIN_64_BIT_INO_T' ./configure Regards, Amar On Thu, Feb 28, 2008 at 8:38 PM, Luke Schierer < [EMAIL PROTECTED]> wrote: > Hi, > The wiki suggests that glusterfs should compile on OSX. When I attempt > to do so, I get the following error: > > gcc -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Wall > -I../../../../libglusterfs/src -shared -nostartfiles -bundle -undefined > suppress -flat_namespace -D__FreeBSD__=10 -o unify.so unify.o > unify-self-heal.o -lpthread > ld: can't insert lazy pointers, __dyld section not found > collect2: ld returned 1 exit status > make[5]: *** [unify.so] Error 1 > make[4]: *** [all-recursive] Error 1 > make[3]: *** [all-recursive] Error 1 > make[2]: *** [all-recursive] Error 1 > make[1]: *** [all-recursive] Error 1 > make: *** [all] Error 2 > > This is with a checkout from arch just today, I am not really sure how > to use arch, so I am unsure how to get a revision number. the last > couple lines of the checkout were: > > * patching for revision: > * [EMAIL PROTECTED]/glusterfs--mainline--2.5--patch-689 > * making pristine copy > * tree version set [EMAIL PROTECTED]/glusterfs--mainline--2.5 > > > I hope this helps. > > Thanks!! > > Luke Schierer > Accelerated Servers. > > > ___ > Gluster-devel mailing list > Gluster-devel@nongnu.org > http://lists.nongnu.org/mailman/listinfo/gluster-devel > -- Amar Tumballi Gluster/GlusterFS Hacker [bulde on #gluster/irc.gnu.org] http://www.zresearch.com - Commoditizing Supercomputing and Superstorage! ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] failure to compile on OSX
Sorry for being bit lazy on documentation part. We are still under testing part with regard to MAC OS X. http://www.gluster.org/docs/index.php/GlusterFS_on_MAC_OS_X Also use '-d disable' while mounting glusterfs, because O_APPEND is not handled in MacFUSE. Regards, Amar On Thu, Feb 28, 2008 at 8:54 PM, Rodney McDuff <[EMAIL PROTECTED]> wrote: > Amar S. Tumballi wrote: > > Hi Luke, > > can you try to configure like below? > > > > CFLAGS='-D__FreeBSD__=10 -D__DARWIN_64_BIT_INO_T' ./configure > > > Yep. That what worked for me on my 10.5 iMac. Except while the server > seems to work the client just gives me. > > > ls -al /mnt > >> ls: glusterfs: Input/output error > > > > > > > What gluster and OS versions do you have? > > > > > Regards, > > Amar > > > > On Thu, Feb 28, 2008 at 8:38 PM, Luke Schierer < > > [EMAIL PROTECTED]> wrote: > > > >> Hi, > >> The wiki suggests that glusterfs should compile on OSX. When I attempt > >> to do so, I get the following error: > >> > >> gcc -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Wall > >> -I../../../../libglusterfs/src -shared -nostartfiles -bundle -undefined > >> suppress -flat_namespace -D__FreeBSD__=10 -o unify.so unify.o > >> unify-self-heal.o -lpthread > >> ld: can't insert lazy pointers, __dyld section not found > >> collect2: ld returned 1 exit status > >> make[5]: *** [unify.so] Error 1 > >> make[4]: *** [all-recursive] Error 1 > >> make[3]: *** [all-recursive] Error 1 > >> make[2]: *** [all-recursive] Error 1 > >> make[1]: *** [all-recursive] Error 1 > >> make: *** [all] Error 2 > >> > >> This is with a checkout from arch just today, I am not really sure how > >> to use arch, so I am unsure how to get a revision number. the last > >> couple lines of the checkout were: > >> > >> * patching for revision: > >> * [EMAIL PROTECTED]/glusterfs--mainline--2.5--patch-689 > >> * making pristine copy > >> * tree version set [EMAIL PROTECTED]/glusterfs--mainline--2.5 > >> > >> > >> I hope this helps. > >> > >> Thanks!! > >> > >> Luke Schierer > >> Accelerated Servers. > >> > >> > >> ___ > >> Gluster-devel mailing list > >> Gluster-devel@nongnu.org > >> http://lists.nongnu.org/mailman/listinfo/gluster-devel > >> > > > > > > > > > -- > Dr. Rodney G. McDuff |Ex ignorantia ad sapientiam > Manager, Strategic Technologies Group|Ex luce ad tenebras > Information Technology Services | > The University of Queensland | > EMAIL: [EMAIL PROTECTED] | > TELEPHONE: +61 7 3365 8220 | > > > -- Amar Tumballi Gluster/GlusterFS Hacker [bulde on #gluster/irc.gnu.org] http://www.zresearch.com - Commoditizing Supercomputing and Superstorage! ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] failure to compile on OSX
Amar S. Tumballi wrote: > Sorry for being bit lazy on documentation part. We are still under testing > part with regard to MAC OS X. > > http://www.gluster.org/docs/index.php/GlusterFS_on_MAC_OS_X Thanks Amar. These instructions worked for me. Both client and server seem to be working fine. However the diskusage seems a bit strange > bash-3.2# df -k /mnt/glusterfs > Filesystem 1024-blocks Used Available Capacity Mounted on > glusterfs741120347136 0%/mnt/glusterfs > > bash-3.2# du -sk /tmp/export > 85500 /tmp/export /tmp/export is where the server is exporting. > > Also use '-d disable' while mounting glusterfs, because O_APPEND is not > handled in MacFUSE. > > Regards, > Amar > > On Thu, Feb 28, 2008 at 8:54 PM, Rodney McDuff <[EMAIL PROTECTED]> wrote: > >> Amar S. Tumballi wrote: >>> Hi Luke, >>> can you try to configure like below? >>> >>> CFLAGS='-D__FreeBSD__=10 -D__DARWIN_64_BIT_INO_T' ./configure >> >> Yep. That what worked for me on my 10.5 iMac. Except while the server >> seems to work the client just gives me. >> >>> ls -al /mnt ls: glusterfs: Input/output error >> >> >> >> >> >> What gluster and OS versions do you have? >> >>> Regards, >>> Amar >>> >>> On Thu, Feb 28, 2008 at 8:38 PM, Luke Schierer < >>> [EMAIL PROTECTED]> wrote: >>> Hi, The wiki suggests that glusterfs should compile on OSX. When I attempt to do so, I get the following error: gcc -fPIC -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -Wall -I../../../../libglusterfs/src -shared -nostartfiles -bundle -undefined suppress -flat_namespace -D__FreeBSD__=10 -o unify.so unify.o unify-self-heal.o -lpthread ld: can't insert lazy pointers, __dyld section not found collect2: ld returned 1 exit status make[5]: *** [unify.so] Error 1 make[4]: *** [all-recursive] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 This is with a checkout from arch just today, I am not really sure how to use arch, so I am unsure how to get a revision number. the last couple lines of the checkout were: * patching for revision: * [EMAIL PROTECTED]/glusterfs--mainline--2.5--patch-689 * making pristine copy * tree version set [EMAIL PROTECTED]/glusterfs--mainline--2.5 I hope this helps. Thanks!! Luke Schierer Accelerated Servers. ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel >>> >>> >> >> -- >> Dr. Rodney G. McDuff |Ex ignorantia ad sapientiam >> Manager, Strategic Technologies Group|Ex luce ad tenebras >> Information Technology Services | >> The University of Queensland | >> EMAIL: [EMAIL PROTECTED] | >> TELEPHONE: +61 7 3365 8220 | >> >> >> > > -- Dr. Rodney G. McDuff |Ex ignorantia ad sapientiam Manager, Strategic Technologies Group|Ex luce ad tenebras Information Technology Services | The University of Queensland | EMAIL: [EMAIL PROTECTED] | TELEPHONE: +61 7 3365 8220 | ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Glusterfs on Mac OS X
Excerpts from Rodney McDuff's message of Fri Feb 29 10:14:06 +0530 2008: > I see on the Glusterfs wiki FAQ that Glusterfs has been tested on Mac OS > X. Can I get a bit more info on this, ie which gluster revision was > used, which OS version, config flags etc. The Mac OS X port is not yet complete. There are still a few issues and much testing needs to be done. I think it would be best for you to wait until we announce the port to begin testing (this should happen within this week). Vikas -- http://vikas.80x25.org/ ___ Gluster-devel mailing list Gluster-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/gluster-devel
