Re: [Gluster-users] Gluster client 32bit
Hi Christian: On Tue, Nov 16, 2010 at 1:34 AM, Christian Fischer wrote: > No statement from the developers about usability of glusterfs client on 32bit > systems. But this was probably discussed in earlier threads. I believe the official comment is that Gluster is not going to support 32-bit systems. However, it doesn't mean that the community cannot support it. If we find bugs and can code up patches, we should still file a bug and submit the patches and hopefully they will be checked into the official repository. Cheers, Bernard ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster client 32bit
Hmm, seems this thread is dead now. That's pity. No statement from the developers about usability of glusterfs client on 32bit systems. But this was probably discussed in earlier threads. I think I'll use NFS with UCARP for the production environment. What about the performance loss if using NFS instead of GlusterFS, any experiences? On Monday 15 November 2010 14:41:23 Christian Fischer wrote: > On Monday 15 November 2010 14:27:34 Stefano Baronio wrote: > > Yes, please, share it with us. > > I've succesfully compiled the rpm packages, but the client is not giving > > any errors when it is not able to connect to a glusterfs share... > > That's normal, the native client exits (as far i've seen) always true. > That is an issue of cleanup_and_exit() if debug is off. > > Christian > > > Thanks > > Stefano > > > > > > 2010/11/13 Dennis Schafroth > > > > > On 12/11/2010, at 18.51, Ken Bigelow wrote: > > > > We have all 32bit server / clients for Gluster. We did have to > > > > compile it from source but so far we have had no problems at all. > > > > > > > > A few things had to be tweaked inside the configuration files like > > > > io thread count and whatnot but in the end it seems to be working > > > > fine from what we can tell. > > > > > > Can you share what you have done? I am running a test on small 32 bit > > > boxes > > > > > > cheers, > > > > > > :-Dennis Schafroth > > > > > > ___ > > > Gluster-users mailing list > > > Gluster-users@gluster.org > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] build problem with MacOS (gcc-4.2.1)
> > > ~/make/glusterfs-3.0.2 > == > [...] > posix.c: In function ‘janitor_walker’: > posix.c:1348: error: ‘FTW_CONTINUE’ undeclared (first use in this function) > > ~/make/glusterfs-3.0.6 > == > server-resolve.c: In function ‘server_resolve_fd’: > server-resolve.c:535: error: ‘EBADFD’ undeclared (first use in this > function) > > ~/make/glusterfs-3.1.0 > == > xlator.c: In function ‘_volume_option_value_validate’: > xlator.c:379: error: ‘FNM_EXTMATCH’ undeclared (first use in this function) > Can you use glusterfs-3.1.1qa7 available @ http://download.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa7.tar.gz, which contains lot of fixes so it gets compiled on Solaris, should work on MacOSX too. Regards, Amar ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
On Mon, 15 Nov 2010 12:17:48 -0800 Craig Carl wrote: > Please don't think we are > not working hard to meet your expectations. Really, Craig, I am not expecting _anything_ for _me_ from glusterfs. I only feel very sorry for an interesting project that gave a great vision but choose featurism over completely solving basic requirements of a fs, not to mention trivial expectations concerning a replication setup - which should have been a true strength. > At a higher level Gluster is changing, and I think improving based > on feedback from the community, our paid subscribers and the storage > industry as a whole. Designing and writing a file system that is used on > thousands of servers in less than 3 years was, and is incredibly > challenging, and expensive. Contrast Gluster with another excellent file > system project, brtfs, which also has paid engineering resources and is > still very experimental [1]. I really don't want to talk about btrfs here, because its problems are unrelated to glusterfs problems. >Our community asked for a couple of things from Gluster 3.1; Well, honestly, whatever the community asked, you managed to create the first project I have seen in more than a decade that is not able to upgrade its older versions because trivial deployment setups have just been _dropped_. I cannot remember ever seeing something like this before. That is really outstanding. > Thanks, > > Craig > > --> > Craig Carl > Senior Systems Engineer > Gluster -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Asymmetrical storage capacity on Glusterfs bricks
On 11/15/2010 12:55 PM, Burnash, James wrote: Has anybody tried asymmetrical storage capacity on Glusterfs bricks? For example, I have my 6 servers configured as mirrored and distributed. Could I add storage to just one mirror pair and make it part of the backend storage for just those servers? I thought I read about this somewhere before on the list, but I can't seem to find it. Thanks, James Burnash, Unix Engineering DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users James - It works out of the box. We do have a best practice around asymmetrical storage servers that will improve performance over just expanding the mount - Ideally each storage brick [1] in a Gluster cluster will be the same size. If bricks differ in size Gluster uses stub files to maintain the Elastic Hashing Algorithm model, the use of stub files [2] for redirection will negatively impact performance. LVM2 is the easiest way to create bricks that don't occupy an entire device. Using LVM2 offers several advantages, by creating more smaller LVs if a file system needs to be fsck'ed the process is faster. Snapshots and clones are also compatible with Gluster. [1] A brick is the combination of a server and a filesystem, ie server1:/dev/vg1/lv1 and server1:/dev/vg1/lv2 are both bricks. [2] http://en.wikipedia.org/wiki/Stub_file Please let me know if you have any other questions. Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Online operations
On 11/15/2010 01:46 PM, Anselm Strauss wrote: But you can not remove a brick without interrupting applications that access data on that brick, right? The documentation says: "Data residing on the brick that you are removing will no longer be accessible at the Gluster mount point." So is there a way of doing a "imbalance" so that data on the brick to be removed will first be allocated in the background and online onto other bricks? What I meant by splitting and merging volumes was motivated by a use case I often see, that is when you decide that certain data in a folder needs to be isolated onto a separate volume. This happens e.g. when a NFS volume needs to be restricted in access only to certain hosts or when one wants to avoid the interruption of an application by another one that runs crazy and fills up all space. Often I see that people first realizing this after the application is already in production or simply when the requirements change. For myself I always see this with ZFS. Data sets are really cheap and easy to do. So I start with one for the whole pool, then later when I see how the application really works I start to split up and set different properties (like quota, compression, etc.) for each data set. But when I do that later on I always have to migrate the data manually to the new mount point and interrupt the application. But I guess this is also a problem of the idea of mount points. How do you move data between mount points without interrupting access to it ... On 11/14/10 03:59, Craig Carl wrote: Anselm - You can remove a brick online, you can't change the type of an existing volume, if you could explain what you what to do with a 'merge' and a 'split' I could give you a better answer, you can 'split' a volume by moving half the data to another volume and 'merge' data by copying all the data from one volume to another, is that what you want to do? Parity based storage in a distributed file system is difficult for several reasons, we are currently investigating some possibilities with erasure coding and will keep everyone up to date on our progress. Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster From: "Anselm Strauss" To: gluster-users@gluster.org Sent: Saturday, November 13, 2010 1:56:03 AM Subject: [Gluster-users] Online operations Hi, I have done some testing with glusterfs on the localhost. I was wondering what all operations you can do online with a glusterfs volume. Is it possible to remove a brick and shrink the volume without taking some data offline? Like a pvmove in Linux LVM that moves all data off a disk before you take it offline? Are the following operations possible to do online? - Change between mirroring and striping - Change the mirror or stripe count - Merge two volumes - Split a volume into two Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster Is there a plan for supporting other redundancy levels that mirror, e.g. RAID 5, 6, ...? Thanks for any ideas, Anselm Strauss ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Anselm - I think you may be looking at older documentation, Gluster 3.1 introduced a migrate command, I think it exactly what you are looking for - http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Migrating_Volumes. I think you identified the problem with 'spliting' a volume, the volume name has to change so there is an application interruption no matter what. Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Online operations
But you can not remove a brick without interrupting applications that access data on that brick, right? The documentation says: "Data residing on the brick that you are removing will no longer be accessible at the Gluster mount point." So is there a way of doing a "imbalance" so that data on the brick to be removed will first be allocated in the background and online onto other bricks? What I meant by splitting and merging volumes was motivated by a use case I often see, that is when you decide that certain data in a folder needs to be isolated onto a separate volume. This happens e.g. when a NFS volume needs to be restricted in access only to certain hosts or when one wants to avoid the interruption of an application by another one that runs crazy and fills up all space. Often I see that people first realizing this after the application is already in production or simply when the requirements change. For myself I always see this with ZFS. Data sets are really cheap and easy to do. So I start with one for the whole pool, then later when I see how the application really works I start to split up and set different properties (like quota, compression, etc.) for each data set. But when I do that later on I always have to migrate the data manually to the new mount point and interrupt the application. But I guess this is also a problem of the idea of mount points. How do you move data between mount points without interrupting access to it ... On 11/14/10 03:59, Craig Carl wrote: > Anselm - > You can remove a brick online, you can't change the type of an existing > volume, if you could explain what you what to do with a 'merge' and a 'split' > I could give you a better answer, you can 'split' a volume by moving half the > data to another volume and 'merge' data by copying all the data from one > volume to another, is that what you want to do? > > Parity based storage in a distributed file system is difficult for several > reasons, we are currently investigating some possibilities with erasure > coding and will keep everyone up to date on our progress. > > > > Thanks, > > Craig > > --> > Craig Carl > Senior Systems Engineer > Gluster > > > > From: "Anselm Strauss" > To: gluster-users@gluster.org > Sent: Saturday, November 13, 2010 1:56:03 AM > Subject: [Gluster-users] Online operations > > Hi, > > I have done some testing with glusterfs on the localhost. I was > wondering what all operations you can do online with a glusterfs volume. > > Is it possible to remove a brick and shrink the volume without taking > some data offline? Like a pvmove in Linux LVM that moves all data off a > disk before you take it offline? > > Are the following operations possible to do online? > > - Change between mirroring and striping > - Change the mirror or stripe count > - Merge two volumes > - Split a volume into two > > Is there a plan for supporting other redundancy levels that mirror, e.g. > RAID 5, 6, ...? > > Thanks for any ideas, > Anselm Strauss > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Asymmetrical storage capacity on Glusterfs bricks
Has anybody tried asymmetrical storage capacity on Glusterfs bricks? For example, I have my 6 servers configured as mirrored and distributed. Could I add storage to just one mirror pair and make it part of the backend storage for just those servers? I thought I read about this somewhere before on the list, but I can't seem to find it. Thanks, James Burnash, Unix Engineering DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
On 11/15/2010 08:04 AM, Stephan von Krawczynski wrote: On Mon, 15 Nov 2010 10:18:28 -0500 Joe Landman wrote: On 11/15/2010 09:47 AM, Stephan von Krawczynski wrote: Stephan - Dovecot has been a challenge in the past. We don't specifically test with it here, if you are interested in using it with Gluster I would suggest testing with 3.1.1, and always keep the index files local, that makes a big difference. Thanks, Craig Well, Craig, I cannot follow your advice as these are 32 bit clients and AFAIK you said 3.1.1 is not expected to be used in such an environment. Really quite a lot of interesting setups for glusterfs turn around mail servers, I judge it to be a major deficiency if the fs cannot be used for such Quick interjection here: We have some customers using Dovecot on our storage units with GlusterFS 3.0.x. There are some issues, usually interactions between dovecot and fuse/glusterfs. Nothing that can't be worked around. Well, a work-around is not the same as "just working". Do you really think that it is no sign of a problem if you need a work-around for a pretty standard usage request? We are seeing strong/growing interest from our customer base in this use case. Well, that means I am right, not? Craig's advice is spot on. purposes. You cannot expect voting for glusterfs if there are other options that have no problems with such a standard setup. I mean is there something more obvious than mailservers for such a fs? Hmmm ... apart from NFS (which isn't a cluster file system), which has a number of its own issues, which other cluster file system are you referring to, that don't have these sorts of issues? Small file and small record performance on any sort of cluster file system is very hard. You have to get it right first, and then work on the performance side later. I am not talking of performance currently (though argueable), I am talking about the shere basic usage. Probably a lot of potential users come from nfs setups and want to make them redundant. And none has ever heard of a fs problem with 32 bit clients (just as an example) ... So this is an obvious problem. "Dovecot has been a challenge in the past", well, and how does the fs currently cope with this challenge? I am no supporter of the idea that fs tuning should be necessary just to make something work at all. For faster performance let there be tuning options, but for general support of a certain environment? I mean, did you ever tune fat,ntfs,extX or the like just to make email work? And don't argue about them not being network related: the simple truth is that this product is only a big hit if it is as easy to deploy as a local fs. That should be the primary goal. Honestly, I got the impression that you're heading away from the mainstream fs usage to very special environments and usage patterns. I feel very sorry about that because 2.X looked very promising. But I did not find a single setup where 3.X could be used at all. While I respect your opinion, I do disagree with it. In our opinion 3.1.x has gotten better than 3.0.x, which was a huge step up from 2.0.x. 2.0.x was something like a filesystem, 3.X is obviously heading to be a storage platform. That makes a big difference. And I'd say it did not get really better in general comparing apples to apples. glusterfs 2.0.x is a lot closer to a useable filesystem (lets say on linux boxes) than glusterfs 3.X is to netapp or emc storage platforms. There is nothing comparable to glusterfs 2.0.X on its boxes whereas one cannot really choose glusterfs storage in comparison to netapp. I mean you're trying to enter the wrong league because the big players will just crash you. Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Stephan - You made some very good points, thank you for your feedback. I'd like to address your Dovecot question directly, then some of your broader points. Around email servers our challenge there is very specific to Dovecot. Organizations regularly run Zimbra, Sendmail, Postfix, James, GroupWise, etc using Gluster with no problems and excellent performance. We also seem to have more community users using Gluster with Dovecot than I suspected. We do have at least one paid support subscription for a group using Gluster with Dovecot, we are actively working with them to improve performance and stability. If you take a look at bugs.gluster.com we have a P5 critical bug (#956) opened for Dovecot support, engineers have been assigned and we are actively working on a solution. Because Gluster is free as in beer as we patch Gluster t
Re: [Gluster-users] GlusterFS on mailservers
We are using glusterFS for our cloud mail servers under 32bit with no problems at all and the cluster is hit fairly hard on a regular basis. I think what Craig meant by a workaround was not so much for GlusterFS but for the Dovecot Imap setup. Using Dovecot with NFS will give the same problems that GlusterFS does so it might not be fair to blame GlusterFS for the imap problems that some see. By adding dotlock_use_excl = no, mail_nfs_storage = yes, mail_nfs_index = yes to our dovecot.conf file we have not had any problems to date. Our versions in use of GlusterFS is 3.0.5 and dovecot is 1.2.10 running on CentOS 5.5. Hope this helps. On Mon, 15 Nov 2010 10:18:28 -0500 Joe Landman wrote: On 11/15/2010 09:47 AM, Stephan von Krawczynski wrote: Stephan - Dovecot has been a challenge in the past. We don't specifically test with it here, if you are interested in using it with Gluster I would suggest testing with 3.1.1, and always keep the index files local, that makes a big difference. Thanks, Craig Well, Craig, I cannot follow your advice as these are 32 bit clients and AFAIK you said 3.1.1 is not expected to be used in such an environment. Really quite a lot of interesting setups for glusterfs turn around mail servers, I judge it to be a major deficiency if the fs cannot be used for such Quick interjection here: We have some customers using Dovecot on our storage units with GlusterFS 3.0.x. There are some issues, usually interactions between dovecot and fuse/glusterfs. Nothing that can't be worked around. Well, a work-around is not the same as "just working". Do you really think that it is no sign of a problem if you need a work-around for a pretty standard usage request? We are seeing strong/growing interest from our customer base in this use case. Well, that means I am right, not? Craig's advice is spot on. purposes. You cannot expect voting for glusterfs if there are other options that have no problems with such a standard setup. I mean is there something more obvious than mailservers for such a fs? Hmmm ... apart from NFS (which isn't a cluster file system), which has a number of its own issues, which other cluster file system are you referring to, that don't have these sorts of issues? Small file and small record performance on any sort of cluster file system is very hard. You have to get it right first, and then work on the performance side later. I am not talking of performance currently (though argueable), I am talking about the shere basic usage. Probably a lot of potential users come from nfs setups and want to make them redundant. And none has ever heard of a fs problem with 32 bit clients (just as an example) ... So this is an obvious problem. "Dovecot has been a challenge in the past", well, and how does the fs currently cope with this challenge? I am no supporter of the idea that fs tuning should be necessary just to make something work at all. For faster performance let there be tuning options, but for general support of a certain environment? I mean, did you ever tune fat,ntfs,extX or the like just to make email work? And don't argue about them not being network related: the simple truth is that this product is only a big hit if it is as easy to deploy as a local fs. That should be the primary goal. Honestly, I got the impression that you're heading away from the mainstream fs usage to very special environments and usage patterns. I feel very sorry about that because 2.X looked very promising. But I did not find a single setup where 3.X could be used at all. While I respect your opinion, I do disagree with it. In our opinion 3.1.x has gotten better than 3.0.x, which was a huge step up from 2.0.x. 2.0.x was something like a filesystem, 3.X is obviously heading to be a storage platform. That makes a big difference. And I'd say it did not get really better in general comparing apples to apples. glusterfs 2.0.x is a lot closer to a useable filesystem (lets say on linux boxes) than glusterfs 3.X is to netapp or emc storage platforms. There is nothing comparable to glusterfs 2.0.X on its boxes whereas one cannot really choose glusterfs storage in comparison to netapp. I mean you're trying to enter the wrong league because the big players will just crash you. Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Help Installing Storage Platform on Intel Atom hardware
Hi Tariq, Can you remove 'rhgb quiet' options from boot option? This will give boot message on console. You can press Tab key during boot menu to get access to boot parameters. Thanks, Regards, Bala Tariq Islam wrote: Hey guys, I'm having issues installing gluster storage platform. I've tried versions 3.0.4/5 and 3.1, my hardware: Intel D530 atom processor (1.8Ghz, dual core, 2 threads/core), 4GB ram, SATA. When I boot using the flash drive, the initial boot screen loads up, Gluster starts booting with the progress bar at the bottom hitting 100%, then the screen goes blank. Keyboard is unresponsive. If someone could help with this that would be awesome. Thanks. Tariq ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
On Mon, 15 Nov 2010 10:18:28 -0500 Joe Landman wrote: > On 11/15/2010 09:47 AM, Stephan von Krawczynski wrote: > > >> Stephan - > >> Dovecot has been a challenge in the past. We don't specifically test > >> with it here, if you are interested in using it with Gluster I would > >> suggest testing with 3.1.1, and always keep the index files local, that > >> makes a big difference. > >> > >> Thanks, > >> > >> Craig > > > > Well, Craig, I cannot follow your advice as these are 32 bit clients and > > AFAIK > > you said 3.1.1 is not expected to be used in such an environment. > > Really quite a lot of interesting setups for glusterfs turn around mail > > servers, I judge it to be a major deficiency if the fs cannot be used for > > such > > Quick interjection here: We have some customers using Dovecot on our > storage units with GlusterFS 3.0.x. There are some issues, usually > interactions between dovecot and fuse/glusterfs. Nothing that can't be > worked around. Well, a work-around is not the same as "just working". Do you really think that it is no sign of a problem if you need a work-around for a pretty standard usage request? > We are seeing strong/growing interest from our customer > base in this use case. Well, that means I am right, not? > Craig's advice is spot on. > > > purposes. You cannot expect voting for glusterfs if there are other options > > that have no problems with such a standard setup. I mean is there something > > more obvious than mailservers for such a fs? > > Hmmm ... apart from NFS (which isn't a cluster file system), which has a > number of its own issues, which other cluster file system are you > referring to, that don't have these sorts of issues? Small file and > small record performance on any sort of cluster file system is very > hard. You have to get it right first, and then work on the performance > side later. I am not talking of performance currently (though argueable), I am talking about the shere basic usage. Probably a lot of potential users come from nfs setups and want to make them redundant. And none has ever heard of a fs problem with 32 bit clients (just as an example) ... So this is an obvious problem. "Dovecot has been a challenge in the past", well, and how does the fs currently cope with this challenge? I am no supporter of the idea that fs tuning should be necessary just to make something work at all. For faster performance let there be tuning options, but for general support of a certain environment? I mean, did you ever tune fat,ntfs,extX or the like just to make email work? And don't argue about them not being network related: the simple truth is that this product is only a big hit if it is as easy to deploy as a local fs. That should be the primary goal. > > Honestly, I got the impression that you're heading away from the mainstream > > fs > > usage to very special environments and usage patterns. > > I feel very sorry about that because 2.X looked very promising. But I did > > not > > find a single setup where 3.X could be used at all. > > While I respect your opinion, I do disagree with it. In our opinion > 3.1.x has gotten better than 3.0.x, which was a huge step up from 2.0.x. 2.0.x was something like a filesystem, 3.X is obviously heading to be a storage platform. That makes a big difference. And I'd say it did not get really better in general comparing apples to apples. glusterfs 2.0.x is a lot closer to a useable filesystem (lets say on linux boxes) than glusterfs 3.X is to netapp or emc storage platforms. There is nothing comparable to glusterfs 2.0.X on its boxes whereas one cannot really choose glusterfs storage in comparison to netapp. I mean you're trying to enter the wrong league because the big players will just crash you. > Regards, > > Joe > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics, Inc. > email: land...@scalableinformatics.com > web : http://scalableinformatics.com > http://scalableinformatics.com/jackrabbit > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
I 2nd.. On Nov 15, 2010, at 9:18 AM, Joe Landman wrote: > While I respect your opinion, I do disagree with it. In our opinion 3.1.x has > gotten better than 3.0.x, which was a huge step up from 2.0.x. Luis E. Cerezo blog: http://www.luiscerezo.org fotofun: http://www.flickr.com/photos/luiscerezo/ twitter: http://twitter.com/luiscerezo/ Voice: +1 412 223 7396 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
On 11/15/2010 09:47 AM, Stephan von Krawczynski wrote: Stephan - Dovecot has been a challenge in the past. We don't specifically test with it here, if you are interested in using it with Gluster I would suggest testing with 3.1.1, and always keep the index files local, that makes a big difference. Thanks, Craig Well, Craig, I cannot follow your advice as these are 32 bit clients and AFAIK you said 3.1.1 is not expected to be used in such an environment. Really quite a lot of interesting setups for glusterfs turn around mail servers, I judge it to be a major deficiency if the fs cannot be used for such Quick interjection here: We have some customers using Dovecot on our storage units with GlusterFS 3.0.x. There are some issues, usually interactions between dovecot and fuse/glusterfs. Nothing that can't be worked around. We are seeing strong/growing interest from our customer base in this use case. Craig's advice is spot on. purposes. You cannot expect voting for glusterfs if there are other options that have no problems with such a standard setup. I mean is there something more obvious than mailservers for such a fs? Hmmm ... apart from NFS (which isn't a cluster file system), which has a number of its own issues, which other cluster file system are you referring to, that don't have these sorts of issues? Small file and small record performance on any sort of cluster file system is very hard. You have to get it right first, and then work on the performance side later. Honestly, I got the impression that you're heading away from the mainstream fs usage to very special environments and usage patterns. I feel very sorry about that because 2.X looked very promising. But I did not find a single setup where 3.X could be used at all. While I respect your opinion, I do disagree with it. In our opinion 3.1.x has gotten better than 3.0.x, which was a huge step up from 2.0.x. Regards, Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster with xfs
On Mon, Nov 15, 2010 at 5:01 PM, Craig Carl wrote: > From a previous XFS & Gluster discussion - > > "The reason we usually recommend Ext3 instead of XFS is because the > implementation of extended attributes in Ext3 is significantly faster than > in XFS. GlusterFS makes use of extended attributes quite a bit, especially > the replicate translator. In an environment with lots of small files and > many creation/deletion operations, using XFS with replication will be slower > than Ext3. > > > Thanks, > > Craig > > --> > Craig Carl > Senior Systems Engineer > Gluster Carl, what would be best in terms of performance for a smal file environment, like for example web hosting & email when setting up a replicated Gluster environment? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster with xfs
On 11/15/2010 06:46 AM, David Lloyd wrote: Thanks for that. With the hardware we've got we're hoping to get an order of magnitude more throughput than that though. We have: 4 nodes 12 sata disks hardware raid in each node 10Gbit ethernet I'm getting over 1GByte/s reads and writes on the xfs filesystems locally. typical filesize will be 10MByte, but lots of bigger and smaller files too. there will be about 50 clients, all mounting glusterfs. 'Introduction to Gluster' also ominously states: 'There are known challenges with other filesystems' (which would include xfs). Can anyone expand on that? Cheers David On Mon, Nov 15, 2010 at 11:09 AM, Liam Slusser wrote: We run two somewhat large gluster clusters in production on xfs with great success. I had to go with xfs as ext4 doesn't support large enough file systems. Make sure you mount your xfs partitions with 64bit inode support and use only 64bit OS's. I'm still running 2.0.9 however the performance is pretty good. We use ours to store media for our website and with our smaller two server four brick 60tb cluster I can easily push 800mbit of http traffic with an average object size of 2-3megs. Not bad for a bunch of slow sata disks! Liam David - From a previous XFS & Gluster discussion - "The reason we usually recommend Ext3 instead of XFS is because the implementation of extended attributes in Ext3 is significantly faster than in XFS. GlusterFS makes use of extended attributes quite a bit, especially the replicate translator. In an environment with lots of small files and many creation/deletion operations, using XFS with replication will be slower than Ext3. However, if your workload consists of mostly large files and relatively fewer create/delete operations, you might find that the performance XFS delivers is acceptable. We have many successful deployments that use XFS in just this way." Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster with xfs
On Mon, Nov 15, 2010 at 1:09 PM, Liam Slusser wrote: > We run two somewhat large gluster clusters in production on xfs with great > success. I had to go with xfs as ext4 doesn't support large enough file > systems. Make sure you mount your xfs partitions with 64bit inode support > and use only 64bit OS's. > > I'm still running 2.0.9 however the performance is pretty good. We use ours > to store media for our website and with our smaller two server four brick > 60tb cluster I can easily push 800mbit of http traffic with an average > object size of 2-3megs. Not bad for a bunch of slow sata disks! > > Liam > On Nov 15, 2010 2:53 AM, "David Lloyd" > wrote: >> Hello, >> >> We're starting to set up a 4 node gluster system. I'm currently trying >> to decide on the low-level options, including what filesystem to use. >> >> For various reasons I would be more comfortable with XFS over ext4, >> but I read in the 'Introduction to Gluster' that 'XFS (can be slow)'. >> >> I haven't found any other details about this, and wondered if anyone >> has more information or experience of using gluster with XFS. Or if >> anything has changed with 3.1. We don't want it to be slow, and I'm >> happy enough using ext4 if necessary, but just wanted to see what >> others thought first. >> >> Thanks >> David >> >> -- >> David Lloyd >> V Consultants >> www.v-consultants.co.uk >> ___ Lian, as matter of interest, what kind of setup do you have? i.e what servers / hard drive combination or configuration / hardware or software RAID / 1GB or 10GB network, etc? -- Kind Regards Rudi Ahlers SoftDux Website: http://www.SoftDux.com Technical Blog: http://Blog.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
We use Gluster v3.0.6 on Gnu/linux Debian 64bits Squeeze, with dovecot 1.2.15. plugin quota, we do not put these options in dovecot configuration, all works fine. bst regards. On Mon, 15 Nov 2010 13:57:43 +0100, Stephan von Krawczynski wrote: Hi all, I just read this one on the dovecot web: --- FUSE / GlusterFS FUSE caches dentries and file attributes internally. If you're using multiple GlusterFS clients to access the same mailboxes, you're going to have problems. Worst of these problems can be avoided by using NFS cache flushes, which just happen to work with FUSE as well: mail_nfs_index = yes mail_nfs_storage = yes These probably don't work perfectly. Can someone comment on that? Does anybody use glusterfs as a storage for mailboxes/mailfolders ? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
On Mon, 15 Nov 2010 06:25:23 -0800 Craig Carl wrote: > On 11/15/2010 04:57 AM, Stephan von Krawczynski wrote: > > Hi all, > > > > I just read this one on the dovecot web: > > --- > > FUSE / GlusterFS > > > > FUSE caches dentries and file attributes internally. If you're using > > multiple > > GlusterFS clients to access the same mailboxes, you're going to have > > problems. > > Worst of these problems can be avoided by using NFS cache flushes, which > > just > > happen to work with FUSE as well: > > > > mail_nfs_index = yes > > mail_nfs_storage = yes > > > > These probably don't work perfectly. > > > > > > Can someone comment on that? Does anybody use glusterfs as a storage for > > mailboxes/mailfolders ? > > > Stephan - > Dovecot has been a challenge in the past. We don't specifically test > with it here, if you are interested in using it with Gluster I would > suggest testing with 3.1.1, and always keep the index files local, that > makes a big difference. > > Thanks, > > Craig Well, Craig, I cannot follow your advice as these are 32 bit clients and AFAIK you said 3.1.1 is not expected to be used in such an environment. Really quite a lot of interesting setups for glusterfs turn around mail servers, I judge it to be a major deficiency if the fs cannot be used for such purposes. You cannot expect voting for glusterfs if there are other options that have no problems with such a standard setup. I mean is there something more obvious than mailservers for such a fs? Honestly, I got the impression that you're heading away from the mainstream fs usage to very special environments and usage patterns. I feel very sorry about that because 2.X looked very promising. But I did not find a single setup where 3.X could be used at all. > --> > Craig Carl > Senior Systems Engineer > Gluster -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster with xfs
Thanks for that. With the hardware we've got we're hoping to get an order of magnitude more throughput than that though. We have: 4 nodes 12 sata disks hardware raid in each node 10Gbit ethernet I'm getting over 1GByte/s reads and writes on the xfs filesystems locally. typical filesize will be 10MByte, but lots of bigger and smaller files too. there will be about 50 clients, all mounting glusterfs. 'Introduction to Gluster' also ominously states: 'There are known challenges with other filesystems' (which would include xfs). Can anyone expand on that? Cheers David On Mon, Nov 15, 2010 at 11:09 AM, Liam Slusser wrote: > We run two somewhat large gluster clusters in production on xfs with great > success. I had to go with xfs as ext4 doesn't support large enough file > systems. Make sure you mount your xfs partitions with 64bit inode support > and use only 64bit OS's. > > I'm still running 2.0.9 however the performance is pretty good. We use ours > to store media for our website and with our smaller two server four brick > 60tb cluster I can easily push 800mbit of http traffic with an average > object size of 2-3megs. Not bad for a bunch of slow sata disks! > > Liam > -- David Lloyd V Consultants www.v-consultants.co.uk ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS on mailservers
On 11/15/2010 04:57 AM, Stephan von Krawczynski wrote: Hi all, I just read this one on the dovecot web: --- FUSE / GlusterFS FUSE caches dentries and file attributes internally. If you're using multiple GlusterFS clients to access the same mailboxes, you're going to have problems. Worst of these problems can be avoided by using NFS cache flushes, which just happen to work with FUSE as well: mail_nfs_index = yes mail_nfs_storage = yes These probably don't work perfectly. Can someone comment on that? Does anybody use glusterfs as a storage for mailboxes/mailfolders ? Stephan - Dovecot has been a challenge in the past. We don't specifically test with it here, if you are interested in using it with Gluster I would suggest testing with 3.1.1, and always keep the index files local, that makes a big difference. Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster client 32bit
On Monday 15 November 2010 14:27:34 Stefano Baronio wrote: > Yes, please, share it with us. > I've succesfully compiled the rpm packages, but the client is not giving > any errors when it is not able to connect to a glusterfs share... That's normal, the native client exits (as far i've seen) always true. That is an issue of cleanup_and_exit() if debug is off. Christian > > Thanks > Stefano > > > 2010/11/13 Dennis Schafroth > > > On 12/11/2010, at 18.51, Ken Bigelow wrote: > > > We have all 32bit server / clients for Gluster. We did have to compile > > > it from source but so far we have had no problems at all. > > > > > > A few things had to be tweaked inside the configuration files like > > > io thread count and whatnot but in the end it seems to be working fine > > > from what we can tell. > > > > Can you share what you have done? I am running a test on small 32 bit > > boxes > > > > cheers, > > > > :-Dennis Schafroth > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster client 32bit
Yes, please, share it with us. I've succesfully compiled the rpm packages, but the client is not giving any errors when it is not able to connect to a glusterfs share... Thanks Stefano 2010/11/13 Dennis Schafroth > On 12/11/2010, at 18.51, Ken Bigelow wrote: > > We have all 32bit server / clients for Gluster. We did have to compile > > it from source but so far we have had no problems at all. > > > > A few things had to be tweaked inside the configuration files like > > io thread count and whatnot but in the end it seems to be working fine > > from what we can tell. > > Can you share what you have done? I am running a test on small 32 bit boxes > > cheers, > :-Dennis Schafroth > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance tragically slow
have you tried larger files? I have seen a note somewhere that refers to tons of itty bitty files, it even cites the kernel source as an example. I can't seem to find it. Could you try larger than 1k files? -luis On Nov 10, 2010, at 8:23 PM, Pavel Snajdr wrote: > Hello, > > I imagine you get this kind of messages all the time, but just in case: > > I have setup with 2 storage servers with debian package of gluster - version > 3.1.0 > > They are connected by dedicated 1 gigabit ethernet cards. > > I've set up simple replicated storage with 2 replicas and transport over TCP > (just followed the how to on the wiki with obvious changes). > > Here goes my problem: > > If I try to copy small files (i.e. extract kernel source) I get a horrible > results: > > praha-storage2:/mnt/test# time tar xf linux-2.6.26.8.tar.bz2 > > > real 15m19.825s > user 0m13.989s > sys 0m5.152s > > likewise when rsyncing OpenVZ VPSes - I just can't get over 2MB/s in syncing. > > I've monitored all resources - CPU load, network, disk I/O - they are all > used up to 0.0nothing %. > > Network latency is about 0.11 ms all the time. > > Is this normal, or am I doing something wrong? > > Please help. I am frustrated :( > > -- > S pozdravem / Best Regards > > Pavel Šnajdr > +420 720 107 791 > > http://vpsfree.cz > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > Luis E. Cerezo blog: http://www.luiscerezo.org fotofun: http://www.flickr.com/photos/luiscerezo/ twitter: http://twitter.com/luiscerezo/ Voice: +1 412 223 7396 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster client 32bit
On 12/11/2010, at 18.51, Ken Bigelow wrote: > We have all 32bit server / clients for Gluster. We did have to compile > it from source but so far we have had no problems at all. > > A few things had to be tweaked inside the configuration files like > io thread count and whatnot but in the end it seems to be working fine > from what we can tell. Can you share what you have done? I am running a test on small 32 bit boxes cheers, :-Dennis Schafroth ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Help Installing Storage Platform on Intel Atom hardware
Hey guys, I'm having issues installing gluster storage platform. I've tried versions 3.0.4/5 and 3.1, my hardware: Intel D530 atom processor (1.8Ghz, dual core, 2 threads/core), 4GB ram, SATA. When I boot using the flash drive, the initial boot screen loads up, Gluster starts booting with the progress bar at the bottom hitting 100%, then the screen goes blank. Keyboard is unresponsive. If someone could help with this that would be awesome. Thanks. Tariq ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] about Gluster initial server
Hi all, It seems the initial server failed all the services down. Is there any way to protect the initial server? Bill ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Possible split-brain
Hi all, I have 4 glusterd servers running a single glusterfs volume. The volume was created using the gluster command line, with no changes from default. The same machines all mount the volume using the native glusterfs client: [r...@localhost ~]# gluster volume create datastore replica 2 transport tcp 192.168.253.1:/glusterfs/primary 192.168.253.3:/glusterfs/secondary 192.168.253.2:/glusterfs/primary 192.168.253.4:/glusterfs/secondary 192.168.253.3:/glusterfs/primary 192.168.253.1:/glusterfs/secondary 192.168.253.4:/glusterfs/primary 192.168.253.2:/glusterfs/secondary [r...@localhost ~]# cat /etc/fstab ... /dev/cciss/c0d0p6 /glusterfs/primary ext4defaults,noatime 1 2 /dev/cciss/c0d1p6 /glusterfs/secondaryext4defaults,noatime 1 2 192.168.253.1:/datastore /mnt/datastore glusterfs defaults,_netdev 0 0 [r...@localhost ~]# gluster volume info Volume Name: datastore Type: Distributed-Replicate Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 192.168.253.1:/glusterfs/primary Brick2: 192.168.253.3:/glusterfs/secondary Brick3: 192.168.253.2:/glusterfs/primary Brick4: 192.168.253.4:/glusterfs/secondary Brick5: 192.168.253.3:/glusterfs/primary Brick6: 192.168.253.1:/glusterfs/secondary Brick7: 192.168.253.4:/glusterfs/primary Brick8: 192.168.253.2:/glusterfs/secondary The platform is not currently running production data and I have been testing the redundancy of the setup (pulling cables etc.). All my servers are now logging the following messages every 1 minute or so: [2010-11-11 14:18:49.636327] I [afr-common.c:672:afr_lookup_done] datastore-replicate-0: split brain detected during lookup of /. [2010-11-11 14:18:49.636388] I [afr-common.c:716:afr_lookup_done] datastore-replicate-0: background meta-data data self-heal triggered. path: / [2010-11-11 14:18:49.636863] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] datastore-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2010-11-11 14:18:49.637080] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] datastore-replicate-0: background meta-data data self-heal completed on / [2010-11-11 14:18:49.637561] I [afr-common.c:672:afr_lookup_done] datastore-replicate-0: split brain detected during lookup of /. [2010-11-11 14:18:49.637588] I [afr-common.c:716:afr_lookup_done] datastore-replicate-0: background meta-data data self-heal triggered. path: / [2010-11-11 14:18:49.638064] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] datastore-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2010-11-11 14:18:49.638265] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] datastore-replicate-0: background meta-data data self-heal completed on / Can anyone tell me what I need to do to fix this? Thanks, Aaron ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] build problem with MacOS (gcc-4.2.1)
Hello, as described @: http://www.gluster.com/community/documentation/index.php/GlusterFS_on_MAC_OS_X I tried to compile glusterfs on MacOS 64.bit Snow Leopard with Xcode: 3.2.4 == MacFUSE: 2.0.3 gcc: i686-apple-darwin10-gcc-4.2.1 == ...unfortunately I have no option to build (as described) with gcc 4.0.1 but just for your records... below are my results with 3 versions hope it helps cheers .v. ~/make/glusterfs-3.0.2 == [...] posix.c: In function ‘janitor_walker’: posix.c:1348: error: ‘FTW_CONTINUE’ undeclared (first use in this function) posix.c:1348: error: (Each undeclared identifier is reported only once posix.c:1348: error: for each function it appears in.) posix.c: In function ‘posix_fsync’: posix.c:2863: warning: implicit declaration of function ‘fdatasync’ make[5]: *** [posix.lo] Error 1 make[4]: *** [all-recursive] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 ~/make/glusterfs-3.0.6 == [...] server-resolve.c: In function ‘server_resolve_fd’: server-resolve.c:535: error: ‘EBADFD’ undeclared (first use in this function) server-resolve.c:535: error: (Each undeclared identifier is reported only once server-resolve.c:535: error: for each function it appears in.) make[5]: *** [server-resolve.lo] Error 1 make[4]: *** [all-recursive] Error 1 make[3]: *** [all-recursive] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 ~/make/glusterfs-3.1.0 == [...] xlator.c: In function ‘_volume_option_value_validate’: xlator.c:379: error: ‘FNM_EXTMATCH’ undeclared (first use in this function) xlator.c:379: error: (Each undeclared identifier is reported only once xlator.c:379: error: for each function it appears in.) make[3]: *** [libglusterfs_la-xlator.lo] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 . ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Write performance tragically slow
Hello, I imagine you get this kind of messages all the time, but just in case: I have setup with 2 storage servers with debian package of gluster - version 3.1.0 They are connected by dedicated 1 gigabit ethernet cards. I've set up simple replicated storage with 2 replicas and transport over TCP (just followed the how to on the wiki with obvious changes). Here goes my problem: If I try to copy small files (i.e. extract kernel source) I get a horrible results: praha-storage2:/mnt/test# time tar xf linux-2.6.26.8.tar.bz2 real 15m19.825s user 0m13.989s sys 0m5.152s likewise when rsyncing OpenVZ VPSes - I just can't get over 2MB/s in syncing. I've monitored all resources - CPU load, network, disk I/O - they are all used up to 0.0nothing %. Network latency is about 0.11 ms all the time. Is this normal, or am I doing something wrong? Please help. I am frustrated :( -- S pozdravem / Best Regards Pavel Šnajdr +420 720 107 791 http://vpsfree.cz ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] ***SPAM*** Re: filling gluster cluster with large file doesn't crash the system?!
Rudi Ahlers wrote: On Tue, Nov 9, 2010 at 8:46 PM, Matt Hodson wrote: I should also note that on this non-production test rig the block size on both bricks is 1KB (1024) so the theoretical file size limit is 16GB. so how then did i get a file of 200GB? -matt On Nov 9, 2010, at 10:34 AM, Matt Hodson wrote: craig et al, I have a 2 brick distributed 283GB gluster cluster on CentoOS 5. we nfs mounted the cluster from a 3rd machine and wrote random junk to a file. i watched the file grow to 200GB on the cluster when it appeared to stop. however the machine writing to the file still lists the file as growing. it's now at over 320GB. what's going on? -matt --- Matt Hodson Scientific Customer Support, Geospiza (206) 633-4403, Ext. 111 http://www.geospiza.com How, exactly, did you fill the file with junk? #perl -e 'print rand while 1' > y.out & ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] GlusterFS on mailservers
Hi all, I just read this one on the dovecot web: --- FUSE / GlusterFS FUSE caches dentries and file attributes internally. If you're using multiple GlusterFS clients to access the same mailboxes, you're going to have problems. Worst of these problems can be avoided by using NFS cache flushes, which just happen to work with FUSE as well: mail_nfs_index = yes mail_nfs_storage = yes These probably don't work perfectly. Can someone comment on that? Does anybody use glusterfs as a storage for mailboxes/mailfolders ? -- Regards, Stephan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setup trouble without DNS
On 11/15/2010 04:20 AM, Uwe Kastens wrote: Graig, Thanks a lot. I will try it today. It would be great to build as deb again. Is there any place I can download the debian build files which with the http://download.gluster.com/pub/gluster/glusterfs/qa-releases/debs/glusterfs_3.1.0qa40-1_amd64.deb is build? BR Uwe Am 15.11.2010 um 11:02 schrieb Craig Carl: Uwe - There is a name resolution bug in version 3.1, that is the issue you are seeing. If you would like to continue testing the most recent QA version, 3.1.1qa6 resolves the issue, you can get it here - http://download.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa6.tar.gz. Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.c...@gmail.com From: "Uwe Kastens" To: gluster-users@gluster.org Sent: Monday, November 15, 2010 1:19:32 AM Subject: [Gluster-users] setup trouble without DNS Hello, I tried to setup an testing environment inside a VMWare Workstation with 3 ubuntu 10.10 server. I can add the probes by hostname, which is configured in /etc/hosts r...@ubuntu01:~# gluster peer statusNumber of Peers: 2 Hostname: ubuntu02 Uuid: b2dc0e78-f91a-4fa2-9faa-ce7d81a281fe State: Peer in Cluster (Connected) Hostname: ubuntu03 Uuid: f127902d-7f46-4438-bfeb-5ce25821ab15 State: Peer in Cluster (Connected) r...@ubuntu01:~# gluster volume create store ubuntu02:/var/store ubuntu03:/var/store Creation of volume store has been unsuccessful But working with the IP Adresses is working r...@ubuntu01:~# gluster volume create store 172.16.101.202:/var/store 172.16.101.203:/var/store Creation of volume store has been successful BR Uwe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users Uwe - We generally only build packages for GA releases, 3.1.1 will be out this week, the .deb will be published here - http://download.gluster.com/pub/gluster/glusterfs/3.1/LATEST/Ubuntu/ - when the GA version ships. Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setup trouble without DNS
Graig, Thanks a lot. I will try it today. It would be great to build as deb again. Is there any place I can download the debian build files which with the http://download.gluster.com/pub/gluster/glusterfs/qa-releases/debs/glusterfs_3.1.0qa40-1_amd64.deb is build? BR Uwe Am 15.11.2010 um 11:02 schrieb Craig Carl: > Uwe - > There is a name resolution bug in version 3.1, that is the issue you are > seeing. If you would like to continue testing the most recent QA version, > 3.1.1qa6 resolves the issue, you can get it here - > http://download.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa6.tar.gz. > > Thanks, > Craig > > --> > Craig Carl > Gluster, Inc. > Cell - (408) 829-9953 (California, USA) > Gtalk - craig.c...@gmail.com > > > From: "Uwe Kastens" > To: gluster-users@gluster.org > Sent: Monday, November 15, 2010 1:19:32 AM > Subject: [Gluster-users] setup trouble without DNS > > Hello, > > I tried to setup an testing environment inside a VMWare Workstation with 3 > ubuntu 10.10 server. > > I can add the probes by hostname, which is configured in /etc/hosts > > r...@ubuntu01:~# gluster peer statusNumber of Peers: 2 > > Hostname: ubuntu02 > Uuid: b2dc0e78-f91a-4fa2-9faa-ce7d81a281fe > State: Peer in Cluster (Connected) > > Hostname: ubuntu03 > Uuid: f127902d-7f46-4438-bfeb-5ce25821ab15 > State: Peer in Cluster (Connected) > > > r...@ubuntu01:~# gluster volume create store ubuntu02:/var/store > ubuntu03:/var/store > Creation of volume store has been unsuccessful > > > But working with the IP Adresses is working > r...@ubuntu01:~# gluster volume create store 172.16.101.202:/var/store > 172.16.101.203:/var/store > Creation of volume store has been successful > > BR > > Uwe > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster with xfs
We run two somewhat large gluster clusters in production on xfs with great success. I had to go with xfs as ext4 doesn't support large enough file systems. Make sure you mount your xfs partitions with 64bit inode support and use only 64bit OS's. I'm still running 2.0.9 however the performance is pretty good. We use ours to store media for our website and with our smaller two server four brick 60tb cluster I can easily push 800mbit of http traffic with an average object size of 2-3megs. Not bad for a bunch of slow sata disks! Liam On Nov 15, 2010 2:53 AM, "David Lloyd" wrote: > Hello, > > We're starting to set up a 4 node gluster system. I'm currently trying > to decide on the low-level options, including what filesystem to use. > > For various reasons I would be more comfortable with XFS over ext4, > but I read in the 'Introduction to Gluster' that 'XFS (can be slow)'. > > I haven't found any other details about this, and wondered if anyone > has more information or experience of using gluster with XFS. Or if > anything has changed with 3.1. We don't want it to be slow, and I'm > happy enough using ext4 if necessary, but just wanted to see what > others thought first. > > Thanks > David > > -- > David Lloyd > V Consultants > www.v-consultants.co.uk > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] gluster with xfs
Hello, We're starting to set up a 4 node gluster system. I'm currently trying to decide on the low-level options, including what filesystem to use. For various reasons I would be more comfortable with XFS over ext4, but I read in the 'Introduction to Gluster' that 'XFS (can be slow)'. I haven't found any other details about this, and wondered if anyone has more information or experience of using gluster with XFS. Or if anything has changed with 3.1. We don't want it to be slow, and I'm happy enough using ext4 if necessary, but just wanted to see what others thought first. Thanks David -- David Lloyd V Consultants www.v-consultants.co.uk ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setup trouble without DNS
On 15/11/2010 09:21, gluster-users-requ...@gluster.org wrote: Message: 6 Date: Mon, 15 Nov 2010 10:19:32 +0100 From: Uwe Kastens Subject: [Gluster-users] setup trouble without DNS To:gluster-users@gluster.org Message-ID:<3c0a0433-59c9-4cab-acab-71f7ece69...@kiste.org> Content-Type: text/plain; charset=us-ascii Hello, I tried to setup an testing environment inside a VMWare Workstation with 3 ubuntu 10.10 server. I can add the probes by hostname, which is configured in /etc/hosts r...@ubuntu01:~# gluster peer statusNumber of Peers: 2 Hostname: ubuntu02 Uuid: b2dc0e78-f91a-4fa2-9faa-ce7d81a281fe State: Peer in Cluster (Connected) Hostname: ubuntu03 Uuid: f127902d-7f46-4438-bfeb-5ce25821ab15 State: Peer in Cluster (Connected) r...@ubuntu01:~# gluster volume create store ubuntu02:/var/store ubuntu03:/var/store Creation of volume store has been unsuccessful But working with the IP Adresses is working r...@ubuntu01:~# gluster volume create store 172.16.101.202:/var/store 172.16.101.203:/var/store Creation of volume store has been successful BR Uwe This happens to me too, except that my host names are registered with the DNS instead of /etc/hosts. I wasn't sure if the IP addresses were supposed to be used instead of the host names, which is why I didn't report this behaviour as a possible bug. It is a bit inconvenient but certainly not a show stopper. -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setup trouble without DNS
Uwe - There is a name resolution bug in version 3.1, that is the issue you are seeing. If you would like to continue testing the most recent QA version, 3.1.1qa6 resolves the issue, you can get it here - http://download.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa6.tar.gz . Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.c...@gmail.com From: "Uwe Kastens" To: gluster-users@gluster.org Sent: Monday, November 15, 2010 1:19:32 AM Subject: [Gluster-users] setup trouble without DNS Hello, I tried to setup an testing environment inside a VMWare Workstation with 3 ubuntu 10.10 server. I can add the probes by hostname, which is configured in /etc/hosts r...@ubuntu01:~# gluster peer statusNumber of Peers: 2 Hostname: ubuntu02 Uuid: b2dc0e78-f91a-4fa2-9faa-ce7d81a281fe State: Peer in Cluster (Connected) Hostname: ubuntu03 Uuid: f127902d-7f46-4438-bfeb-5ce25821ab15 State: Peer in Cluster (Connected) r...@ubuntu01:~# gluster volume create store ubuntu02:/var/store ubuntu03:/var/store Creation of volume store has been unsuccessful But working with the IP Adresses is working r...@ubuntu01:~# gluster volume create store 172.16.101.202:/var/store 172.16.101.203:/var/store Creation of volume store has been successful BR Uwe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setvolume failed (Stale NFS file handle) when volfile is changed
Mohan - With versions of Gluster pre 3.1 any changes to the Gluster configuration, including adding servers (bricks) requires any gluster services on all servers and clients be stopped simultaneously, the new vol files installed, then gluster restarted. Version 3.1 introduced dynamic volumes, eliminating that requirement. Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.c...@gmail.com From: mki-gluste...@mozone.net To: gluster-users@gluster.org Sent: Monday, November 15, 2010 1:45:19 AM Subject: Re: [Gluster-users] setvolume failed (Stale NFS file handle) when volfile is changed On Mon, Nov 15, 2010 at 03:20:43AM -0600, Craig Carl wrote: > > When the client volume file as supplied by one of the servers in a > > distribute/replicate setup changes, my clients can't remount the > > filesystem correctly. Turning on debug mode shows these messages: > > > > [2010-11-13 01:46:45] D [client-protocol.c:6178:client_setvolume_cbk] > > 10.12.47.106-3: setvolume failed (Stale NFS file handle) > > All the client and server volume files must be in sync, having different > client vol files on different clients will result in these types of errors, > it is also the primary cause of split-brain, so please be cautious when > making these kind of changes. Thanks Craig! On a related note, if that's the case wouldn't that mean that adding new bricks requires that you unmount all the client nodes first before you can even attempt to remount the filesystem on them? Or is the typical approach to adding new bricks to copy the updated volume file to the client nodes, and mount the filesystem that way until all your client nodes have successfully unmounted the old config? Thanks. Mohan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setvolume failed (Stale NFS file handle) when volfile is changed
On Mon, Nov 15, 2010 at 03:20:43AM -0600, Craig Carl wrote: > > When the client volume file as supplied by one of the servers in a > > distribute/replicate setup changes, my clients can't remount the > > filesystem correctly. Turning on debug mode shows these messages: > > > > [2010-11-13 01:46:45] D [client-protocol.c:6178:client_setvolume_cbk] > > 10.12.47.106-3: setvolume failed (Stale NFS file handle) > > All the client and server volume files must be in sync, having different > client vol files on different clients will result in these types of errors, > it is also the primary cause of split-brain, so please be cautious when > making these kind of changes. Thanks Craig! On a related note, if that's the case wouldn't that mean that adding new bricks requires that you unmount all the client nodes first before you can even attempt to remount the filesystem on them? Or is the typical approach to adding new bricks to copy the updated volume file to the client nodes, and mount the filesystem that way until all your client nodes have successfully unmounted the old config? Thanks. Mohan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] setvolume failed (Stale NFS file handle) when volfile is changed
Mohan - All the client and server volume files must be in sync, having different client vol files on different clients will result in these types of errors, it is also the primary cause of split-brain, so please be cautious when making these kind of changes. Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.c...@gmail.com From: mki-gluste...@mozone.net To: gluster-users@gluster.org Sent: Friday, November 12, 2010 6:06:55 PM Subject: [Gluster-users] setvolume failed (Stale NFS file handle) when volfile is changed Hi When the client volume file as supplied by one of the servers in a distribute/replicate setup changes, my clients can't remount the filesystem correctly. Turning on debug mode shows these messages: [2010-11-13 01:46:45] D [client-protocol.c:6178:client_setvolume_cbk] 10.12.47.106-3: setvolume failed (Stale NFS file handle) The config was generated using glusterfs-volgen. All I was trying to accomplish was comment out statprefetch volume definition and remount the fs but remounting results in only the first primary/backup server in the replicate group to get mounted. Heck if I even change the transport.remote-port to just read report-port and update the config, the clients cant mount the filesystem anymore. The moment I revert the config back, then they are fine... This is with 3.0.4, although I've seen this happen with 3.0.5 as well. Yes I know 3.1 is out, but I'm not comfortable moving to it just yet, so it's not an option... If I copy that exact volfile to the client and then use that to mount the filesystem, it has no problems... Any ideas as to what is going on here? Why would changing the client volume file on the volfile server break the mount? Thanks. Mohan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] setup trouble without DNS
Hello, I tried to setup an testing environment inside a VMWare Workstation with 3 ubuntu 10.10 server. I can add the probes by hostname, which is configured in /etc/hosts r...@ubuntu01:~# gluster peer statusNumber of Peers: 2 Hostname: ubuntu02 Uuid: b2dc0e78-f91a-4fa2-9faa-ce7d81a281fe State: Peer in Cluster (Connected) Hostname: ubuntu03 Uuid: f127902d-7f46-4438-bfeb-5ce25821ab15 State: Peer in Cluster (Connected) r...@ubuntu01:~# gluster volume create store ubuntu02:/var/store ubuntu03:/var/store Creation of volume store has been unsuccessful But working with the IP Adresses is working r...@ubuntu01:~# gluster volume create store 172.16.101.202:/var/store 172.16.101.203:/var/store Creation of volume store has been successful BR Uwe ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Small Tests in EC2 failing...
Adam - You may be experiencing bug #1053, http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1053 . Gluster 3.1.1qa6 is available here - http://download.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-3.1.1qa6.tar.gz , please try moving to this version on a 64-bit platform. Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.c...@gmail.com From: "Adam Lindsay" To: gluster-users@gluster.org Sent: Sunday, November 14, 2010 10:03:21 AM Subject: [Gluster-users] Small Tests in EC2 failing... A little background. I have gone through a lot of GlusterFS documentation and outdated tutorials on installing and setting up a standard 2 server replication with them acting as clients as well. I am using Ubuntu 10.04 and GlusterFS 3.1. My goals are not that ambitious. I don't have terabytes of data and only need the most modest of replication, to the point where I have strongly considered rsync or unison. GlusterFS seems to be the hotness so I figured I would give it a try. Initially I spawned 2 m1.micro and got everything installed and running. I setup Gluster using the command line tool. The commands that are relevant are below. I do have a bit of questions regarding this, which documentation isn't very clear on. # On Server 1 gluster peer probe gluster volume create websites replica 2 transport tcp :/exp1 :/exp2 gluster volume start websites mkdir -p /mnt/websites modprobe fuse mount -t glusterfs :/websites /mnt/websites As you can see this is extremely straight forward. What is weird is when I start down the path of only simple tests like creating a text file in the /mnt/websites mount and saving, it doesn't take long for the /mnt/websites on both servers to not match. Whats odd is that the /exp1 and /exp2 directories match nearly instantly. I figure the problem lies between the client and the volume. I have tried all kinds of configurations. Mounting both clients on each server to the server1 ip, also their own local IP, I even tried crossing them. Finally I figured, maybe the m1.micro are just too small. So I redid this with m1.small's. Yes these are 32bit, so I had to compile the code to install. This went smoothly, and yet same results. So my questions: 1) Do I have to use clients or can I just read/write to the /exp1 and /exp2 directories directly? 2) Am I expecting too much from an m1.micro or even m1.small? Again this was a simple text file and only a single one. Kinda surprised it would take more CPU just to do that much. 3) I feel this is probably a configuration/optimization issue. It seems as though the replication to the /exp1 and /exp2 directories happen quickly and are ready to go, but something with the default configuration to the client isn't good. 4) Could it be the way I am connecting the clients? Do they always point to server1 ip? or to localhost? Before its recommended, m1.large and a 4 server config is probably out of the budget. If this is what it takes tough than I will simply need to search for another solution. DRBD has come up as a potential for what I want, but seems as though it might suffer from split brain on EC2. Again though given the very very simple test, I would expect this to work even if the instances are a bit underpowered for what most people use on this list. Any advice or help is greatly appreciated. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster client 32bit
I asked about 32bit support when glusterfs 3.1.0 was released: http://www.mail-archive.com/gluster-de...@nongnu.org/msg07150.html They focus on 64bit due to their own clients requiring it - hence I understand their commitment. It is a pity that us 32bit users are without support, but the great thing about opensource and mailing list communities, we can supply support for each other and let the devs continue in their great work. I had not fully tested the 3.1.0 release but did find the fuse client better (file locking is non existent when mounting over NFS) - but the performance hit was quite large on my tiny Virtual Machine cluster setup - so I am unsure if i had short writes just using NFS alone. (I have a feeling it may be related to a bug that some other 64bit users encountered). I will be testing 3.1.1 as soon as it appears. Martin On 15 November 2010 17:59, Christian Fischer wrote: > On Friday 12 November 2010 11:29:52 Bernard Li wrote: >> Hi Stefano: >> >> On Fri, Nov 12, 2010 at 2:18 AM, Stefano Baronio >> >> wrote: >> > is there a way to have a 32bit Glusterfs client? >> >> You can definitely build it yourself, but it is not officially >> supported by Gluster. They recommend you use GlusterFS on 64-bit >> architecture servers. > > The 3.1 documentation states x64 as requirement for server appliances, but no > word about a x64 limitation for clients. Where did you read that? > >> >> Cheers, >> >> Bernard >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
