Re: [Gluster-users] Very slow writes through Samba mount to Gluster with crypt on
On Tue, Dec 20, 2016 at 10:32:17AM -0500, Jeff Darcy wrote: > > Is there some known formula for getting performance out of this stack, or is > > Samba with Glusterfs with encryption-at-rest just not that workable a > > proposition for now? > > I think it's very likely that the combination you describe is not workable. > The crypt translator became an orphan years ago, when the author left a > highly idiosyncratic blob of code and practically no tests behind. Nobody > has tried to promote it since then, and "at your own risk" has been the > answer for anyone who asks. If you found it in the source tree and > decided to give it a try, I'm sorry. Even though it's based in large part > on work I had done for HekaFS, I personally wouldn't trust it to keep my > data correctly let alone securely. Jeff, Thanks for straightening me out. I didn't find it in the source tree. I happened on documentation here, which seemed to imply this was a current project as of 2015: https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/Disk%20Encryption.md And those instructions work for Gluster as provide from the Ubuntu PPA at https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.8 which is of course an "unsupported" version -- if commonly used with Ubuntu. Curious that it includes an orphan translator. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Very slow writes through Samba mount to Gluster with crypt on
Hi, We have Glusterfs 3.8.7 with the crypt translator enabled on Ubuntu 16.04 with Samba 4.3.11. With or without the VFS module enabled, files dropped through an SMB mount from a Windows workstation are painfully slow -- a minute for a file that transfers in several seconds through a similarly-configured Samba to a plain filesystem. Glusterfs mounts accept file transfers far more quickly. But with Samba, the response of staff has been "We can't use this." There's no speed advantage in this use for the VFS module either, surprisingly. Is there some known formula for getting performance out of this stack, or is Samba with Glusterfs with encryption-at-rest just not that workable a proposition for now? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Conflicting info on whether replicated bricks both online
On the one hand: # gluster volume heal foretee info healed Gathering list of healed entries on volume foretee has been unsuccessful on bricks that are down. Please check if all brick processes are running. root@bu-4t-a:/mnt/gluster# gluster volume status foretee Status of volume: foretee Gluster process TCP Port RDMA Port Online Pid -- Brick bu-4t-a:/mnt/gluster 49153 0 Y 9807 Brick bu-4t-b:/mnt/gluster 49152 0 Y 24638 Self-heal Daemon on localhost N/A N/AY 2743 Self-heal Daemon on bu-4t-b N/A N/AY 12819 Task Status of Volume foretee -- There are no active volume tasks On the other: # gluster volume heal foretee info healed Gathering list of healed entries on volume foretee has been unsuccessful on bricks that are down. Please check if all brick processes are running. And: # gluster volume heal foretee info ... Status: Connected Number of entries: 3141 Both systems have their bricks in /mnt/gluster, and the volume then mounted in /backups. I can write or delete a file in /backups on either system, and it appears in both /backups on the other, and in /mnt/gluster on both. So Gluster is working. There have only ever been the two bricks. But there are 3141 entries that won't heal, and a suggestion that one of the bricks is offline -- when they're both plainly there. This is with glusterfs 3.8.5 on Ubuntu 16.04.1. What's my next move? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Conflicting info on whether replicated bricks both online
On the one hand: # gluster volume heal foretee info healed Gathering list of healed entries on volume foretee has been unsuccessful on bricks that are down. Please check if all brick processes are running. root@bu-4t-a:/mnt/gluster# gluster volume status foretee Status of volume: foretee Gluster process TCP Port RDMA Port Online Pid -- Brick bu-4t-a:/mnt/gluster 49153 0 Y 9807 Brick bu-4t-b:/mnt/gluster 49152 0 Y 24638 Self-heal Daemon on localhost N/A N/AY 2743 Self-heal Daemon on bu-4t-b N/A N/AY 12819 Task Status of Volume foretee -- There are no active volume tasks On the other: # gluster volume heal foretee info healed Gathering list of healed entries on volume foretee has been unsuccessful on bricks that are down. Please check if all brick processes are running. And: # gluster volume heal foretee info ... Status: Connected Number of entries: 3141 Both systems have their bricks in /mnt/gluster, and the volume then mounted in /backups. I can write or delete a file in /backups on either system, and it appears in both /backups on the other, and in /mnt/gluster on both. So Gluster is working. There have only ever been the two bricks. But there are 3141 entries that won't heal, and a suggestion that one of the bricks is offline -- when they're both plainly there. This is with glusterfs 3.8.5 on Ubuntu 16.04.1. What's my next move? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] How should ps's VSZ and RSS be interpreted for Gluster?
Hi, I've got a smallish Gluster 3.1.5 instance across two systems with most of the service being by NFS mounts. Yeah, that's old. But it's generally stable and there are other priorities ahead of upgrading it. Recently it started to lag on directory listings on the box providing Gluster's NFS mounts. The current %CPU and %MEM readings in ps looked modest, but the VSZ and RSS (aka VIRT and REZ in htop) readings were way into what looked like impossibly high figures. Restarting Gluster brought those down, and I hope will turn out to fix the sluggish directory listings (seems to, but they were intermittent). Trying to understand this better, I've found this article: http://emilics.com/blog/article/mconsumption.html So I have a rough idea, but I'm still not entirely clear on what it means when over many months those VSZ and RSS values go way up for a process, and what defines sanity for those processes for Gluster, as compared to the danger zone where I really had better restart the thing. If someone can suggest a rule-of-thumb way to calculate the threshold of insanity for these, I'll probably write up a simple Nagios plugin to watch for that being approached, to remind me to restart Gluster before the users start complaining again. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster and OpenStack
On Thu, Dec 05, 2013 at 08:19:04PM +, Nux! wrote: > I don't understand what Rackspace has to do in all this. AFAIK Cinder does > support Gluster. > http://www.gluster.org/category/glusterfs-openstack-cinder/ Rackspace Private Cloud is where they rent the dedicated hardware to the customer with OpenStack installed. Rackspace handles the OpenStack installation. Rackspace is responsible for maintaining the hardware and OpenStack. The customer is responsible for all aspects of the VMs. Since Rackspace has deep OpenStack expertise, and we don't, this makes sense. But Gluster, from Rackspace's perspective, suffers from "not invented here." So my question is whether despite all this we might wedge it into the stack - or overlay it on top - without screwing things up with all the parts with very much want Rackspace to be responsible for. Make sense? Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster and OpenStack
Hi, A firm I work with is seriously considering Rackspace's Private Cloud, which is OpenStack, but Rackspace does not support Gluster, despite reports elsewhere that Gluster, with IBM's and Red Hat's recent contributions, does integrate well with OpenStack. The VMs in this case don't require fast disks, but do require as much HA as we can implement. That's somethihg that in other contexts I've achieved with DRBD, but going forward it looks like Gluster will at some date be the easier, better way to do that. Has anyone using Rackspace managed to shoehorn Gluster in depsite the lack of support from them? Or are there other ways within Rackspace's OpenStack variant to achieve a roughly-equivalent solution without Gluster? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Hardware Sizing Requirments
> We will be deploying 2 gluster nodes (relica-mode) across 2 of our sites. > The sites are connected over 8Gig dark fibre. Our storage disks will be > assigned from EMC VNX storage (SAS/NL-SAS disks). > > This gluster volume will be mounted on 2 application servers, 1 on each > site. The application will be load balanced for user access. This volume > mount-point will act as the data directory for user data. We will have > about 500 users performing simultaneous read-writes to the volume. > > In this scenario, what would be the recommended memory/CPU requirements > for the gluster nodes. Each node gluster node will be a virtual machine > (Centos 6.4 / GlusterFS 3.3.2). My understanding is that replica mode is designed for local replication, not replication between sites. There's also geo-replication, but that's a different animal entirely, a master-slave relationship, not master-master, so it wouldn't work if your users are distributed across two distant sites. If this doesn't rule out Gluster for your use-case, that is if all your users are on one end and it's geo-replication, not replica mode you'll be using, you'll still need to let us know about typical file sizes, number of files, and so on. The number of users doesn't tell us much. If it's 500 employees who all might write a Word document at the same time, that's far different from 500 high-volume websites backed by complex operations on a LAMP stack. Also, EMC VNX may not be suitable. Haven't used it, but EMC says you connect to it by "CIFS, NFS, pNFS, MPFS." Gluster needs to be able to directly deal with an XFS or EXT4 file system. Gluster then allows mounting that file system by NFS or GlusterFS, but you can't put Gluster _on_ a file system mounted by CIFS or NFS. They don't provide the extended attributes Gluster requires. The underlying storage has to be locally, natively mounted, AFAIK. Now, you _might_ be able to do something like put an XFS or EXT4 filesystem within a TrueCrypt file which is in turn on a file system within a remote mount on the EMC VNX, but your performance will suck. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Hardware Sizing Requirments
Bobby, I'm not the one who can say. But those who can will need to know much more than you've specified. What kind of drives are you using, with what controllers? What's are the network components between your two nodes? What is the range of file sizes, in what distribution over that range? How many simultaneous reads and writes do you expect on average, and at peak? How many simultaneous users will there be? What sorts of daemons and services will the file storage support? Best, Whit On Wed, Aug 28, 2013 at 01:52:23PM +, Bobby Jacob wrote: > Hi, > > > > We have a need for setting up 2-Node replica Gluster volume. The storage will > have high read-write requests as this volume will be serving as a data store > for user’s data. > > > > Brick size = 15 TB. > > > > What is the recommended memory/CPU requirement for such a deployment that each > gluster node should have. ? > > > > Thanks & Regards, > > Bobby Jacob > > P SAVE TREES. Please don't print this e-mail unless you really need to. > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Odd error with older Gluster/NFS3 tail swallowing
On Tue, Apr 09, 2013 at 09:44:26AM -0400, Whit Blauvelt wrote: > Had some data loss with an older - 3.1.4 - Gluster share last night. Now > trying to see what the best lessons are to learn from it. Obviously it's too > old a version for a bug report to matter. Wondering if anyone recognizes > this particular sort of error condition though. > > It's a 300 G replicated share, mounted by Gluster's NFS3 to several systems. > It was getting fuller than we like it to be, at over 80%, so I copied a > directory containg 26 G off of it. Checked the copy and it was good. Then I > went and "rm -r"'d that directory. After a few minutes it complained "Cannot > delete directory, directory not empty," citing a subdirectory. Strange. > > So I stopped the process and looked in that subdirectory. The subdirectory > had within it ... the whole of the Gluster share. Damn. Yes, the "rm -r" > had, due to this illusion, managed to wipe out over half of the share > because it had descended into other directories at its root level via this > illusion. Would the likely culprit in this sort of error in the appearance of the filesystem from the NFS client likely be failing RAM on the client? Is the scheme something along the lines of there being a base address for the root of the NFS mount, with addresses within the mount being at an offset from that, so that loss of the offset for the address of a subdirectory could result in the subdirectory seeming itself to contain the whole NFS mount? Obviously I have no knowledge on this level. Any of your filesystem gurus seen this before, or have a hypothesis? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Odd error with older Gluster/NFS3 tail swallowing
Had some data loss with an older - 3.1.4 - Gluster share last night. Now trying to see what the best lessons are to learn from it. Obviously it's too old a version for a bug report to matter. Wondering if anyone recognizes this particular sort of error condition though. It's a 300 G replicated share, mounted by Gluster's NFS3 to several systems. It was getting fuller than we like it to be, at over 80%, so I copied a directory containg 26 G off of it. Checked the copy and it was good. Then I went and "rm -r"'d that directory. After a few minutes it complained "Cannot delete directory, directory not empty," citing a subdirectory. Strange. So I stopped the process and looked in that subdirectory. The subdirectory had within it ... the whole of the Gluster share. Damn. Yes, the "rm -r" had, due to this illusion, managed to wipe out over half of the share because it had descended into other directories at its root level via this illusion. And it's an illusion. It wasn't some bizarre tail-eating self-mounting. The copy of the directory shows it populated by three subdirectories with a few files in them. The version of the directory on the Gluster NFS share has none of those three. Instead it has the whole of the root-level directies of the share visible there. Both of the backing shares also have those three directories properly populated - no tail-eating mount. Another system with the same NFS3 mount of this share shows those three directories properly. So it's an illusion based in some error in how Gluster and/or the NFS client is presenting the directory structure on the one client. The backing stores are ext4; the kernel is old enough to be fully compatible with Gluster in ext4. Anyway, no apparent error at that level. I'm a generalist as a sysadmin, in a smallish shop, half-ignorant on filesystems. Aside from putting more weight on the "pursuing the cutting edge may sometimes be safer than staying with the apparent time-proven stability of an older version" side of the perennial debate between progressive and conservative approaches, what should I learn from this? The NFS client is running a 2.6.24 kernel - yeah, that's old, but it's been very reliable up until this. Is this some known NFS client problem since fixed? Or a really-bizarre one-off? If it is something that can still "just happen," what's safe procedure? Looking through every subdirectory of a tree about to be deleted to make sure it hasn't in a virtual way and without anything that shows to "mount" mounted the whole filesystem its within to itself seems much. I did have, fortunately, an rsnapshot backup of the whole thing, so have been able to restore. But I'd like to avoid the whole experience next time. What's the wisest way to go? Thanks Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Slow read performance
As I read this: > https://bugzilla.redhat.com/show_bug.cgi?id=838784 the bug is, from Gluster's POV, between ext4 and the kernel. Is this correct, that Gluster can't safely use ext4 on recent kernels until ext4's relationship with the kernel is fixed? If Gluster can't be simply patched to fix this, and we want to use ext4 (for which there are good arguments, including a rich toolset), should we be leaning more on the ext4 developers? I see there's a wiki we might document the brokenness on at https://ext4.wiki.kernel.org/ which mentions nothing about Gluster compatibility, and a bugzilla at https://bugzilla.kernel.org/buglist.cgi?product=File+System&bug_status=NEW&bug_status=REOPENED&bug_status=ASSIGNED&component=ext4 Searching for "ext Gluster" there gives "Zarro Boogs found" though. Should someone who can make a fuller report on this bug than I can be making sure that the ext4 project is focused on this problem? Update: Got interrupted before sending this off. I see from other emails since that T'so has been leaned on, and apparently doesn't want to fix ext4?! I know Ted's got rank. But should we collectively be trying to push this to Linus's attention? I'm unclear whether for practical purposes Ted just _is_ ext4, or whether his co-developers count there. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] timestamps getting updated during self-heal after primary brick rebuild
A small question: We know that one or two members of the dev team read these emails. One said just yesterday he's more likely to see emails than bug reports. Now, sometimes the response to an email showing an obvious bug is "File a bug report please." But for something like this - yes timestamps are data so this is a serious bug - it would be a healthy thing if someone on the dev team would make a point of both acknowledging that it's a bug, and taking responsibility for being sure the bug report is filed and assigned to the right people, whether or not the email writer has taken that step. If the team's too small to follow through like this, is someone advocating with Red Hat for more staff? They've made a large investment in Gluster, which they might want to protect by fully staffing it. It's the fault of the firm, not current project staff, if the current staffing is too thin. Apologies if these reflections are out of place in a community discussion. But it's in the community's interest that Red Hat succeeds and profits from its Gluster purchase. Best, Whit On Wed, Mar 06, 2013 at 03:28:39AM -0500, John Mark Walker wrote: > Hi Todd, > > A general note here: when someone posts a question and noone responds, > it's generally because either no one has seen that particular behavior and > they don't know how to respond, or they didn't understand what you were > saying. In this case, I'd say it is the former. > > - Original Message - > > something entirely different. We see the same behavior. After > > rebuilding the > > first brick in the 2-brick replicate cluster, all file timestamps get > > updated > > to the time self-heal copies the data back to that brick. > > > > This is obviously a bug in 3.3.1. We basically did what's described > > here: > > > > > > http://gluster.org/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server > > > > and timestamps get updated on all files. Can someone acknowledge > > that this > > sounds like a bug? Does anyone care? > > Please file a bug and include the relevant information at >https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS > > - after searching for any similar bugs, of course. > > > > > Being relatively new to glusterfs, it's painful to watch the mailing > > list and > > even the IRC channel and see many folks ask questions with nothing > > but > > silence. I honestly wasn't sure if glusterfs was actively being > > supported > > ??? Our IRC channel is one of the most active in the open source world. I'm > honestly not sure what mailing lists or IRC channels you've been watching. > > > > anymore. Given the recent flurry of mail about lack of documentation > > I see > > that's not really true. Unfortunately, given that what I'm seeing is > > a form > > of data corruption (yes, timestamps do matter), I'm surprised > > nobody's > > interested to help figure out what's going wrong. Hopefully it's > > something > > about the way I've build out cluster (though it seems less and less > > likely > > given we are able to replicate the problem so easily). > > I can understand your frustration. I would be, also. However, given that I > haven't heard of this problem before, I don't know how you were able to > reproduce it. The best I can offer is that we'll investigate your bug report. > > Thanks, > JM > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS 3.4 Alpha Release
On Mon, Feb 11, 2013 at 12:18:38PM -0500, John Mark Walker wrote: > Here's a write-up on the newly-released GlusterFS 3.4 alpha: > > http://www.gluster.org/2013/02/new-release-glusterfs-3-4alpha/ If we want to test the QEMU stuff, is there a more thorough/current writeup than Rao's blog post at http://raobharata.wordpress.com/2012/10/29/qemu-glusterfs-native-integration/? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] high CPU load on all bricks
On Fri, Feb 01, 2013 at 12:53:24PM -0800, Michael Colonno wrote: > Forgot to mention: on a client system (not a brick) the glusterfs > process is consuming ~ 68% CPU continuously. This is a much less powerful > desktop system so the CPU load can’t be compared 1:1 with the systems > comprising the bricks but still very high. So the issue seems to exist with > both glusterfsd and glusterfs processes. You said the host systems were also running the gluster client to have the gluster mount locally available. Does the high load go away on them if you unmount that? Some versions back (3.1.5), on some older systems (Ubuntu 8.04 with Gluster built from source), I had problems with the gluster client running away on me. Those systems are happy using nfs mounts instead. And the gluster servers, which aren't doing local mounting of gluster at all, don't use much by way of CPU. Then again, they're not highly stressed by use, so it's not much of a test. That's a long winded way of asking, are you sure it's glusterfsd too, and not just glusterfs sucking up the CPU cycles? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Meta
On Tue, Jan 22, 2013 at 08:37:03AM -0500, F. Ozbek wrote: > I am surprised at Jeff's responses. What happened to freedom of speech? > I can't say we tested 3 products and 2 failed and one passed our tests? In fairness, you didn't say you'd been testing for 3 months. And you still haven't said what the tests were. Jeff has suggested a range of tests where Moose should be expected to fail, and Gluster not. He has also mentioned areas where Gluster doesn't lead the pack. And he's posted something of his methods and results in various places. Facts do not insult. But opinions without facts can. It's not enough that you've got the facts hidden away somewhere. You should present the data along with the opinions. Without data, it's about you - whether we trust your reputation, of which you have none here. No offense. With data, it's not longer just about personalities. We've not only got freedom of speech. We've got freedom of guns. Still, walking into the meeting with your gun drawn will get you viewed as rude or worse. We're supposed to be data pros here, not cowboys. So, data please. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Meta
On Sun, Jan 20, 2013 at 11:42:44AM -0500, John Mark Walker wrote: > On the other hand, do you see people recommending Fedora on the Ubuntu > lists? Raises the question: Where do we have meta-discussions and have it be appropriate? This isn't religion, and shouldn't be. If someone on an Ubuntu list is trying to do something where a RH-type OS has the advantage over a Debian-derived one, does discussion of that have to be whispered off list? Or is it healthy for everyone to be aware of the alternatives, whether for their own use or as the competition? For general use, all the major distros are great, and arguing over which is or isn't "enterprise class" only reveals who is a fool for marketing campaigns. But for a specific use case, there are times when a particular distro has a clear-cut advantage. For the filesystem space that Gluster and its peers are in, arguably none has attained true greatness yet. My current bet, in my own deployments, is on Gluster doing well in the near term. So it's the Gluster list I'm following closely. But news of how it compares to the others is welcome to me and I'd guess others of us here. Short of subscribing to more lists than we can reasonably follow what with other duties, a bit of the meta-discussion leaking into this space can be welcome to us. While "This rocks! That sucks!" is just ignorant noise, always, fact- and experience-based analysis is both rare and valuable. I hope there remains room for it here. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs performance issues - meta
On Tue, Jan 08, 2013 at 04:49:30PM +0100, Stephan von Krawczynski wrote: > > Pointing out that a complex system can go wrong doesn't invalidate complex > > systems as a class. It's well established in ecological science that more > > complex natural systems are far more resilient than simple ones. A rich, > > complex local ecosystem has a higher rate of stability and survival than a > > simple, poorer one. That's assuming the systems are evolved and have niches > > well-fitted with organisms - that the complexity is organic, not just > > random. > > That is a good example for excluded corner cases, just like the current split > brain discussion. All I need to do to your complex natural system to > invalidate is to throw a big stone on it. Ask dinosaurs for real life > experience after that. Throw a big enough stone and anything can be totally crushed. The question is one of resilience when the stone is less than totally crushing. The ecosystem the big stone was thrown at which included the dinosaurs survived, because in its complexity it also included little mammals - which themselves were more complex organisms than the dinosaurs. Not that some simpler organisms didn't make it through the extinction event too. Plenty did. The chicken I ate for dinner is a descendant of feathered dinosaurs. Take two local ecosystems, one more complex than the other. Throw in some big disturbance, the same size of disruption in each. On average, the complex local ecosystem is more likely to survive and bounce back, while the simple one is more likely to go into terminal decline. This is field data, not mere conjecture. Your argument here could be that technological systems don't obey the same laws as ecosystems. But work in complexity theory shows that the right sorts of complexity produce greater stability across a broad range of systems, not just biological ones. Free, open source software's particular advantage is that it advances in a more evolutionary manner than closed software, since there is evolutionary pressure from many directions on each part of it, at every scale. Evolutionary pressure produces complexity, the _right sort_ of complexity. That's why Linux systems are more complex, and at the same time more stable and manageable, than Windows systems. Simplicity does not have the advantage. Even when smashing things with rocks, the more complex thing is more likely to survive the assault, if it has the right sort of complexity. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs performance issues
On Tue, Jan 08, 2013 at 02:42:49PM +0100, Stephan von Krawczynski wrote: > What an dead-end argument. _Nothing_ will save you in case of a split-brain. So then, to your mind, there's _nothing_ Gluster can do to heal after a split brain? Some non-trivial portion of the error scenarios discussed in this thread result from a momentary or longer split-brain situation. I'm using "split-brain" in the broad sense of any situation where two sides of a replicated system are out-of-touch for some period and thus get out-of-sync. Isn't that exactly what we're discussing, how to heal from that? Sure, you can have instances of specific files beyond algorithmic treatment. But aren't we discussing how to ensure that the largest possible portion of the set of files amenable to algorithmic treatment are so-handled? > > That's the thing about complex systems. Trivial solutions are usually both > > simple and wrong. Some work most of the time, but there are corner cases. As > > we see with Gluster even complex solutions tend to have corner cases; but at > > least in complex solutions the corners can be whittled down. > > Can they? I'd rather say if it is non-trivial it is broken most of the time. > Ask btrfs for confirmation. Pointing out that a complex system can go wrong doesn't invalidate complex systems as a class. It's well established in ecological science that more complex natural systems are far more resiliant than simple ones. A rich, complex local ecosystem has a higher rate of stability and survival than a simple, poorer one. That's assuming the systems are evolved and have niches well-fitted with organisms - that the complexity is organic, not just random. Computer software, hardware, and the human culture that supports them also form complex, evolved ecosystems. Can there be simple solutions that help optimize such complex systems? Sure. But to look only for simple solutions is to be like the proverbial drunk looking for his keys under the streetlight, even though he heard them drop a half-block away, because "The light is better here." When people try to apply simple solutions to complex, evolved ecosystems, the "law of unintended consequences" is more the rule than the exception. Solutions that appear simple and obvious should always be suspect. Granted, complex, obscure ones also require scrutiny. It's just, the simple stuff should never get a pass. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs performance issues
On Tue, Jan 08, 2013 at 01:11:24PM +0100, Stephan von Krawczynski wrote: > Nobody besides you is talking about timestamps. I would simply choose an > increasing stamp, increased by every write-touch of the file. > In a trivial comparison this assures you choose the latest copy of the file. > There is really no time needed at all, and therefore no time synchronisation > issues. So rather than the POSIX attribute of a time stamp, which is I'm pretty sure what we all thought you were talking about, you're asking for a new xattribute? And you want that to be simply iterative? Okay, so in a split-brain, a file gets touched 5 times on one side, and actually written to just once, not touched at all, on the other. Then the system's brought back together. Your "trivial comparison" will choose the wrong file version. That's the thing about complex systems. Trivial solutions are usually both simple and wrong. Some work most of the time, but there are corner cases. As we see with Gluster even complex solutions tend to have corner cases; but at least in complex solutions the corners can be whittled down. Regards, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Meta-discussion
There's a strong trend against documentation of software, and not just in open source. I'm old enough to remember when anything modestly complex came with hundreds of pages of manuals, often over several volumes. Now, I can understand why commercial software with constrained GUIs wants to pretend that what's underneath is as simple as the GUI suggests, so as not to scare away customers. And I can understand why some open source projects might want to withhold knowledge to motivate consulting contracts, as cynical as that may be. But something on the scale of Gluster should have someone hired full time to do nothing but continuously write and update documentation. If you need a business model for that, print the results in a set of thick books, and sell it for $250 or so. Print JIT so you can track point releases. What Brian asks for should be the core of it. Even when stuff breaks for people who have paid for their RedHat Solution Architect, it will give that architect a place to look up the fix quickly, rather than having to go bother the development team, who are more profitably deployed in development. Best, Whit On Wed, Jan 02, 2013 at 01:19:17PM +0100, Fred van Zwieten wrote: > +1 for 2b. > > I am in de planning stages for an RHS 2.0 deployement and I too have suggested > a "cookbook" style guide for step-by-step procedures to my RedHat Solution > Architect. > > What can I do to have this upped in the prio-list? > > Cheers, > Fred > > > On Wed, Jan 2, 2013 at 12:49 PM, Brian Candler wrote: > > On Thu, Dec 27, 2012 at 06:53:46PM -0500, John Mark Walker wrote: > > I invite all sorts of disagreeable comments, and I'm all for public > > discussion of things - as can be seen in this list's archives. But, for > > better or worse, we've chosen the approach that we have. Anyone who > would > > like to challenge that approach is welcome to take up that discussion > with > > our developers on gluster-devel. This list is for those who need help > > using glusterfs. > > > > I am sorry that you haven't been able to deploy glusterfs in production. > > Discussing how and why glusterfs works - or doesn't work - for > particular > > use cases is welcome on this list. Starting off a discussion about how > > the entire approach is unworkable is kind of counter-productive and not > > exactly helpful to those of us who just want to use the thing. > > For me, the biggest problems with glusterfs are not in its design, feature > set or performance; they are around what happens when something goes > wrong. > As I perceive them, the issues are: > > 1. An almost total lack of error reporting, beyond incomprehensible > entries > in log files on a completely different machine, made very difficult to > find > because they are mixed in with all sorts of other incomprehensible log > entries. > > 2. Incomplete documentation. This breaks down further as: > > 2a. A total lack of architecture and implementation documentation - such > as > what the translators are and how they work internally, what a GFID is, > what > xattrs are stored where and what they mean, and all the on-disk states you > can expect to see during replication and healing. Without this level of > documentation, it's impossible to interpret the log messages from (1) > short > of reverse-engineering the source code (which is also very minimalist when > it comes to comments); and hence it's impossible to reason about what has > happened when the system is misbehaving, and what would be the correct and > safe intervention to make. > > glusterfs 2.x actually had fairly comprehensive internals documentation, > but > this has all been stripped out in 3.x to turn it into a "black box". > Conversely, development on 3.x has diverged enough from 2.x to make the > 2.x > documentation unusable. > > 2b. An almost total lack of procedural documentation, such as "to replace > a > failed server with another one, follow these steps" (which in that case > involves manually copying peer UUIDs from one server to another), or "if > volume rebalance gets stuck, do this". When you come across any of these > issues you end up asking the list, and to be fair the list generally > responds promptly and helpfully - but that approach doesn't scale, and > doesn't necessarily help if you have a storage problem at 3am. > > For these reasons, I am holding back from deploying any of the more > interesting features of glusterfs, such as replicated volumes and > distributed volumes which might grow and need rebalancing. And without > those, I may as well go back to standard NFS and rsync. > > And yes, I have raised a number of bug reports for specific issues, but > reporting a bug whenever you come across a problem in testing or > production > is not the right answer.
Re: [Gluster-users] Turning GlusterFS into something else (was Re: how well will this work)
On Sun, Dec 30, 2012 at 05:12:04PM +0100, Stephan von Krawczynski wrote: > If I delete > something on a disk that is far from being full it is just plain dumb to > really erase this data from the disk. It won't help anyone. It will only hurt > you if you deleted it accidently. Read my lips: free disk space is wasted > space, just like free mem is wasted mem. > And _that_ is the true reason for undelete. It won't hurt anybody, and will > help some. And since it is the true goal of a fs to organise data on a drive > it is most obvious that "undelete" (you may call it lazy-delete) is a very > basic fs feature and _not_ an add-on patched onto it. Stephan, It's good to have a strong debater here like yourself. But you overlooked Jeff's citing "compliance reasons." I don't know what sort of data you deal in. But if it's anything financial, at all, there is serious jeopardy if deleted files aren't really deleted. Much of it has both regulatory and contractual requirements, plus potential legal liability. Yeah, I know parts of deleted files still often linger on the disk anyway. But maintaining an index to those files, which would be what your request would require, would put many of us in violation of these requirements in a way that that simply does not. If a system is compromised, it's going to be far easier for the compromiser to find deleted data if there's an available index to it. It's far more work, and a far more obvious intrusion, if they have to go sector-by-sector through the storage. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster and public/private LAN
On Tue, Dec 18, 2012 at 03:26:57PM -0700, Shawn Heisey wrote: > I have an idea I'd like to run past everyone. Every gluster peer > would have two NICs - one "public" and the other "private" with > different IP subnets. The idea that I am proposing would be to have > every gluster peer have all private peer addresses in /etc/hosts, > but the public addresses would be in DNS. Clients would use DNS. > > The goal is to have all peer-to-peer communication (self-heal, > rebalance, etc) happen on the private network, leaving all the > bandwidth on the public network available for client connections. > > Will this work on 3.3.1 or newer? ... Don't know. But works fine with 3.1.5. Seems like the right way to do it. good NICs are cheap, relatively. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] CIFS options - anyone done A-B comparisons on various setups?
On Wed, Dec 12, 2012 at 09:28:24AM +0100, Gunnar wrote: > The problem was that the open file limit was too low, after raising it > with: ulimit -n 65536 (which is probably to high) I had no further crash. That makes sense. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster machines slowing down over time
Have you used top or htop to see if it's CPU use, by what? Whit On Tue, Dec 11, 2012 at 04:13:15PM +, Tom Hall wrote: > I have 2 gluster servers in replicated mode on EC2 with ~4G RAM > CPU and RAM look fine but over time the system becomes sluggish, particularly > networking. > I notice when sshing into the machine takes ages and running remote commands > with capistrano takes longer and longer. > Any kernel settings people typically use? > > Thanks, > Tom > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] CIFS options - anyone done A-B comparisons on various setups?
On Tue, Dec 11, 2012 at 10:10:32AM +0100, Gunnar wrote: > after testing for a while (after copying several 10 files) it > seems that either glusterfs or glusternfs is crashing under load. > The the average load on the machine goes up to 8 or 9, before it was > max around 1, but there is no according process Is this uniquely with a Samba re-export of a Gluster NFS export? Or do you also see it with a Gluster NFS export alone? If it's only with Samba, are Samba's logs showing anything? Many months back, with Gluster 3.1.5 NFS re-exported though Samba 3.0.28a, we hit a situation where Samba's logs were filling far too fast with an error (not precisely remembered) that led me to add "posix-locking = no" to the individual share configs in smb.conf and "unix extensions = no" to [global]. I can't speak to the exact need for either - it was more a matter of googling other's responses to the error message and seeing those sometimes suggested in other contexts as cures. IIRC the Samba project itself had no useful documentation on these options or when they're useful. On the one hand, it's since been half a year without Samba running away like that again. On the other, the setup had been running for half a year before it did the first time. So this may be an anecdote with no diagnostic relevance. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] CIFS options - anyone done A-B comparisons on various setups?
Gunnar, > Second fastest is #1, nfs mount shared by Samba 4000 files in around 6 min > Slowest is #2 where I need more than 12 min for 4000 files. Thanks for running that test. That's a significant difference. I wonder in the Samba > Gluster client > Gluster server scenario whether the slowness is the Gluster client transacting with both servers rather than just the local one. You've at least confirmed my suspicion that Samba > NFS > Gluster is not at any speed disadvantage. And in many months of running that way, as I said, there have been no performance complaints - although with this an unsupported configuration it could turn out we've just been lucky and that there's something yet that can go wrong. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] CIFS options - anyone done A-B comparisons on various setups?
Gunnar, I claim nothing special in terms of Samba knowledge. Not even that this is optimal in any dimension. All I can say is that none of my users have complained about performance, in a situation where speed's not critical as long as the overall system is dependable. But my current Samba conf, for a CIFS share run from a third system exporting an NFS share via CIFS that originates on Glusterfs (3.1.5) is: [global] workgroup = xyz netbios name = abc interfaces = eth1 192.168.1.250/24 encrypt passwords = true wins server = 192.168.1.9 create mask = 0666 force create mode = 0666 directory mask = 0777 force directory mode = 0777 hosts allow = 192.168.1. load printers = no printing = none printcap name = /dev/null disable spoolss = yes unix extensions = no [sharename] path = /path/to/nfsmountof/glusterfs valid users = qwert yuiop writeable = Yes posix-locking = No I recall having strong reasons to turn off unix extensions and posix-locking, in terms of hitting errors otherwise. I should have kept notes though, as that was long enough ago I don't remember the specifics. What are you using as a Windows NFS client? I had the impression Windows didn't have a good option there. Whit On Tue, Dec 04, 2012 at 11:06:45PM +0100, Gunnar wrote: > Hi Whit, > > could you post your smb.conf? I'm currently a bit lost with a performance > optimized setting for millions of small files accessed via a Samba share > (local Gluster fuse mount). I would be glad to try out your approach and > see how the results will be since a NFS access from Windows gives me a > much better throughput than the Samba share. > > Thanks, > > Gunnar ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Does brick fs play a large role on listing files client side?
Avanti, For those of us willing compile kernels when there's a distinct advantage, has this patch made it into a kernel release? If so, which? Thanks, Whit On Tue, Dec 04, 2012 at 04:35:39PM -0800, Anand Avati wrote: > Support for READDIRPLUS in FUSE improves directory listing performance > significantly. You will have to hold on till http://git.kernel.org/?p=linux/ > kernel/git/mszeredi/fuse.git;a=commit;h= > 81aecda8796572e490336680530d53a10771c2a9 trickles down into your distro kernel > however. > > Avati ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] CIFS options - anyone done A-B comparisons on various setups?
Hi, I'm about to set up Gluster 3.3.1 in a cloud environment. The clients are going to be nfs and cifs as well as Gluster. The Gluster docs suggest setting up cifs as a share of a local Gluster client mount. My setup in another, cloudless setup (w/ Gluster 3.1.5) has been of Gluster mounted on a separate system via nfs, and then that nfs mount shared via cifs - which I don't see discussed in the Gluster docs but which has worked fine for us. There are different scenarios which I'd expect would all pretty much work. I'm wondering if any stand out in anyone's experience as particularly better or worse. The constant is two replicated Gluster storage VMs, and use of Gluster's built-in nfs, most likely with ucarp handling nfs failover. Beyond that the cifs options include: 1. Local Gluster client mounts on both storage systems, with cifs locally providing exports for remote systems, sharing the nfs-ucarp failover 2. A third VM, using a Gluster client to mount the Gluster storage, with cifs providing re-export to remote systems 3. A third VM, using an nfs client to mount the Gluster storage, with cifs providing re-export to remote systems Those last two as-is introduce a single point-of-failure, but the third VM could have a fourth VM in ucarp-failover relation to it (assuming ucarp failover works for cifs). The profusion of VMs, in the billing structure of our cloud provider, isn't costly. Billing is mostly based on CPU cycles rather than cores or number of VMs provisioned. From that POV an nfs client re-exported to cifs may hold an advantage over a Glusterfs client - being built into the kernel is more efficient than running in fuse. Also, probably unfairly based on experience with even older versions of Gluster, I trust Gluster-as-server more than I do the Gluster client. So if it's scenario (2) above, the Gluster client is running on a separate instance than the storage. If it runs away and overloads that VM, the storage VMs shouldn't be affected and can keep serving nfs even if cifs chokes. Scenarios (1) and (3) could be set up so that an nfs channel to one of the storage pair, in normal running, serves nfs to clients, while a second nfs channel to the other of the storage pair is the basis for the cifs export. Maybe any of these would work reasonably well. Maybe not. I don't have the leisure of setting up extensive testing on A-B comparisons right now - management says I have to deploy this "yesterday." Any advice is most welcome. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] worth upgrading from 3.2.7 to 3.3.1?
Gerald, How's the VM's on Gluster thing working? Stable? Fast enough where speed's not essential? Thanks, Whit On Tue, Nov 27, 2012 at 09:36:24AM -0600, Gerald Brandt wrote: > Hi, > > I have speed and stability increases with 3.3.0/3.3.1. If you're running > VM's on gluster, it's a no brainer as well. > > Gerald ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Inviting comments on my plans
On Sun, Nov 18, 2012 at 07:56:36PM -0500, David Coulson wrote: > On 11/18/12 7:53 PM, Whit Blauvelt wrote: > >Red Hat does not support upgrades between major versions. Thus Cent OS and > >Scientific don't either. > I work in an Enterprise environment, and in general once a RHEL > release goes out of support it is time to replace the hardware > anyway - We usually stick to a 4-5yr max lifecycle for hardware, > which fits well with the RedHat 7yr support model. Assuming you are > running non-RHEL software, an upgrade between major versions has to > be a complete mess, if not impossible. Please excuse me as I go off topic perhaps. Then again, OS variant choice can be as important as hardware choice for a Gluster environment, and we certainly discuss hardware a lot here. There are times when there are compelling upgrades available for the major daemons that are the reason for a particular system's life. With Debian or Ubuntu, upgrading to support more recent daemons is generally trivial. I've recently upgraded several systems in place (yes, systems vital to an enterprise) from Ubuntu 8.04 to 12.04. There can be one or two packages that need a small fix along the way, but I've never hit a blocker. By contrast, taking a system for a guy doing advanced Perl stuff from RH 5 to RH 6 meant having to go through and totally rebuild the environment (several years of accumulated customizations) on the new system. Took a few days, where a similar upgrade over a similar span of OS evolution in Debian or Ubuntu would take a few hours. I've also been known to pull a drive from an old server that's been complexly configured, shove it in new hardware, boot from it, clone the system to new drives, and then upgrade in place. Far simpler than reconfiguring everything on a fresh OS install - as long as the OS supports rolling upgrades and doesn't block the path between major versions. There's no reason your configuration work should need to be redone just because the hardware's being replaced. But if you're doing fairly stock stuff, and your custom configuration on top of the OS is trivial, sure the RH way is solid. I'm not knocking RH. I respect the hell out of the corporate culture and the quality of much of the fundamental development work there. The concern with a Gluster system, even with RH the developer, is that at some point the latest Gluster won't have its prerequisites met by the then-current RHEL 6.x. So if you're building Gluster storage and really want to have the best Gluster version running on it, say, two years from now, it may have advanced far beyond whatever RH is using for a kernel then (despite backports). Or if not Gluster, other daemons which may be vital to the enterprise use of the system may have advanced beyond RH 6.x compatibility. Debian variants handle that better. To each his or her own. All the major Linux variants are heavily and successfully used in enterprise. And none is better across the board. It depends on your niche and circumstances. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Inviting comments on my plans
On Sun, Nov 18, 2012 at 09:58:58PM +, Brian Candler wrote: > On Sun, Nov 18, 2012 at 09:27:41AM -0700, Shawn Heisey wrote: > > Do you happen to know if it will be possible > > to upgrade from CentOS 6 to CentOS 7? The lack of an upgrade path > > from 5 to 6 has been a major headache. > > Sorry I don't; I wasn't even aware that 5 to 6 upgrade was not possible. Red Hat does not support upgrades between major versions. Thus Cent OS and Scientific don't either. That's a major part of why I generally run Ubuntu or Debian instead, except for users who are really wedded to the Red Hat way. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3.1 breaks NFS CARP setup
On Mon, Oct 22, 2012 at 09:49:21AM -0400, John Mark Walker wrote: > Dan - please file a bug re: the NFS issue. Glad to hear this will be treated as a bug. If NFS is to be supported at all, being able to use a virtual-IP setup (whether mediated by CARP or otherwise) is essential. And considering that NFS is available to some clients who can't run FUSE locally, or can't run FUSE efficiently, NFS is a good thing to support. Just my cent-and-a-half. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Protocol stacking: gluster over NFS
On Fri, Sep 14, 2012 at 09:41:42AM -0700, harry mangalam wrote: > > > What I mean: > > > - mounting a gluster fs via the native client, > > > - then NFS-exporting the gluster fs to the client itself > > > - then mounting that gluster fs via NFS3 to take advantage of the > > > client-side caching. Harry, What is "the client itself" here? I'm having trouble picturing what's doing what with what. No doubt because it's too clever for me. Maybe a bit more description would clarify it nonetheless. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Has anyone used encrypted filesystems with Gluster?
On Thu, Sep 13, 2012 at 05:02:48PM -0700, Adam Brenner wrote: > > Yeah. But then, if it could do an unattended boot, then someone who steals > > the system isn't hindered by the encryption either. > > In this hypothetical situation, is securing your machine room a > consideration? I suspect someone could do far worse things, destroying > your entire infrastructure, that is pretty important as well, > hypothetical speaking :) The computer room is well secured. But some of the data we handle belongs to other firms. Several have recently been strongly suggesting that they prefer that any file systems their data rests on be encrypted. I knew a guy who lived in Kathmandu in the 60s. There was a restaurant favored by Westerners, who would often ask, "Do you boil your water?" The answer: "Yes!" One day he went into the kitchen and there on the stove was a pot of boiling water. It wasn't the water they served anyone. They just thought that Westerners believed it as auspicious to keep a pot of water boiling in the kitchen. So do I believe someone's going to steal our systems? Not a chance in a thousand. But it will help the boss's business to be able to say, "Yes, we boil our water." Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Has anyone used encrypted filesystems with Gluster?
On Thu, Sep 13, 2012 at 02:42:13PM -0700, Joshua Baker-LePain wrote: > I haven't, but given that Gluster runs on top of a standard FS, I > don't see any reason why this wouldn't work. Rather than just > Gluster on top of ext3/4/XFS, it would be Gluster on top of > ext3/4/XFS on top of an LUKS encrypted partition. > > The main stumbling block I see isn't Gluster related at all, it's > simply how to do an unattended boot of a system with an encrypted > partition... Yeah. But then, if it could do an unattended boot, then someone who steals the system isn't hindered by the encryption either. So this gets into the "On 4 a.m. reboot, automate waking an administrator" territory. Then there's the problem of how to get Gluster to behave and wait for the underlying storage to be mounted before trying to use it - since as I've seen it will just use the mountpoints without the mount. But as was kindly suggested, if the mountpoints are subdirectories rather than / of the underlying partitions, that will stop Gluster. Would the encrytion overhead noticeably decrease the speed of Gluster? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Has anyone used encrypted filesystems with Gluster?
Hi, This may be crazy, but has anyone used filesystem encryption (e.g. LUX) under Gluster? Or integrated encryption with Gluster in some other way? There's a certain demand to encrypt some of our storage, in case the hypothetical bad guy breaks into the server room and walks out with the servers. Is this a case where we can have encryption's advantages _or_ Gluster's? Or is there a practical way to have both? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster with iNotify
Hi, I don't know about Gluster support, but I use inotify via incrontab to watch mounted Gluster filesystems and it works well. Most of my use of it is just triggering scripts when new files arrive. See: http://inotify.aiken.cz/?section=incron&page=doc&lang=en There are limitations. It has to have a line in the table for each subdirectory watched - no recursion. And it only works if the file comes in through the mounting system. So if system A has the Gluster share separately mounted, and adds a file, incrontab on system A can see it, but incrontab on system B won't be triggered. From a brief discussion with inotify's author, there's no likely way to ever get the second to work. Was the Gluster project considering some sort of signalling between clients to accomplish that? Whit On Mon, Sep 10, 2012 at 02:19:26PM +0800, Kent Liu wrote: > Hi there, > > > > Is there any news on inotify support for GlusterFS? I see this in 3.4 list but > I didn’t see any progress anywhere. > > > > http://www.gluster.org/community/documentation/index.php/Planning34/Inotify > > > > Thanks, > > Kent > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Throughout over infiniband
On Mon, Sep 10, 2012 at 11:13:11AM +0200, Stephan von Krawczynski wrote: > If you have small files you are busted, if you have workload on the clients > you are busted and if you have lots of concurrent FS action on the client you > are busted. Which leaves you with test cases nowhere near real life. > I replaced nfs servers with glusterfs and I know what's going on in these > setups afterwards. If you're lucky you reach something like 1/3 of the NFS > performance. This is an informative debate. But I wonder, Stephan, how much of the bustedness you report is avoided by simply using NFS clients with Gluster. For my own purposes, Gluster (3.1.4) has performed well in production. And it's hardly an optimal arrangement. For instance, there's a public-facing FTP/SFTP server NFS mounting mirrored Gluster servers over a gigabit LAN. The FTP/SFTP server also re-exports the NFS Gluster shares via Samba, to a medium-sized office. Numerious files of a wide range of sizes continuously go in an out of this system from every direction. We haven't always run this way. It used to be the same setup, but with local storage on the FTP/SFTP/Samba server. Yet nobody since the switch to Gluster has so much as commented on the speed, and half of our staff are programmers who are not shy about critiquing system performance when they see it lagging. Now, when I tested Gluster for KVM - yeah, that doesn't work. And there are going to be environments in which file servers are far more stressed than ours. But since part of the slowness of Gluster for you is about the Gluster clients rather than servers - well, I'd like to see the NFS-client-to-Gluster test as comparison. Some here have reported it's faster. It's certainly in my own use case fast enough. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Unexpected Gluster behavior on startup without backing stores mounted
On Fri, Sep 07, 2012 at 07:32:14AM +0100, Brian Candler wrote: > On Thu, Sep 06, 2012 at 03:19:53PM -0400, Whit Blauvelt wrote: > > Here's the unexpected behavior: Gluster restored the nfs export based on the > > backing store. But without that backing store really mounted, it used > > /mnt/xyz, which at that point was a local subdirectory of /. This is not > > optimal. An error or warning or refusal to run would be preferable to > > presenting the export when the "real" backing store has gone missing. > > For a different reason, I have been using a subdirectory of the mountpoint > for the brick - e.g. if the filesystem is /mnt/xyz then I have done > mkdir /mnt/xyz/brick1 and set the brick as server1:/mnt/xyz/brick1 > > However I think this also fixes your problem, because if you try to mount > when the brick1 directory is not present, the glusterfsd process doesn't > start. The client doesn't get a good error message, but buried in the > server logs you can see what happened. I like that. And never would have thought of it. Thanks Brian! Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Unexpected Gluster behavior on startup without backing stores mounted
Hi, This may just be unexpected by me. But I'm wondering if there are safeguards against it, either that are in 3.1.4 that I haven't set up, or in subsequent versions. What happened: Despite dual power supplies and UPSes, two Gluster hosts managed to get suddenly shut down at once. On coming back up, one of the several mirrored Gluster shares, whose backing store is normally a logical volume mounted at /mnt/xyz on both hosts, didn't end up with that backing store mounted because someone (me) must have been distracted the day I should have added that to fstab. Here's the unexpected behavior: Gluster restored the nfs export based on the backing store. But without that backing store really mounted, it used /mnt/xyz, which at that point was a local subdirectory of /. This is not optimal. An error or warning or refusal to run would be preferable to presenting the export when the "real" backing store has gone missing. Due to particularities of the use case, the error wasn't immediately obvious (although those who remember my string of messages on a log-filling samba-related problem a week or so back, well, those were from this instance, but days later). Learn something new every day. Didn't visualize that Gluster could work at all without having the backing stores be unique, full partitions. Yet it mostly did (aside from the runaway log problem a week back) appear to be exporting the share normally. Things are back where they belong now, after I finally found the underlying cause. But should - and does current if not 3.1.x - Gluster make the attempt to use, and export, a share based on a backing store other than what it was originally set up on? It seems to me there are various ways it could have recognized there was something wrong in this case, logged that, and refused to export the share. Would have been highly useful if startup had a sanity check on this. Obviously the main error was between chair and keyboard here. Still, it seems the safety of Gluster in this sort of case can be improved, if it hasn't been already. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Samba > NFS > Gluster leads to runaway lock problem after many stable months
On Thu, Aug 30, 2012 at 06:12:46PM +0100, Brian Candler wrote: > On Thu, Aug 30, 2012 at 09:44:30AM -0400, Whit Blauvelt wrote: > > Given that my XFS Gluster NFS mount does not have inode64 set (although it's > > only 500G, not the 2T barrier beyond which that's evidently strictly > > required), and the likelihood that that led to yesterday's problem, what's > > the course to fix this? > > I very much doubt that was the problem. > > The whole reason that inode64 is not the default is so that inode numbers > are kept in the range less than 2^32, precisely so that ancient NFS clients > can deal with them. My reason for suspecting it is that Samba is doing 64-bit, and the NFS client is recent enough to work with 64-bit - whether Gluster as NFS server is doing 64-bit I don't know - and with XFS by default as 32-bit rather than 64 ... well the question is what caused Samba and lockd to get into such a noisey conflict. Given the smallish size of the Gluster NFS mount, seems odd for it to run beyond 32-bit space. Maybe it didn't. But if it _was_ a 64-bit 32-bit mismatch, setting XFS to be 64-bit might be advised. Then again, if I understand Samba's "posix locks" option right, having that off might avoid the problem, whether or not its a 64-bit to 32-bit mismatch. But I haven't found a clear description of the full implications of posix locks, beyond the suggestion that Samba has its own, independent file locking independent of the file system's, which it should use with this turned off. Does that independence go so far as to leave lockd out of it? Trying to guess the odds that turning that off will result in no repetition, since the system in question is the definition of "mission critical." Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Samba "posix locking" = yes or no for Gluster?
On Thu, Aug 30, 2012 at 10:42:01AM -0500, John Jolet wrote: > i had to turn off posix locking in order to get windows machines to > be able to write to the shares at all. Haven't seen that problem. Looking for background I found this presentation on SMB2 - the next version of Samba basically, incorporating Windows 8 features - according to which posix locking will be about the last thing to be implemented there: http://www.samba.org/~sfrench/presentations/smf-linux-collab-summmit-future-of-file-protocols-smb2.2.pdf Makes it sound as if posix locking's incredibly hard to implement right. Which could explain why there are many reports of situations where people find it necessary to turn it off in the current Samba, despite the assurance in the Samba docs that there should never be reason to do so. It also argues somewhat reasonably (even allowing for the presenter's bias) that going forward SMB2 is likely to progress much faster than NFS4 - and here we are with Gluster still at NFS3. Wonder if Gluster has/should have plans for including SMB2 support? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Samba "posix locking" = yes or no for Gluster?
Whether or not this is related to my particular catastrophe yesterday, there's inconsitent advice on whether Samba should have posix locking on or off when Gluster is involved. On the one hand, on is advised: http://gluster.org/community/documentation/index.php/Gluster_3.1:_Adding_the_CIFS_Protocol On the other, there's specific advice out there to turn it off, and reports of better behavior afterwards: http://comments.gmane.org/gmane.comp.file-systems.gluster.user/8494 Any further light on this issue will be appreciated. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Samba > NFS > Gluster leads to runaway lock problem after many stable months
Just a note: I see that "posix locking = on" (the default) in Samba was at least at one point a real problem for GFS2, as mentioned here: https://fedoraproject.org/wiki/Features/GFS2ClusteredSamba Not sure if that was ever resolved. I bring that up here because I'm wondering if it's similarly a problem now for Gluster. As I understand discussion elsewhere, turning posix locking off in Samba leaves locking still fully functional between CIFS clients, and at least partially functional between CIFS clients and actions on the files directly from the NFS-mounting system. Since in our case the same file being written to simultaneously over CIFS and directly might happen once a decade if ever, perhaps that should just be off anyway to decrease load and complications. But could it have been why Samba ran out of locks on us yesterday? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Samba > NFS > Gluster leads to runaway lock problem after many stable months
> Hoping someone has a clearer view of this. Are there respects in which > Gluster's NFS is 32-bit rather than 64? Or that xfs on a 64-bit system is? I'm seeing that XFS is said to require an inode64 mount option to not run into various problems, and that this if often considered an example of XFS not having been properly updated in recent years. See: http://www.linux-archive.org/centos/639945-xfs-inode64-nfs.html Given that my XFS Gluster NFS mount does not have inode64 set (although it's only 500G, not the 2T barrier beyond which that's evidently strictly required), and the likelihood that that led to yesterday's problem, what's the course to fix this? The XFS FAQ says it's possible to switch to inode64 and then switch back, if desired. Does the Gluster context complicate doing this? http://xfs.org/index.php/XFS_FAQ Also, that FAQ says some older, unspecified flavors of NFS can't handle inode64. I should assume Gluster's NFS isn't one of them, right? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Samba > NFS > Gluster leads to runaway lock problem after many stable months
Taking the message from Samba to be: > No locks available error. This can happen when using 64 bit lock offsets > on 32 bit NFS mounted file systems. Is Gluster 3.1.4's NFS itself 32-bit or 64-bit? From here: http://europe.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions#Application_fails_with_.22Invalid_argument.22_or_.22Value_too_large_for_defined_data_type.22_error. it looks like 3.1 is 64-bit NFS. The file systems in the 2 Gluster mounts are ext4 for one and xfs for the other. All the systems are themselves, and have always been, 64-bit. The one that's ext4 I had at some point added "posix locking = no" to smb.conf to avoid a problem that I don't recall clearly, but it wasn't the present one. The Samba docs advise that that setting should never be required, and that it's about coordinating the smb lock table with the posix one. But there are old reports on the Samba mailing list of it curing various things. I've added that for the xfs-based Gluster share now. Hoping someone has a clearer view of this. Are there respects in which Gluster's NFS is 32-bit rather than 64? Or that xfs on a 64-bit system is? Or ext4? I haven't been able to work out just which operation exhausted the locks, but it's more likely to have been on the xfs Gluster nfs mount, as more was going on there at the time things went bad. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Samba > NFS > Gluster leads to runaway lock problem after many stable months
Hi, I have a couple of Gluster 3.1.4 shares on the LAN NFS mounted at 192.168.1.242 to a couple of other systems. One of those systems in turn has access via Samba that includes those shares. This has been a stable system for a year. Today it went crazy, where the most immediate bad effect was an immense swelling of the logs of the system with both the Gluster shares mounted by NFS, and Samba access from other systems. That swelling was in the neighborhood of 5000 lines per second repeating approximately this: Aug 29 18:47:01 system2 smbd[7118]: [2012/08/29 18:47:01, 0] locking/posix.c:posix_fcntl_getlock(244) Aug 29 18:47:01 system2 smbd[7118]: on 32 bit NFS mounted file systems. Aug 29 18:47:01 system2 smbd[7118]: [2012/08/29 18:47:01, 0] locking/posix.c:posix_fcntl_getlock(252) Aug 29 18:47:01 system2 smbd[7118]: Offset greater than 31 bits. Returning success. Aug 29 18:47:01 system2 smbd[7118]: [2012/08/29 18:47:01, 0] locking/posix.c:posix_fcntl_getlock(242) Aug 29 18:47:01 system2 smbd[7118]: posix_fcntl_getlock: WARNING: lock request at offset 2886282524, length 1 returned Aug 29 18:47:01 system2 smbd[7118]: [2012/08/29 18:47:01, 0] locking/posix.c:posix_fcntl_getlock(243) Aug 29 18:47:01 system2 smbd[7118]: an No locks available error. This can happen when using 64 bit lock offsets Aug 29 18:47:01 system2 smbd[7118]: [2012/08/29 18:47:01, 0] locking/posix.c:posix_fcntl_getlock(244) Aug 29 18:47:01 system2 kernel: [14164811.340891] lockd: couldn't create RPC handle for 192.168.1.242 Not good. The /var partition filled totally, fast. Turned Samba off. Cleared logs out of the way. Restarted Samba, and everything's happy and not complaining. But obviously I need to avoid whatever set that off like the devil. It's an older Samba on that system, Version 3.0.28a, if that makes a difference. What are my options for avoiding this, both the core problem, and a repeat of anything throwing 5000 lines into the logs per second? The log level's set low in Samba. I suppose I could set up something to shut down syslog if /var gets to 99% full again, as a stop gap. There's 3X the space there that the logs normally fill. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Thu, Jul 19, 2012 at 05:16:24AM -0400, Krishna Srinivas wrote: > It was pretty confusing to read this thread. Hope I can clarify the > questions here. Thanks. I was confused. > The other discussion in this thread was related to NLM which has been > implemented in 3.3.0. This is to support locking calls from the NFS > clients to support fcntl() locking for the applications running on nfs > client. NLM server is implemented in glusterfs as well as kernel. NLM > server implemented in kernel is used by kernel-nfsd as well as > kernel-nfs-client. Hence if you have an nfs mount point, the > kernel-nfs-client automatically starts kernel NLM server. So if > glusterfs-nfs process is already running on a system (and hence it > also runs its own NLM server) and if you try to do "mount -t nfs > someserver:/export /mnt/nfs" on the same system it fails as > kernel-nfs-client won't be able to start kernel-NLM-server (because > glusterfs NLM server would have already registered with portmapper for > NLM service and hence kernel-NLM-server registration with portmapper > fails). Workaround is "mount -t nfs -o nolock someserver:/export > /mnt/nfs" if you really want to have an nfs mount on the same machine > where glusterfs-nfs process is running. So, if you want to run both at once, only one can lock. Is the architecture of NLM such that there could never be a single NLM server for both Gluster and kernel (whether that single server be Gluster's or kernel's)? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
On Wed, Jul 18, 2012 at 01:56:04AM -0400, Rajesh Amaravathi wrote: > As to whether we can disable parts of kernel NFS (I'm assuming e.g NLM), I > think > its not really necessary since we can mount other exports with nolock option. > If we take out NLM or disable NLM at the kernel level, then every time we need > NLM from kernel, we need to recompile the kernel/have a secondary kernel with > NLM > and reboot, much tedious than simply killing Gluster/fuse NFS and after > kernel NLM's > work is done, restart Gluster/fuse NFS. My $0.02 :) Since there's good reason to want locking with Gluster NFS, wouldn't the answer be to add code to kernel that would allow the kernel's locking to be turned off and on in the standard way - a file called something like kernel_nfs_locking that would hold either a 0 or 1? Obviously the kernel's NFS support was built on the assumption that no one with kernel NFS available would want to run userland NFS. Gluster shows that assumption is wrong. So wouldn't it be sensible for someone on the Gluster team to be submitting kernel patches to fix this oversight? Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Sorry my question was too vague. What I meant to ask is if it is possible, since there is a conflict between the locking requests from the kernel's NFS and from Gluster/fuse's NFS, that the kernel might be compiled so with some or all of its NFS support disabled, so that then Gluster/fuse NFS-locking would work. Perhaps longer term there needs to be a way to have the kernel shut its NFS locking attempts off, just if there is a userland NFS such as Gluster's running. Meanwhile can enough of NFS be taken out of a custom kernel to allow Gluster to lock? Thanks, Whit On Tue, Jul 17, 2012 at 08:03:03AM -0400, Rajesh Amaravathi wrote: > it should be possible to mount another kernel export with -o nolock option and > compile kernel on it. I'm just guessing when you mount with nolock option, > we are mounting for mostly read purposes and not for critical writes. > > Regards, > Rajesh Amaravathi, > Software Engineer, GlusterFS > RedHat Inc. > > - Original Message - > From: "Whit Blauvelt" > To: "Rajesh Amaravathi" > Cc: "David Coulson" , "Gluster General Discussion > List" > Sent: Monday, July 16, 2012 9:56:28 PM > Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable > or not? > > Say, is it possible to compile a kernel without whatever part of its NFS > support competes with Gluster's locking? > > Whit > > On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: > > I hope you do realize that two NLM implementations of the same version > > cannot operate simultaneously in the same machine. I really look forward > > to a solution to make this work, that'd be something. > > > > Regards, > > Rajesh Amaravathi, > > Software Engineer, GlusterFS > > RedHat Inc. > > > > - Original Message - > > From: "David Coulson" > > To: "Rajesh Amaravathi" > > Cc: "Tomasz Chmielewski" , "Gluster General Discussion > > List" > > Sent: Friday, July 13, 2012 5:28:04 PM > > Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - > > reliable or not? > > > > Was that introduced by the same person who thought that binding to > > sequential ports down from 1024 was a good idea? > > > > Considering how hard RedHat was pushing Gluster at the Summit a week or > > two ago, it seems like they're making it hard for people to really > > implement it in any capacity other than their Storage Appliance product. > > > > Luckily I don't need locking yet, but I suppose RedHat will be happy > > when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) > > > > David > > > > On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: > > > Actually, if you want to mount *any* nfs volumes(of Gluster) OR > > > exports (of kernel-nfs-server), you cannot do it with locking on > > > a system where a glusterfs(nfs process) is running(since 3.3.0). > > > However, if its ok to mount without locking, then you should be > > > able to do it on localhost. > > > > > > Regards, > > > Rajesh Amaravathi, > > > Software Engineer, GlusterFS > > > RedHat Inc. > > > > > > - Original Message - > > > From: "David Coulson" > > > To: "Tomasz Chmielewski" > > > Cc: "Rajesh Amaravathi" , "Gluster General Discussion > > > List" > > > Sent: Friday, July 13, 2012 3:16:38 PM > > > Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - > > > reliable or not? > > > > > > > > > On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: > > >> Killing the option to use NFS mounts on localhost is certainly quite > > >> the opposite to my performance needs! > > >> > > > He was saying you can't run kernel NFS server and gluster NFS server at > > > the same time, on the same host. There is nothing stopping you from > > > mounting localhost:/volume on all your boxes. That is exactly how our > > > 3.2.5 and 3.3.0 environments access volumes for the performance reasons > > > you identified. > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?
Say, is it possible to compile a kernel without whatever part of its NFS support competes with Gluster's locking? Whit On Fri, Jul 13, 2012 at 08:14:27AM -0400, Rajesh Amaravathi wrote: > I hope you do realize that two NLM implementations of the same version > cannot operate simultaneously in the same machine. I really look forward > to a solution to make this work, that'd be something. > > Regards, > Rajesh Amaravathi, > Software Engineer, GlusterFS > RedHat Inc. > > - Original Message - > From: "David Coulson" > To: "Rajesh Amaravathi" > Cc: "Tomasz Chmielewski" , "Gluster General Discussion List" > > Sent: Friday, July 13, 2012 5:28:04 PM > Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable > or not? > > Was that introduced by the same person who thought that binding to > sequential ports down from 1024 was a good idea? > > Considering how hard RedHat was pushing Gluster at the Summit a week or > two ago, it seems like they're making it hard for people to really > implement it in any capacity other than their Storage Appliance product. > > Luckily I don't need locking yet, but I suppose RedHat will be happy > when I do since I'll need to buy more GFS2 Add-Ons for my environment :-) > > David > > On 7/13/12 7:49 AM, Rajesh Amaravathi wrote: > > Actually, if you want to mount *any* nfs volumes(of Gluster) OR > > exports (of kernel-nfs-server), you cannot do it with locking on > > a system where a glusterfs(nfs process) is running(since 3.3.0). > > However, if its ok to mount without locking, then you should be > > able to do it on localhost. > > > > Regards, > > Rajesh Amaravathi, > > Software Engineer, GlusterFS > > RedHat Inc. > > > > - Original Message - > > From: "David Coulson" > > To: "Tomasz Chmielewski" > > Cc: "Rajesh Amaravathi" , "Gluster General Discussion > > List" > > Sent: Friday, July 13, 2012 3:16:38 PM > > Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - > > reliable or not? > > > > > > On 7/13/12 5:29 AM, Tomasz Chmielewski wrote: > >> Killing the option to use NFS mounts on localhost is certainly quite > >> the opposite to my performance needs! > >> > > He was saying you can't run kernel NFS server and gluster NFS server at > > the same time, on the same host. There is nothing stopping you from > > mounting localhost:/volume on all your boxes. That is exactly how our > > 3.2.5 and 3.3.0 environments access volumes for the performance reasons > > you identified. > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster v 3.3 with KVM and High Availability
On Wed, Jul 11, 2012 at 05:56:00PM +, Barlow, Jim D wrote: > As far as fencing goes, I have done nothing. I'm manually as > carefully as I can manage using virt-manager. I've already accidentally > started the same VM on two bricks. Watch your autostart settings on the > VMs :-) There's another, much cruder way to prevent running from two systems at once. If you create a directory /etc/libvirt/hooks and then put a file "qemu" in it, that file will be run each time you start a VM (and a few other events - depends on your libvirt version which). The script gets passed the VM name and the operation (e.g. "start"). So I have a script there that first tries to ping the VM, and if that fails runs virsh via ssh on the other system to see if the VM is either "shut off" or "paused" (necessary for live migration) there. So if the VM's not pingable (via the LAN interface) and either the other host isn't pingable either (via a crossover between hosts) or it is and virsh confirms the VM is shut off or paused there, then I have virsh start the VM locally. Obviously there will be better ways to handle this. But it's better than nothing. After creating the hooks directory, libvirt requires a restart to see it. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] 3.1.5 to 3.3.0 upgrade question
Hi, We've several systems running 3.1.5 since it was new, happily without incident. The time has come to upgrade to 3.3.0, so I want to make sure I have the instructions right to minimize downtime. From here: http://gluster.org/community/documentation//index.php/Main_Page I'm sent to here: http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3/ which has as its most recent comment: Hi, I tried to upgrade from 3.2.1 to 3.3.0. It looked fine at first sight. But I found that adding new 3.3.0 nodes to the upgraded cluster fails. It’s probably because 3.2.1′s volume config lacks “username/password” authentication entries in “info” and “…vol”. My workaround is: Which has me thinking If this is a problem for 3.2.x, it's probably a problem for 3.1.x. Since the workaround suggested isn't fully specified -- "Modify above files by hand (adding username/password auth entries.)" doesn't lead to a complete picture of what the modifications should look like -- I'd like to ask Is it the general case that this step will be needed? If so, what's the complete description of what to do there? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] AutoFS NFS Problems
Don't know if "proto" is a synonym. But in my working auto.fs mounts I have "mountproto=tcp" not proto. Whit On Sun, Jun 10, 2012 at 04:33:50AM -0500, Chris LeBlanc wrote: > Hi friends, > > I'd like to use autofs but I had mixed luck. It seems as thought it mounts > itself instead of the gluster volume which leads me to believe it's not using > the correct NFS protocol. > > auto.master > /- /etc/auto.srv > > auto.srv > /srv -fstype=nfs,nfsvers=3,rw,proto=tcp sld-web-3:/srv > > I've tried all sorts of variations for options in auto.srv, tried > adding MOUNT_NFS_DEFAULT_PROTOCOL=3 to auto.master (not sure if that's where I > should be putting that), and even saw somewhere to add vers=3 to the right of > the auto.master direct mount . > > I'm using cluster 3.2.5 and Ubuntu Precise. > > Many thanks for your help. > > -- > Chris LeBlanc > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Striped replicated volumes in Gluster 3.3.0
> I'm sorry if the release didn't address the specific features you need, > and I'm sorry if we gave the impression that it would. Our additional > features for 3.3 were always pretty clear, or so I thought. If you can > find any statements from the past year that were misleading, I would be > happy to address them directly, but your statement above was a bit vague. John Mark, Please consider my comments in the context of Amar's comment, which I quoted above it: On Tue, Jun 05, 2012 at 11:09:39AM +0530, Amar Tumballi wrote: > I saw that mail and I agree that the target of 3.3.0 was to make > glusterfs more stable and get the features which would make Virtual > machine hosting on GlusterFS a possibility by 3.4.0 time-frame. Does that not seem to say that virtual machine hosting is not really to be a possibility before the 3.4.0 time-frame? From subsequent responses, including yours, Amar likely misspoke. Excuse me for responding to a message that seemed to be saying that Gluster won't be fully ready for VM hosting before 3.4.0. Regards, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Striped replicated volumes in Gluster 3.3.0
On Tue, Jun 05, 2012 at 08:13:37AM -0700, Anand Avati wrote: > Whit, > There has been no drop in goal. There are many parts to "supporting a VM > workload". The goals listed for 3.3 are definitely met - to make VMs work good > enough (fix self-heal locking issues, some FUSE upstream work for supporting > O_DIRECT etc.) However, for 3.4 we have bigger goals of supporting VM image > use > case in a much better way - libglusterfsclient integration for QEMU etc. This > was what Amar was referring to. I hope I clarified your doubt, and apologies > for the confusion. Thanks for the clarification, Avanti. So if "VMs work good enough" now, the recommendation is that in general Gluster is ready for production work with VMs? Performance should be acceptable, if not always spectacular? Or should VM use be considered more in the beta area still, until 3.3.1 or whatever? I know there's no substitute for testing it in a particular environment. Just trying to get the general overview before committing to that. Regards, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Striped replicated volumes in Gluster 3.3.0
On Tue, Jun 05, 2012 at 11:09:39AM +0530, Amar Tumballi wrote: > >I tried it to host Virtual Machines images and it didn't work at all. Was > > hoping to be able to spread the IOPS more through the cluster. That's > > part of what I was trying to say on the email I sent earlier today. > > I saw that mail and I agree that the target of 3.3.0 was to make > glusterfs more stable and get the features which would make Virtual > machine hosting on GlusterFS a possibility by 3.4.0 time-frame. Odd thing is, there have been statements going back a year saying that Gluster would be ready for VM hosting in the 3.3 version. On the one hand, it's appreciated that Gluster is pursuing ambitious goals. On the other, when a goal is dropped from a release that many people are waiting on a long time, specifically because of that advertised goal, it would be common courtesy (and good business practice) to announce the change in target in all the places where it had previously been stated. Like this list. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Gluster in cloud environments
Hi, My experience with Gluster has been entirely with local hardware. And the discussion here has been entirely about such use. But I the Gluster docs hint of Gluster use within cloud environments - or at least in Amazon's. Note this is a different issue than using Gluster as backing storage for a private clould. (Exciting that it might finally be good for VMs - I look forward to testing that - but it's a different issue.) Currently on the Gluster site there's mention of the Red Hat Cloud Appliance for Amazon, with a link to this article: http://www.wired.com/cloudline/2012/02/red-hat-cloud-appliance/ The article mentions the usefulness of Posix compliance. That and replication features have obvious use. But I'm not asking about an appliance here, nor Amazon's cloud. What I'm wondering about is uses of Gluster based on a generic VM (RH, CentOS, Ubuntu, Debian ... whatever), on top of brand X clouds. In this instance one by a backwater provider that promises: VMware's vSphere 4 virtualization platform Cisco Unified Computing System (UCS) IBM XiV storage Mezeo Cloud Storage platform DoubleTake DR and replication software Avoiding DoubleTake, an expensive option, could be a real advantage with Gluster, since we'd do well to maintain fail-back capability to our current office if the whole cloud thing blows up - where we've already got (older) Gluster handling vital parts of our storage. But I've no clear picture of how vSphere fits with XiV fits with Mezeo - and whether Gluster can be a useful compliment there. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Data checksumming
I'm told it doesn't, but may in the future. Whit On Fri, Feb 17, 2012 at 01:32:25PM -0500, Matty wrote: > I'm pretty new to Gluster, though I really what like I've seen and > played with so far!! Does anyone happen to know if Gluster checksums > data that is written to disk? I've used ZFS quite a bit in the past, > and dig the fact that it checksums every block of data that is written > to disk. This allows the file system to self-heal itself when a > checksum doesn't match (this assumes you are utilizing mirroring or > RAIDZ), and you can use the scrub feature to check the integrity of > your data. Here is one example of this: > > http://prefetch.net/blog/index.php/2011/10/15/using-the-zfs-scrub-feature-to-verify-the-integrity-of-your-storage/ > > I'm curious if something similar exists for Gluster, or if a > translator could be developed to do this? I've been reading Jeff > Darcy's translator 101 series, and it seems like this wouldn't be > super difficult to add. Curious what others think. > > - Ryan > -- > http://prefetch.net > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster behavior
Yes, it's correct behavior. Only writes and edits through the gluster mount are handled immediately by gluster. Whit On Wed, Mar 28, 2012 at 10:30:03AM +0800, nhadie ramos wrote: > Hi All, > > I am just starting to do some tests on gluster, i have 2 servers which > i setup as replicate: > > > server1: > > /dev/md0 mounted on /myharddisk > > created a gluster volume called testvolume > mounted the glustervolume as /testgluster > txtfile.txt on /myharddisk > > server2: > > /dev/md0 mounted on /myharddsik > created a gluster volume called testvolume > mounted the glustervolume as /testgluster > txtfile.txt on /myharddsik > > editedt txtfile.txt on server1:/testgluster changes took effect on > server1:/testgluster, server2:/testgluster and serv2:/myharddisk > (expected result) > edited txtfile.txt on server1:/myharddsik changes did not replicate on > server1:/testgluster and server2:/testgluster unitl i did a self-heal > but server2:/myharddisk did not update, unless i delete the existing > file on it. > > Is this the correct behavior? Thanks in advance! (sorry if the email > confuses you) > > Regards, > Ron > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] question re. current state of art/practice
On Wed, Feb 15, 2012 at 08:22:18PM -0500, Miles Fidelman wrote: > i. Is it now reasonable to consider running Gluster and Xen on the same > boxes, without hitting too much of a performance penalty? What's in the hardware? What kind of loads are you expecting it to handle? For a bunch of lightly-to-moderately-used services, I have two modest servers, each with 2 8-core AMD CPUs and big-but-slow SAS drives in RAID5, that are running KVM VMs that happen to be each on dedicated DRBD partitions mirrored across the servers, while the servers are also providing a couple of Gluster filesystems primarily mounted from elsewhere by NFS. Which is certifiably insane. Except it works. I'm sure I would be in serious trouble if any of the services were high load, or if the demands on the Gluster file systems were greater - especially in combination with the DRBD traffic. But it's all been very well behaved. So there are at least some cases where VMs and Gluster (although note the VMs are not _on_ Gluster in this case) can share hardware without coming close to melting it down. A lot of cores helps. A situation where nothing's seriously pounding any of the file systems helps. And there's a reason I'm not running the VMs on Gluster (yet). I experimented with that on an older version, and as noted by others it wasn't suited for it. The newest version is reputed to be far improved, but is only in beta. In other words having file systems on Gluster that are mounted by VMs, which are themselves on the same systems but not themselves stored on Gluster, given not-too-power-hungry stuff, you may be fine. To put the VMs themselves on Gluster ... probably better to wait until 3.3 is out of beta. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Exorbitant cost to achieve redundancy??
On Tue, Feb 14, 2012 at 08:39:31AM +, Brian Candler wrote: > On Mon, Feb 13, 2012 at 11:45:10PM -0500, Whit Blauvelt wrote: > > If Gluster gets the geo-rep > > thing working right, it'll be the low-cost solution there too. > > Do I hear implied from that that geo-rep *doesn't* work right yet? Have you > some pointers on what its limitations are? This is something I was thinking > of deploying (but haven't got as far as testing) Sorry, I implied too much. Haven't tested. Just a second-hand impression from other reports here. Since there've been fewer complaints lately, for all I know they've already got the kinks out. It would be nice to see a few reports of success. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Exorbitant cost to achieve redundancy??
On Mon, Feb 13, 2012 at 09:18:34PM -0600, Nathan Stratton wrote: > > On Mon, 13 Feb 2012, Whit Blauvelt wrote: > > >You don't have to leave all your redundancy to Gluster. You can put Gluster > >on two (or more) systems which are each running RAID5, for instance. Then it > >would take a minimum of 4 drives failing (2 on each array) before Gluster > >should lose any data. Each system would require N+1 drives, so double your > >drives plus two. (There are reasons to consider RAID other than 5, but I'll > >leave that discussion out for now.) > > That's great with a few nodes, but the problem is with Gluster and > many notes. We run all our notes with RAID6, but the more nodes you > have the more likely you will have a node failure. This is what I > think Jeff was worried about. Sure, nodes can fail. But Jeff's subject was "Exorbitant cost to achieve redundancy??" That was a nice contrast to those who've found Gluster far cheaper than solutions they'd gone with before. Your systems could always be hit by natural or man-made disaster at any location too. Seems to be more of the former lately, and there's plenty of threat of the latter too. Remote DR should go without saying if the data's valuable. If Gluster gets the geo-rep thing working right, it'll be the low-cost solution there too. If someone has a design to get redundancy more cost-effectively, that'd be great. But as far as I can see - and I'm only in the trenches here, not on the mountain top - as long as we're essentially on drives, we're going to need a RAIDed set locally, synchronously mirrored on anther RAIDed set by whatever tech you like (e.g. Gluster), and a third set of possibly slower, cheaper drives far away, with your data asynchronously mirrored for DR. No equivalent solution is going to be cheaper than the collection of physical drives. I'm sure my thinking's too conventional. Please suggest better. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Exorbitant cost to achieve redundancy??
You don't have to leave all your redundancy to Gluster. You can put Gluster on two (or more) systems which are each running RAID5, for instance. Then it would take a minimum of 4 drives failing (2 on each array) before Gluster should lose any data. Each system would require N+1 drives, so double your drives plus two. (There are reasons to consider RAID other than 5, but I'll leave that discussion out for now.) As for "reasonable storage costs," have you priced the alternatives? Best, Whit On Mon, Feb 13, 2012 at 04:15:16PM -0800, Jeff Wiegley wrote: > I'm trying to justify a GlusterFS storage system for my technology > development group and I want to get some clarification on > something that I can't seem to figure out architecture wise... > > My storage system will be rather large. Significant fraction of a > petabyte and will require scaling in size for at least one decade. > > from what I understand GlusterFS achieves redundancy through > replication. And from the documentation: Section 5.5 Creating > Distributed Replicated Volumes the note says "The number of bricks > should be a multiple of the replica count for a distributed replicated > volume." > > Is this telling me that if I want to be able to suffer 2 bricks failing > that I have to deploy three bricks at a time and the amount of space > I wind up with available is essentially equal to only that provided > by a single brick? > > In other words... GlusterFS TRIPLES all my storage costs to provide > 2 brick fault tolerance? > > How do I get redundancy in GlusterFS while getting reasonable > storage costs where I am not wasting 50% of my investment or > more in providing copies to obtain redundancy? > > Thank you. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Would difference in size (and content) of a file on replicated bricks be healed?
On Mon, Feb 06, 2012 at 08:28:55PM +0530, Pranith Kumar K wrote: > Modifying the data directly on the brick in a replica > pair(Brick A, Brick B) is a bad thing because it will not be > possible to decide in which direction to heal the contents of the > file B -> A or A -> B. If the file is modified from the mount point > then the glusterfs client marks the extended attributes so that it > knows in which direction to heal the files. Thanks Pranith. I take it then that if there is corruption of a file due to, say, a degraded disk on one side of a replica pair that gluster has no way to spot and heal this? There's nothing built in (such as a stored CRC value) by which it could conclude "Brick A's file is corrupted, so I'll restore it from Brick B"? I'm not say there should be. It could be useful if there was, but might also require too much overhead and be outside gluster's proper scope. I'm just curious. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Would difference in size (and content) of a file on replicated bricks be healed?
On Sun, Feb 05, 2012 at 10:46:53PM +0100, Ove K. Pettersen wrote: > >>* Append to file on one of the bricks: hostname>>data.txt > >Again, through a gluster/nfs mount, or a local mount? > This was done directly to a brick (local mount) to try to simulate > some disk-problems. > Appending to the file would keep the extended attributes. Gluster > should still handle the file > as its own. This is an open question. I don't know the answer. But I wonder. Is there any real-world problem that would result in the same pattern? That is, you've changed a file outside of gluster, not in any way that results in an actually-corrupt file, but instead a file that in terms of the way you wrote to it is still totally good. So how often is some random disk corruption going to happen that results in a file which is in fact fully legitimate? > But my goal is to simulate something going wrong on a disk. > That is not predicted events done through a glusterfs-mounted > filesystem but rather happens directly to a brick. A better simulation might be through using a hex editor to change a few bytes in the file, without touching attributes or inodes at all. Haven't tried it. Don't know whether gluster would spot the corruption and fix it or not. But that would be a lot closer to bits flipping from hardware errors. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Would difference in size (and content) of a file on replicated bricks be healed?
Not sure if you're asking your questions precisely enough. The clues may be in your inclusions, but I'm not going to read all that to figure it out so I'll ask directly: > Short description of my test > > * 4 replicas on single machine > * glusterfs mounted locally > * Create file on glusterfs-mounted directory: date >data.txt Did you write it through a gluster (or nfs) mount of the gluster file system, or sneak around behind it with a direct local mount of the partition? > * Append to file on one of the bricks: hostname >>data.txt Again, through a gluster/nfs mount, or a local mount? > * Trigger a self-heal with: stat data.txt Again, stat'ing it on a gluster/nfs mount, or a local mount? > => Modified file on the brick is still different from the three > other bricks. Again ... well you know. > Q1: Is the modified file supposed to get corrected in my case? > > Q2: Is my test-case invalid for what gluster is supposed to handle? If you're writing to files through mounts which gluster isn't handling, you'll get inconsistent stuff. Eventually gluster should catch up. But by design you should do everything through gluster. (An exception may be if you want a faster local file read ... anything that writes or touches the file should be through gluster in any case.) Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Hanging writes after upgrading "clients" to debian squeeze
On Sun, Feb 05, 2012 at 07:36:55PM +, Brian Candler wrote: > On Sun, Feb 05, 2012 at 08:02:08PM +0100, Stefan Becker wrote: > >After the debian upgrade I can > >still mount my volumes. Reading is fine as well but it hangs on writes. > > Could it be that on the post-upgrade machines one brick is reachable but not > the other? Compare iptables rules between the pre-upgrade and post-upgrade > machines? Compare tcpdump or ntop between them? If you can, try dropping iptables out of the picture entirely. If you are running it, and have it logging what it drops, the docs say "Ensure that TCP ports 111, 24007,24008, 24009-(24009 + number of bricks across all volumes) are open on all Gluster servers. If you will be using NFS, open additional ports 38465 to 38467." So I'd check your logs to see if iptables is dropping any traffic to/from the IPs in question on those ports. Or us "netstat -tc" while doing some file operations, and you should see the traffic on the IPs/ports. Another utility to see the same thing is "iptraf." Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Best practices?
On Tue, Jan 24, 2012 at 12:50:58PM -0500, John Mark Walker wrote: > True. Also, note that XFS is the recommended disk FS, although Ext3/4 are > certainly still supported and will continue to be so. Are the reasons listed somewhere? It used to be the opposite. From http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_Checking_GlusterFS_Minimum_Requirements: File System Requirements Gluster recommends Ext4 (for Linux kernel 2.6.31 or higher) and Ext3 (for all earlier versions) when formatting the disk sub-subsystem. Any other POSIX compliant disk file system such as XFS or ZFS may also work, but has not been tested widely. The 3.2 doc has modified that somewhat (http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Checking_GlusterFS_Minimum_Requirements): File System Requirements Gluster recommends Ext4 (for Linux kernel 2.6.31 or higher) and Ext3 (for all earlier versions) when formating the disk sub-subsystem. For workloads involving huge files, Gluster recommends XFS file system. Any other POSIX compliant disk file system, such as ZFS may also work, but has not been tested widely. So is the preference now that even for workloads _not_ involving huge files, XFS is better? For non-huge-file systems is Ext4 more likely to break, or suffer in performance speed? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Auto-mount on boot failing, works later
Anything you can do with fstab you can do with autofs, except of course that it requires the system to be up first, while fstab can bring the basic system up initially. Autofs mounts are more durable, in that they will be recreated as necessary. I'm not using the most current Gluster, so can't speak to any changes in volfiles. Your line: 127.0.0.1:/shared /var/lib/sitedata nfs defaults,_netdev 0 0 would translate to autofs as something like: /var/lib/sitedata -fstype=nfs,mountproto=tcp 127.0.0.1:/shared providing that was in "/etc/auto.nfs" and /etc/auto.master had something like: /- /etc/auto.nfs I'm taking that example from an older box which expects NFS3. IIRC on a more recent OS you have to specify also to use NFS3 for Gluster rather than NFS4. If mounting as gluster (unless some change in volfiles now negates this, works for 3.1.4 anyhow): /var/lib/sitedata -fstype=glusterfs 127.0.0.1:/shared There are other options, but that should be enough. Autofs is packages for every Linux distro AFAIK. Whit On Tue, Jan 17, 2012 at 06:58:05PM +0100, Marcus Bointon wrote: > On 16 Jan 2012, at 18:59, Whit Blauvelt wrote: > > > I find it well worth it to use autofs for anything beyond the most basic > > local mounts. That is, mount your backing store with fstab, but your gluster > > mounts (whether local or remote) with autofs. > > > > If there's an argument against autofs other than the minor setup involved, I > > don't know what it would be. > > I'd never heard of autofs before. The gluster docs on using it are > confusing/old/incomplete - current gluster doesn't seem to use volfiles > and the suggested config for local/remote server doesn't include a volume > name, nor allow you to target a single volume, only a group of them in a > folder. Are there some better docs (if this is the only way that actually > works!)? > > I read that _netdev only applies to nfs mounts, so I switched to the nfs > client (which is probably more suitable for what I'm using gluster for > anyway), however that's even worse - it hangs the whole boot process > requiring a boot into single user mode to recover it! I see this error on > the console when it hangs: > > mount.nfs: DNS resolution failed for 127.0.0.1: Name or service not known > > My line in fstab is: > > 127.0.0.1:/shared /var/lib/sitedata nfs defaults,_netdev 0 0 > > It works perfectly when mounted with mount -a after boot. > > I read in the RHEL bug tracker that nfs looking up localhost (bypassing > hosts file) is a bug, however, it seems to remain unfixed in Ubuntu Lucid. > > Marcus > -- > Marcus Bointon > Synchromedia Limited: Creators of http://www.smartmessages.net/ > UK info@hand CRM solutions > mar...@synchromedia.co.uk | http://www.synchromedia.co.uk/ ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Auto-mount on boot failing, works later
On Mon, Jan 16, 2012 at 06:32:00PM +0100, Marcus Bointon wrote: > You can see it's failing to connect to the local node, and then recording > the gluster server starting a short time later. I'm pointing at localhost > to get the share point rather than either of the external IPs. Is there > some way to make it wait for gluster to be up? Alternatively, is it safer > to point the mount at the other node on the basis that it's unlikely to be > down at the same time as the current node? OTOH it's nice fro each node to > be self-sufficient. I find it well worth it to use autofs for anything beyond the most basic local mounts. That is, mount your backing store with fstab, but your gluster mounts (whether local or remote) with autofs. If there's an argument against autofs other than the minor setup involved, I don't know what it would be. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] What are the most stable iterations of Gluster so far?
Just a general question to the group: What's the current feeling about the most stable releases so far? I've got several different instances of 3.1.4 gluster clusters running in undemanding circumstances for many months without incident (aside from the known group-ownership issues). Since my priority isn't new features, but the most dependable roll-out of the core, I keep wondering when and if I should upgrade. At some point geo-rep and good VM support will matter to me, but that will be in other contexts on fresh storage. For the core features, has there been anything to strongly motivate moving ahead from 3.1.4? Thanks, Whit On Thu, Jan 05, 2012 at 05:31:41PM +, Dan Bretherton wrote: > I found .*.gfs* files in a volume after rebalance with version > 3.2.5, so it doesn't just happen with 3.2.4 or below. I reported > this and other problems with migrate-data separately in another > thread (http://gluster.org/pipermail/gluster-users/2012-January/009343.html). > Is anybody else having problems with rebalance...migrate-data in > version 3.2.5? > > -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] small cluster question
On point 2, I've been running an older version of Gluster for some months on several pairs of systems which are also serving other functions. In one case they're running multiple VMs (replicated by individual DRBD mirrorings) while also providing replicated Gluster storage used (via NFS) by other systems. I'm sure this would be insane if this all added up to a heavy load in any respect, but in this case these are all lightly-used services, and there have been no complaints. DRBD and Gluster replication share a dedicated cable between the systems, so they're competing, but at least not with the rest of the LAN. Experimenting with VMs stored on older Gluster versions wasn't encouraging, thus the DRBD. I look forward to 3.3's release. As for running VMs on the same servers hosting Gluster (and other) storage - works fine for me. YMMV. It's going to depend on how hard the VMs are working, and how heavy they're hitting the storage. Whit On Tue, Nov 01, 2011 at 08:22:23PM -0500, Gerald Brandt wrote: > Hi, > > I can answer point 1. GlusterFS 3.3 (still in beta), does finer locking > during self-heal, which is what the VM images like. > > Gerald > > - Original Message - > > From: "Miles Fidelman" > > > > 2. It looks like the standard Gluster configuration separates storage > > bricks from client (compute) nodes. Is it feasible to run virtual > > machines on the same servers that are hosting storage? (I'm working > > with 4 multi-core servers, each with 4 large drives attached - I'm > > not > > really in a position to split things up.) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 32 bit and 64 bit
On Tue, Nov 01, 2011 at 01:01:09PM -0700, Brian Pontz wrote: > I'm wondering why the 32 bit binary has no trouble on a standard nfs mount but > it does have trouble on the gluster mount with the gluster client. Isn't NFS 32 bit throughout? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] FW: Save the Date: Building New File Systems with GlusterFS Translators
What's the technical level required to build translators? What languages are appropriate? Is this a capability that is likely to be useful to an advanced sysadmin, or most likely only of real interest to those with more serious software engineering resources to devote to it? Whit On Thu, Oct 20, 2011 at 07:00:25PM +, John Mark Walker wrote: > FYI... this will be an interesting webinar. If you've wanted to learn how > GlusterFS translators work, here's your opportunity. > > Session to be led by Jeff Darcy, who has used the translator subsystem > significantly for building HekaFS (fka CloudFS). > > -JM > > > ━━━ > From: Gluster Webinars [heat...@gluster.com] > Sent: Thursday, October 20, 2011 11:33 AM > To: John Mark Walker > Subject: Save the Date: Building New File Systems with GlusterFS Translators > > Gluster > Register Today for the Gluster for Geeks Webinar Series! > GlusterFS is a stackable filesystem with a layered approach that > allows developers to easily build new extensions and add new > layers to the base filesystem. Join Red Hat engineers Jeff Darcy > Webinar: Building New File > and Anand Avati as they walk through the GlusterFS architecture >Systems with GlusterFS > and demonstrate how to get started building new translators. > Translators > GlusterFS translator contributions in the past have included > such things as data encryption, IMAP, FTP, a trash can, and > Wednesday, Oct 26 > much more. In fact, much of what is considered core > at 9am PT / 12pm ET > functionality of GlusterFS is implemented via translators. > >Click here to register! > GlusterFS is not just a filesystem, but a framework for new > filesystems implemented as translators. Join us for this > webinar, and you too can build filesystems on GlusterFS. > > Topic: Building New Filesystems with GlusterFS Translators > Date: Wednesday, October 26 > Time: 9 AM PT / 12 PM ET > > Speakers: > Jeff Darcy, Principal SW Engineer, Red Hat > Anand Avati, Senior Principal Engineer, Red Hat > > This will be a 60 minute webinar with the first 30 minutes dedicated to > content > and the last 30 minutes scheduled for Q&A. We look forward to seeing > you > there! > > Register Now > > For more information, visit us at www.gluster.com | sa...@gluster.com > | 1.800.805.5215 > [1] > > > > Gluster > 640 W. California Ave > Sunnyvale, CA 94086 > > Unsubscribe from email communications > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] configuration questions & advice
On Mon, Oct 10, 2011 at 10:56:11AM -0400, Miles Fidelman wrote: > GlusterFS stands out as the package that seems most capable of > supporting this (if we were using KVM, I'd probably look at Sheepdog > as well). The Gluster folks have said it's best to wait for 3.3's improved VM support. My understanding is current versions can get bogged down in this context. Not that there aren't some using them this way. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster performance
What's Apache doing in terms of file writing? Why not have it read from one domain, and write to another? Then you could have a DocumentRoot for the domain to be only read from which points to the backing store on each server, and a DocumentRoot for the domain that's used for writing files which points to the Gluster mount. You'd want separate logs in any case, for the separate Apache instances. Since Apache only writes to files if you set up scripting specifically to do that, it's trivial to have all your writes go to a separate domain or subdomain. Just have every GET or POST that can result in a write to a file go to the domain with a DocumentRoot on the Gluster share, and all your GETs that just return static content go to the domain whose DocumentRoot is the backing store. If some of your GETs or POSTs don't write to files, but change a backing database, put that database in a different area and use its native mirroring capacity. Whit On Tue, Sep 27, 2011 at 09:50:36PM +0100, Lone Wolf wrote: > Kind of bizzare I have to agree, but as far as I can understand from the docs > geo-replication is one way, and I need two way because there can be writes on > both sides, rsync was an option but I would have to cron it and each side at a > time, and from my understanding could give me problems of files either going > missing or not being deleted. > Using the local for read is an option but since the files are being read and > written by the same application (apache) I see no way to split that. > What is more weird is that I managed to get it to read just one side for a > moment, after restarting gluster because of some adjustments, it started > reading both nodes again, even restoring config backups had no effect. > > On Tue, Sep 27, 2011 at 1:37 PM, Whit Blauvelt wrote: > > Okay, what you're doing is a bizarre use of Gluster anyway. (They'd say > you > should use the "georeplication" instead - at which point a rsync run from > cron will get you to about the same place without using Gluster at all.) > But > you can have local systems read from the backing store without corrupting > the Gluster mirroring. Your writes all have to go through the Gluster > mounts, but your reads don't. > > Whit > > On Tue, Sep 27, 2011 at 01:22:24PM +0100, Lone Wolf wrote: > > Just checked it is mounted with noatime > > > > No dia 27 de Set de 2011 12:56, "Whit Blauvelt" < > whit.glus...@transpect.com> > > escreveu: > > > On Tue, Sep 27, 2011 at 12:15:50PM +0100, Lone Wolf wrote: > > > > > >> So I am assuming gluster is distributing the reads, being created > with > the > > >> CLI I tried to edit the configurations manually and added "option > > >> read-subvolume volume_name" to the cluster/replicate volume, being > the > > >> read-subvolume the local one. > > > > > > Do you have your filesystems mounted "noatime"? If not, every read > requires > > > the atime (last access time) to be written to the files on both > mirrors. > > > > > > Whit > > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster performance
On Tue, Sep 27, 2011 at 12:15:50PM +0100, Lone Wolf wrote: >So I am assuming gluster is distributing the reads, being created with the >CLI I tried to edit the configurations manually and added "option >read-subvolume volume_name" to the cluster/replicate volume, being the >read-subvolume the local one. Do you have your filesystems mounted "noatime"? If not, every read requires the atime (last access time) to be written to the files on both mirrors. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Replica out of sync using NFS
> I have two replicated bricks. I mounted brick1 as NFS, and started > writing/creating files. Did you mount brick1 (the backing store) through some other NFS daemon, or through the GlusterFS as NFS? If you do the first, stuff won't sync. If you do the second, the Gluster daemons handle the sync between the mirrors. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS re-export of guster mounted disks
On Tue, Sep 13, 2011 at 11:14:06AM -0500, Gerald Brandt wrote: > Hi, > > I hope I can explain this properly. > > 1. I have a two brick system replicating each other. (10.1.4.181 and > 10.1.40.2) > 2. I have a third system that mounts the gluster fileshares (192.168.30.111) > 3. I export the share on 192.168.30.111 as NFS to a XenServer. > > What I'm hoping for, is the aggregate speeds of gluster to the 2 servers > shows up when exported as NFS (roughly 2 GigE). Hi Gerald, Think you have an unusual setup there. Generally Gluster would be used to _either_ provide Gluster _or_ NFS shares from 10.1.4.181 and 10.1.40.2. So you'd have (1) and (3) in a normal setup, no (2). Also, I don't believe you can get aggregate speeds out of it, since each replica needs to receive everything, and it's synchronous. So if your write goes to .181, then .181 has to write to .2 before the operation is done, and it just has a single connection to .2, not an aggregate one. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] UCARP with NFS
On Thu, Sep 08, 2011 at 01:02:41PM +, anthony garnier wrote: > I got a client mounted on the VIP, when the Master fall, the client switch > automaticaly on the Slave with almost no delay, it works like a charm. But > when > the Master come back up, the mount point on the client freeze. > I've done a monitoring with tcpdump, when the master came up, the client send > paquets on the master but the master seems to not establish the TCP > connection. Anthony, Your UCARP command line choices and scripts would be worth looking at here. There are different UCARP behavior options for when the master comes back up. If the initial failover works fine, it may be that you'll have better results if you don't have a preferred master. That is, you can either have UCARP set so that the slave relinquishes the IP back to the master when the master comes back up, or you can have UCARP set so that the slave becomes the new master, until such time as the new master goes down, in which case the former master becomes master again. If you're doing it the first way, there may be a brief overlap, where both systems claim the VIP. That may be where your mount is failing. By doing it the second way, where the VIP is held by whichever system has it until that system actually goes down, there's no overlap. There shouldn't be a reason, in the Gluster context, to care which system is master, is there? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] How to know clients currently connected ?
>In my understanding of autofs is that it will mount as you need the >filesystem but will it remount the FS when there is a server crash ? Anthony, I haven't specifically tested it in this precise context, but in general that's exactly what it does. We went to using autofs because we had a server that was prone to crash. With autofs, as soon as that server came back up, it got remounted. Very reliable for that. When a server crashes the mount goes away. When you ask for a filesystem with autofs, if it's not there, it tries to mount it. I don't see how it can even know if you've switched which system has the IP and filesystem it's trying to mount. I know for sure it works well in a setup with DRBD/Heartbeat/NFS, so it ought to work as well for Gluster/UCARP/NFS. From autofs's perspective, it's just NFS. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] How to know clients currently connected ?
Hi Anthony, If you need the client to remain mounted, you need _some_ way of doing IP takeover. You could write your own script for that. As for remounting though, if your client is a *nix (Linux, OSX, whatever) you can use autofs to establish the mount, and that will also handle remounting automatically. Not sure if there's an autofs-type option for Windows. Whit On Tue, Sep 06, 2011 at 12:21:07PM +, anthony garnier wrote: >Hi folks, > >Maybe it's a strange question but I 'd like to know if there is a way to >see clients currently connected to the filesystem ( with NFS) ? > >Other question, except ucarp or heartbeat, is there an other way to do HA >in NFS ? ( I mean when client is connected to a server and then the server >crash, the client will be binded to an other one without remounting the >share. > >Anthony ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3beta2 - upgrade path?
> ... the new hotness of GlusterFS 3.3, the 2nd beta of > which we released today: > > http://www.gluster.com/community/documentation/index.php/3.3beta If that beta is installed over a 3.1.6 system (not too critical to production of course) will it work, or is there an upgrade path that simply hasn't been specified yet (anywhere I've looked)? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] cluster.min-free-disk separate for each, brick
Hi Deyan, This may not be a useful suggestion, but why not just partition your storage so that your bricks remain uniform? When you get more disk capacity because you've got 3TB drives instead of 2TB in RAID5, so that you have 15TB rather than 10TB per system, why not split the storage into 10TB and 5TB partitions, use the 10TB for a uniform-sized brick to add to an existing Gluster storage area, and then take your 5TB partitions and set up another Gluster share using that as your uniform brick size for it? Might not fit your requirements at all - depends on what you're storing and whether it makes sense to separate it into a couple of different areas. Best, Whit On Wed, Aug 17, 2011 at 04:28:42PM +0300, Deyan Chepishev - SuperHosting.BG wrote: > Hello, > > This is really bad news, because I already migrated my data and I > just realized that I am screwed because Gluster just does not care > about the brick sizes. > It is impossible to move to uniform brick sizes. > > Currently we use 2TB HDDs, but the disks are growing and soon we > will probably use 3TB hdds or whatever other larges sizes appear on > the market. So if we choose to use raid5 and some level of > redundancy (for example 6hdds in raid5, no matter what their size > is) this sooner or later will lead us to non uniform bricks which is > a problem and it is not correct to expect that we always can or want > to provide uniform size bricks. > > With this way of thinking if we currently have 10T from 6x2T in > hdd5, at some point when there is a 10T on a single disk we will > have to use no raid just because gluster can not handle non uniform > bricks. > > Regards, > Deyan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS secondary groups not working.
On Sat, Aug 13, 2011 at 01:59:20PM +0200, Dipeit wrote: > We noticed this bug too using the gluster client. I'm surprised that not > more people noticed this lack of posix compliance. This makes gluster > really unusable in multiuser environments. Is that because gluster is > mostly used in large web farms like pandora? It's been noticed, with much discussion here. It's supposed to have been fixed. There was just a notice that it has been in 3.1.16, just out. Haven't tested yet. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster Installation and Benchmarks
On Wed, Aug 10, 2011 at 03:30:22PM +0200, David Pusch wrote: > The objective is to create a redundant system. Shouldn't gluster be writing on > all 6 nodes simultaneously rather than sequentially? Else it would seem like a > rather poor choice for highly redundant systems. I get the redundant part. You'll see in recent discussion here that it's typical to deploy on top of hardware RAID. So you've got redundancy in each node. The if you replicate across 2 nodes, you've got redundancy on top of redundancy. You're far more likely to lose the data center to a catastrophy at that point than to lose your data on both nodes at once. The next move might be geo-replication, not more local replication. Since it's asynchronous, it won't delay local file operations. I'm far from an expert on replication across a network. But consider that you're starting from one copy of the file. Your file is not going to copy to all six Gluster nodes. It's going to copy to one, but that one node is going to copy to each of the five others, and insist on completing those copies while it completes its reception of the file. So your bottleneck is there, between the Gluster server your file has gone to, and the five other systems you've instructed it to write the file to as it comes in. Now, there's a very interesting report yesterday that the FUSE stuff works much better with the latest kernels, if that's a factor for you. But replicating to six systems at once is a strange thing to do. If your systems don't have RAID locally, there could be an argument for replicating to three, so that a coincidental failure of two drives on two of them doesn't cost you data. But six? You expect five systems to fail at once, from a cause that wouldn't take the sixth down too? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster Installation and Benchmarks
On Wed, Aug 10, 2011 at 02:22:55PM +0200, David Pusch wrote: > we now did another test where we mounted the Volume on the client and shut > down all Servers but one. We then transferred a 1 GB test file to the Volume. > The transfer took around 10 seconds. We then brought up another server from > the > Cluster and again transferred a 1 GB file. Transfer time now was roughly 20 > seconds. We proceeded in this manner for the last two servers, and each time > the transfer time increased by ca. 10 seconds. > an 18 brick distributed replicated Volume with a replica 6 setting. David, Why "replica 6"? That means you're keeping a copy of each file physically on each server. So if writing the file file to one takes 10 seconds, writing the file to a second takes another 10 seconds, and so on, that kind of makes sense. You can't transfer a single file to two places as fast as to one. Having more than two copies of any single file is unusual. Having more than three - I'm not sure why anyone would do that for local storage. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] issue with GlusterFS to store KVM guests
On Sat, Jul 30, 2011 at 04:33:00PM +0530, Anand Avati wrote: > Looks like your VM image is being attempted to open with O_DIRECT flag which > is > not supported in FUSE at the moment. You may try to configure your > libvirt/qemu > settings to open the VM without O_DIRECT. If not you may find https:// > github.com/avati/liboindirect useful. There's a May 11 post from Aaron Roberts saying: I'm guessing you're using qemu-kvm? If so, qemu-kvm will use O_SYNC (cache=writethough) to open a raw disk image and O_DIRECT (cache=none) to open a .qcow2 image. Would it then be accurate to say that fuse and qemu are currently compatible for raw kvm images, but not for qcow2 images? If so, that would explain why it's been working okay in my instance. I use raw images, which are said to be faster, if lacking qcow2 features like internal snapshotting. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] issue with GlusterFS to store KVM guests
On Fri, Jul 29, 2011 at 03:22:23PM -0700, Eric wrote: > i would prefer to use the glusterFS client for the automatic fail-over that it > offers. is this possible? I've built KVM VMs with virt-install writing to img files via the Gluster client. My invocation of virt-install differed from yours in a number of ways, though. I'm not at all an expert on that. But you might try altering your recipe. There are specifications that will work. Reportedly there are improvements coming soon in Gluster that will make it more suitable to using as a backing store VMs - possibly more in terms of efficiency than possibility though, since it's certainly possible now. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Configuration questions
> seeing as how the archives are, as is the norm for many lists, > non-searchable You can often search such archives with Google, specifying for instance site:http://gluster/org/pipermail/gluster-users/ along with the search terms (or fill in the site on the Advanced Search menu). Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] View from one client's gone subbornly bad
On Fri, Jul 22, 2011 at 10:26:13AM +0530, Anand Avati wrote: > Can you post gluster client logs and check if there are any core dumps? Okay, they are at http://transpect.com/gluster/ - may be a bit confused because of system reboots and gluster being shut down and restarted as I was working on it, but there you are. No core dumps. I'm wondering if Gluster might get into trouble sometimes when there's more than one replicated storage pair between the same two machines. These machines have three such, but mostly only one was active. Earlier yesterday I was doing work that more heavily used another of them, and it was in the midst of that that the more-used one got dodgy. Last night, Me: > And what do I do to wake that disconnected endpoint in the morning? Strangely, it's now working again for that one. It didn't display the files at first this morning. I tried a "gluster volume start kvm" which only got a statement that it was "already running." I did a restart of autofs then, and the files were available, and have remained so. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] View from one client's gone subbornly bad
Okay ... Finally got that one replicated partition back in line. A few of the recommended find /mnt/point -print0 | xargs --null stat from each side seems to have done some good. Then while I'm away a second replicated partition on the same two systems ends up with a Transport endpoint is disconnected and even totally shutting down all the Gluster processes on that box and restarting them does nothing for this - doesn't even create more entries in the log for it. The other two replicated Gluster shares between these machines are operating still - including the one I first had the trouble with today. But this third one that decided it would be disconnected seems intent to stay that way - despite that it's the same physical connection betweent the machines - which is fine - and the same Gluster daemons running on both. Again, this was all happy for many weeks with 3.1.3. So I'd give pretty good odds that 3.1.5 has some deep bugs. Should I go back, or do things finally look better going forward? And what do I do to wake that disconnected endpoint in the morning? Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] View from one client's gone subbornly bad
> The client on the other system in the pair continues through this to have > normal access. The system with the Gluster client problem shows no other > symptoms. Update: Now the second system's having the same symptoms. Maybe I need to just go back to 3.1.3? Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] View from one client's gone subbornly bad
Hi, This setup is two system which have replicated Gluster storage, and also are the clients for it. In both cases the Gluster mount is handled by autofs. The setup was quite stable with Gluster 3.1.3 for some weeks. Then a few days back I upgraded to 3.1.5 - wanted to try it on a less essential system before risking a production instance with the upgrade. Today the view from one of the systems, as client, has gone sour. The mount keeps disappearing, in a way that requires a manual "umount -l" on it to get automount to bring it back - for a bit. Operations that involve writing to it or even viewing the directory tend to lock up. I've rebooted the system several times with no improvement. I've copied all the files out of the backing store with Gluster not running, and then after restarting Gluster back in through the mount - which seemed to work. But then no improvement. The Gluster logs look totally uneventful through this - not many messages and nothing they hadn't had on prior days. The client on the other system in the pair continues through this to have normal access. The system with the Gluster client problem shows no other symptoms. What's the right way to troubleshoot this? Need to learn how before the production system I'm using Gluster with gets into this sort of trouble. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Tools for the admin
On Thu, Jul 21, 2011 at 05:17:05PM -0400, Joe Landman wrote: > Depending upon distro, you would need libattr, attr-dev, etc. > Sadly, there is little/no consistency in naming these. On Ubuntu 10.04 attr-dev is an alias for libattr1-dev, which seems to install okay but there's still no ExtAttr.pm on the system. However fetching the source for File::ExtAttr from CPAN and building that by hand, rather than with the cpan utility, does work. Then Getopt::Lucid does install successfully with the cpan utility - couldn't find it as an Ubuntu package. At which point your script runs. Thanks, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Tools for the admin
On Thu, Jul 14, 2011 at 12:15:00AM -0400, Joe Landman wrote: > Tool #2: scan_gluster.pl Joe, Thanks for this contributions. scan_gluster.pl seems to depend on File::ExtAttr, which when I try to install it from cpan to Perl v5.10.1 fails at the "make" step for reasons cpan leaves unclear. Is there some prerequisite that might not be properly requested? Looking at this now because I just had a simple Gluster replication setup blow up in a nonobvious way. Not obvious from the logs anyhow. It may well be because I switched it from 3.1.3 to 3.1.5 a few days back. Anyway, it gives me a good excuse to learn how to diagnose a good failure, so I thought I'd start by seeing what your tools can show. Best, Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] how to rotate log of client side
> Is there any way to rotate client side’s log > > It’s getting bigger every day. You can also configure logrotate - which is on most Linux systems - to handle this. "man logrotate". It'll handle compression and deletion of older files, too. Whit ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
