Re: [Gluster-users] GlusterFS 3.4.0 and 3.3.2 released!
"Jon Tegner" writes: > Great! > > Much appreciated! > > Two (stupid) questions: > > 1. Which of these two releases would work "best" using RDMA? > 2. In release Notes for 3.4.0 it is stated that ext4-issue has been > addressed - does this mean that it is safe to use ext4 with later > kernels now? Also interested in #2. Are there any caveats to this fix (performance, stability)? Additionally: In the "known issues" for 3.4 is the ominous and vague "replace-brick operation does not work fine in all cases in this release". Is any further detail available on this problem? Will replace-brick be a roll of the dice until 3.5? In general, links to the bug-tracker for know issues are appreciated in the release notes. -- Shawn Nock (OpenPGP: 0x65118FA5) pgpp5MJsFzgWG.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 40 gig ethernet
Justin Clift writes: >> The other issue I have is with hardware RAID. I'm not sure if folks >> are using that with gluster or if they're using software RAID, but >> the closed source nature and crappy proprietary tools annoys all the >> devops guys I know. What are you all doing for your gluster setups? >> Is there some magical RAID controller that has Free tools, or are >> people using mdadm, or are people just unhappy or ? > > Before joining Red Hat I was using Areca hardware. But Areca (the > company) was weird/dishonest when I tried to RMA a card that went bad. > > So, I advise people to keep away from that crowd. Haven't tried any > others in depth since. :/ I second the thoughts on Areca. They are a terrible company; avoid at all costs. I've RMA'd every card I've installed of theirs that had been in service for more that 6 months, some servers have had RMA returns fail within months. Their only US support option is "we'll ship it to Taiwan for repair and return it is 6-8 weeks". There is no option to pay for advanced replacement. I had to keep a stock of spares in-house until I migrated to 3ware (now LSI). I haven't had any trouble with these cards in several years (and haven't needed to RMA or contact support). -- Shawn Nock (OpenPGP: 0x65118FA5) pgpyp9RwsWROw.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] Beyond 3.4.0alpha2
Vijay Bellur writes: > On 04/06/2013 11:05 AM, Emmanuel Dreyfus wrote: >> Gluster Build System wrote: >> >>> SRC: >> http://bits.gluster.org/pub/gluster/glusterfs/src/glusterfs-3.4.0alpha2.> >> tar.gz >> This is almost one moth old. Are we going to have a new snapshot? >> alpha3 or beta1? >> > > Yes, intend having a release this week. Please stay tuned. Any word on backports to 3.3? 3.3.1 is six months old this week. -- Shawn Nock (OpenPGP: 0x65118FA5) pgpLtF8ru6TBV.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to evaluate the glusterfs performance with small file workload?
nlxswig writes: > Hi guys > > I have met some troubles when I want to evaluate the glusterfs > performance with small file workload. > > 1: What kind of benchmark should I use to test the small file > operation ? I use fio (http://freecode.com/projects/fio). Here's an old tutorial: https://www.linux.com/learn/tutorials/442451-inspecting-disk-io-performance-with-fio/ You'll need to read the docs to use it well, but the tutorial gives some idea of the operation. -- Shawn Nock (OpenPGP: 0x65118FA5) pgpv7JLYLubma.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Slow read performance
Joe Julian writes: > This bug is in the kernel. > > "If a change results in user programs breaking, it's a bug in the > kernel. We never EVER blame the user programs." - Linus Torvalds > > http://lkml.org/lkml/2012/12/23/75 Understood. However, there was an update to your post here: http://www.gluster.org/2012/08/glusterfs-bit-by-ext4-structure-change/ : "It [a patchset for gluster addressing the ext4 changes] is still being actively worked on, though, and is a high priority." The 'high priority' bit raised a few expectations; that was more than 6 months ago. While I feel like the gluster devs don't owe me anything, this issue does effect pretty much every user with an ext3/4 brick. There (to my recollection) hasn't been any guidance on how the community should address this in their installations. It's pretty clear in the Redhat Storage docs that Gluster has XFS (and lvm) as a hard dependency... but the 3.3 admin guide dosn't say anything useful (about fs choice) and it 3.1/3.2 docs used to recommend ext4 as a well tested option. It doesn't seem like there's any talk of reverting the commit in the mainline kernel. It seems to be very useful for preventing hash collisions for a lot of kNFS+ext3/4 workflows and it's been backported to the enterprise distributions. It's not a gluster bug, but it's here to stay. What do we do now? (Rhetorical question, I am removing bricks and reformatting with xfs and the re-adding/rebalancing... slowly). -- Shawn Nock (OpenPGP: 0x65118FA5) pgp6tottTaH5G.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Disappointing documentation?
Shawn Nock writes: > 2. Copying the excellent info about gluster's xattrs from this blog > post (http://cloudfs.org/2011/04/glusterfs-extended-attributes/) into > the admin guide would be a start. The correct link is: http://hekafs.org/index.php/2011/04/glusterfs-extended-attributes/ , but so many of Jeff Darcy's posts have proven useful to me. Perhaps the correct word is "adapt" and not "copy". -- Shawn Nock (OpenPGP: 0x65118FA5) pgp0G6F6_0K5V.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Disappointing documentation?
Joe Julian writes: > It comes up on this list from time to time that there's not sufficient > documentation on troubleshooting. I assume that's what some people > mean when they refer to disappointing documentation as the current > documentation is far more detailed and useful than it was 3 years ago > when I got started. I'm not really sure what's being asked for here, > nor am I sure how one would document how to troubleshoot. In my mind, > if there's a trouble that can be documented with a clear path to > resolution, then a bug report should be filed and that should be > fixed. Any other cases that cannot be coded for require human > intervention and are already documented. It is true that the documentation has gotten better. However, since the switch to the new release cycle, bugs don't seem to get fixed (within a release) and the documentation could do a better job listing some of the holes new users starting with the current GA will likely fall into: Examples: - Don't use ext4 (https://bugzilla.redhat.com/show_bug.cgi?id=838784) - Don't use fix-layout after adding a brick (https://bugzilla.redhat.com/show_bug.cgi?id=913699), maybe fixed by 10617e6cbc73329f259b471327d88375352042b0 in 3.3.1 but: - Don't upgrade from 3.3 to 3.3.1 if you need NFS (https://bugzilla.redhat.com/show_bug.cgi?id=893778) 1. Perhaps a wiki entry like "Known Issues" with links to all these bugs? 2. Copying the excellent info about gluster's xattrs from this blog post (http://cloudfs.org/2011/04/glusterfs-extended-attributes/) into the admin guide would be a start. 3. A brief guide on how to collect info on problematic files (permissions, xattrs, client log, brick log) would probably help generate more helpful bug reports and help users sort out many of their own problems. It's all stuff you pickup after you've been in the game for a while, but they must really flummox new users. -- Shawn Nock (OpenPGP: 0x65118FA5) pgpXeto644M6M.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] I/O error repaired only by owner or root access
Dan Bretherton writes: > Hello Rajesh- Here are the permissions. The path in question is a > directory. > > [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl > drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37 > /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl [sms05dab@jupiter ~]$ ls -ld > /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN lrwxrwxrwx 1 gcs nemo 49 Feb 1 > 2012 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN -> > /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN [sms05dab@jupiter > ~]$ ls -ld /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN > drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14 > /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN > > As you can see the parent directory in this case was a symlink but > that's not significant. I ran the "ls -l" commands using my account - > sms05dab, but the problem was originally reported by user vq901510. > until I did "ls -l" as root neither of us could access the directory, > because the parent directory was owned by user gcs. Usually the > problem is related to ownership of the file or directory itself. This > is the first time I have seen the I/O error caused by parent directory > permissions. > > This problem seems to have started following an add-brick operation a > few weeks ago, after which I started "gluster volume rebalance > fix-layout" (which is still running). It occurred to me that > the problem could be related to link files, many of which need to be > rewritten following add-brick operations. This could explain why the > ownership of the parent directory is significant, because users > sms05dab and vq901510 don't have permission to write in the parent > directory owned by user gcs. Normally this wouldn't be a problem > because only read access to other users' data is required, but it > appears as though read access was being denied because the new link > file couldn't be written by unprivileged users. Is this a plausible > explanation of the I/O error do you think? This sounds like my recent bug: https://bugzilla.redhat.com/show_bug.cgi?id=913699 In the bug, I said that writing on the fuse mount of one of the brick servers fixed the problem but those were the only hosts I was attempting access as root. -- Shawn Nock (OpenPGP: 0x65118FA5) pgpdFuvpbtdLl.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Conservative merge fails on client3_1_mknod_cbk
Since I've added two bricks to my 2x2 (now 3x2) distribute-replicate volume (and run fix-layout), accessing many of the volumes files via fuse fails with "invalid argument". [2013-02-21 12:48:53.121691] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-mirror-replicate-2: [...]/140.ACQ: Performing conservative merge [2013-02-21 12:48:54.100924] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-mirror-client-5: remote operation failed: Permission denied. Path: [...]/140.ACQ/1.3.12.2.1107.5.2.32.35052.2011011711023368433700039.IMA (ddef2b1a-b3cc-424c-a663-995bb77cd4c4) [2013-02-21 12:48:54.101005] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-mirror-client-4: remote operation failed: Permission denied. Path: [...]/140.ACQ/1.3.12.2.1107.5.2.32.35052.2011011711023368433700039.IMA (ddef2b1a-b3cc-424c-a663-995bb77cd4c4) [2013-02-21 13:20:31.971211] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 1360169: OPEN() [...]/140.ACQ/1.3.12.2.1107.5.2.32.35052.2011011711023368433700039.IMA => -1 (Invalid argument) Additional information is here: https://bugzilla.redhat.com/show_bug.cgi?id=913699 Strangely, opening the troublesome files on a fuse mount on any of the block-servers completes without issue, AND afterward other fuse clients can open the file as well. Any thoughts would be helpful... right now I am opening every file on that volume on one of the block-servers as a work around. I have many small files and directories; It'll likely take days. -- Shawn Nock (OpenPGP: 0x65118FA5) pgp14AOA7bIBp.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
