[Gluster-users] unable to delete 'empty' dirs with 3.3qa42
Just before updrading to 3.3final, we had an rsync collision on our gluster filesystem which left us with undeletable dirs. The transport is IPoIB over 4 bricks as shown below. $ gluster volume info Volume Name: gli Type: Distribute Volume ID: 76cc5e88-0ac4-42ac-a4a3-31bf2ba611d4 Status: Started Number of Bricks: 4 Transport-type: tcp,rdma Bricks: Brick1: pbs1ib:/bducgl Brick2: pbs2ib:/bducgl Brick3: pbs3ib:/bducgl Brick4: pbs4ib:/bducgl Options Reconfigured: performance.io-cache: on performance.quick-read: on performance.io-thread-count: 64 The last 100 line of the gluster log is here: http://pastie.org/4036243 The problem started when I was running an rsync with --delete while moving a large /home to the gluster dir to prune the target of older cruft in a /home move.. A user directed a job to start reading from a directory that was simultaneously being pruned and all hell broke loose. Now we're left with a number of dirs and files that can't be deleted. 'du' says there are files in an apparently empty dir: root@bduc-login:/gl/tvanerp/adni 561 $ du -sh * 40K dicom altho 'ls' can't see them from the client; root@bduc-login:/gl/tvanerp/adni 562 $ ls -lat dicom total 40 drwxrwxr-x 3 tvanerp tvanerp 94214 Jun 5 23:40 ./ drwxrwxr-x 3 tvanerp tvanerp72 Apr 23 13:11 ../ From the bricks you can see that there are still files there: Tue Jun 05 23:18:02 [3.06 2.42 1.65][457.48/606] root@bduc-login:~ 556 $ ssh pbs2 'ls -lat /bducgl/tvanerp/adni/dicom' total 24 drwxrwxr-x 3 7335 7335 28672 2012-06-05 22:59 . drwxr-xr-x 2 7335 7335 8192 2012-06-05 14:50 128_S_0947_20080422_MPRAGE drwxrwxr-x 3 7335 733518 2012-04-23 13:11 .. Tue Jun 05 23:18:26 [2.82 2.42 1.67][457.62/606] root@bduc-login:~ 557 $ ssh pbs3 'ls -lat /bducgl/tvanerp/adni/dicom' total 28 drwxrwxr-x 3 7335 7335 28672 2012-06-05 23:00 . drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:52 128_S_0947_20080422_MPRAGE drwxrwxr-x 3 7335 733518 2012-04-23 13:11 .. Tue Jun 05 23:18:51 [2.73 2.43 1.69][457.7/606] root@bduc-login:~ 558 $ ssh pbs4 'ls -lat /bducgl/tvanerp/adni/dicom' total 32 drwxrwxr-x 3 7335 7335 36864 2012-06-05 22:59 . drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:50 128_S_0947_20080422_MPRAGE -T 2 root root 0 2012-05-20 18:35 adni_pib_subjects.txt drwxrwxr-x 3 7335 733518 2012-04-23 13:11 .. but the client is not able to see/delete them, even as root. Suggestions? harry -- Harry Mangalam,Research Computing, OIT, Rm 225 MSTB, UC Irvine [mailcode 2225] Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] Lat/Long: (33.642025,-117.844414) [paste into Google Maps] -- ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Striped replicated volumes in Gluster 3.3.0
Hi All, I'm sorry if the release didn't address the specific features you need, and I'm sorry if we gave the impression that it would. Our additional features for 3.3 were always pretty clear, or so I thought. If you can find any statements from the past year that were misleading, I would be happy to address them directly, but your statement above was a bit vague. John Mark, Please consider my comments in the context of Amar's comment, which I quoted above it: On Tue, Jun 05, 2012 at 11:09:39AM +0530, Amar Tumballi wrote: I saw that mail and I agree that the target of 3.3.0 was to make glusterfs more stable and get the features which would make Virtual machine hosting on GlusterFS a possibility by 3.4.0 time-frame. Does that not seem to say that virtual machine hosting is not really to be a possibility before the 3.4.0 time-frame? From subsequent responses, including yours, Amar likely misspoke. Excuse me for responding to a message that seemed to be saying that Gluster won't be fully ready for VM hosting before 3.4.0. I am really sorry as the words in my earlier mail gave raise to so many confusions. What I spoke or mentioned was surely misleading. What I wanted to say was mostly from many different angle. With 3.3.0, we made changes which provide almost all the VM hosting functionality in GlusterFS (like what avati mentioned). But as a company, our internal QA team's main focus was to get the 'stable' version out, which solves the general storage pain points. My words were mostly based on internal QA perspective, because we couldn't test every possibility with VM hosting internally, hence the advise is that, please have a test run in your test setup before putting it in production. I am not saying its not ready yet, but we would surely like more testing time for glusterfs as VM hosting, to 'officially' say we are good with VM Hosting. 3.4.0 just a indicative version release I used to say in later versions, I should have used 3.3.1, I guess. To fix any bugs which crops up with VM hosting, We request you to run more tests and report issues, so we can close the loop on this. -Amar ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
RDMA is definitely still supported - although we may have simply rearranged how the packaging is done. You should be able to use it as before. But if the provided debian package doesn't include RDMA, how can I use it as before? Only option I see is compiling myself. flip -- Rádio Zero www.radiozero.pt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Gluster 3.3 and xen 4.1.2 under Kernel 3.2 problem
Hello i have following problem, i'm trying to run gluster server 3.3.0 with xen 3.1.2 with kernel 3.2.12. I'm running glusted on dom0, not inside virtual env. Glusterd is dieing with following message: E [glusterd.c:270:glusterd_check_gsync_present] 0-glusterd: geo-replication module not working as desired D [glusterd.c:298:glusterd_check_gsync_present] 0-glusterd: Returning -1 E [xlator.c:385:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again With hypervisor turned off same kernel and exactly same settings works fine. thanks Matus ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] suggestions to improve our existing setup
Hi, We brought our two node gluster replication down as we were blaming poor gluster for what turns out to be filesystem issues, can't blame gluster for replicating disappearing folders. While the system is down there's a chance to rethink things. Would anyone suggest a better approach regarding OS and filesystem than our original file server setup? :: Xen Cloud Platform :: Two CentOS 5.7 Virtual Machines with latest gluster (one on iscsi san and other using host local drive for storage) replicating 1TB of files on same lan :: 1TB accessed by separate VM serving 1s of files under 1Mb using samba and glusterfs mount Thanks, Chris ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
We're looking into it - expect a resolution shortly. -JM - Original Message - RDMA is definitely still supported - although we may have simply rearranged how the packaging is done. You should be able to use it as before. But if the provided debian package doesn't include RDMA, how can I use it as before? Only option I see is compiling myself. flip -- Rádio Zero www.radiozero.pt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Troubleshooting Unified Object and File Storage in 3.3
Hi everyone, I've been working on testing UFO in Gluster 3.3, but I've had a hard time getting it working. I've been following along with the admin guide at http://www.gluster.org/wp-content/uploads/2012/05/Gluster_File_System-3.3.0-Administration_Guide-en-US.pdf, on chapter 12. I've been testing on CentOS 6.2. The only command from the Admin guide I've run successfully has been: curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass:testing' -k http://127.0.0.1:8080/auth/v1.0. I started out with a centos machine running gluster-swift, which I was connecting to a four node gluster cluster. It wasn't clear to me from the admin guide where I was supposed to mount my gluster volume, so I've tried various permutations, including mounting it at the location specified in the /etc/swift/fs.conf file, with a symlink to the /srv/1/node location cited in the 1.conf files provided by the gluster-swift-plugin package. I also tried creating a gluster volume on the same server as the gluster-swift, but that didn't work for me, either. Is there a better source of docs than the admin guide? I'll be happy to help write this up once I get everything working. One more thing, the 3.3 admin guide talks about downloading the 3.2 version of gluster-swift and gluster-swift-plugin -- I'd assumed that that was a typo, and I've been working with the packages from 3.3, but maybe I was wrong there -- are the 3.2 packages the right ones? Thanks, Jason ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Performance optimization tips Gluster 3.3? (small files / directory listings)
Hi, I'm using Gluster 3.3.0-1.el6.x86_64, on two storage nodes, replicated mode (fs1, fs2) Node specs: CentOS 6.2 Intel Quad Core 2.8Ghz, 4Gb ram, 3ware raid, 2x500GB sata 7200rpm (RAID1 for os), 6x1TB sata 7200rpm (RAID10 for /data), 1Gbit network I've it mounted data partition to web1 a Dual Quad 2.8Ghz, 8Gb ram, using glusterfs. (also tried NFS - Gluster mount) We have 50Gb of files, ~800'000 files in 3 levels of directories (max 2000 directories in one folder) My main problem is speed of directory indexes ls -alR on the gluster mount takes 23 minutes every time. It don't seem like any directory listing information cache, with regular NFS (not gluster) between web1-fs1, this takes 6m13s first time, and 5m13s there after. Gluster mount is 4+ times slower for directory indexing performance vs pure NFS to single server, is this as expected? I understand there is a lot more calls involved checking both nodes but I'm just looking for a reality check regarding this. Any suggestions of how I can speed this up? Thanks, ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Performance optimization tips Gluster 3.3? (small files / directory listings)
Could you post the logs of the mount process so that we can analyse what is going on. Did you have data on bricks before you created the volume? Did you upgrade from 3.2? Pranith - Original Message - From: olav johansen luxis2...@gmail.com To: gluster-users@gluster.org Sent: Thursday, June 7, 2012 8:02:14 AM Subject: [Gluster-users] Performance optimization tips Gluster 3.3? (small files / directory listings) Hi, I'm using Gluster 3.3.0-1.el6.x86_64, on two storage nodes, replicated mode (fs1, fs2) Node specs: CentOS 6.2 Intel Quad Core 2.8Ghz, 4Gb ram, 3ware raid, 2x500GB sata 7200rpm (RAID1 for os), 6x1TB sata 7200rpm (RAID10 for /data), 1Gbit network I've it mounted data partition to web1 a Dual Quad 2.8Ghz, 8Gb ram, using glusterfs. (also tried NFS - Gluster mount) We have 50Gb of files, ~800'000 files in 3 levels of directories (max 2000 directories in one folder) My main problem is speed of directory indexes ls -alR on the gluster mount takes 23 minutes every time. It don't seem like any directory listing information cache, with regular NFS (not gluster) between web1-fs1, this takes 6m13s first time, and 5m13s there after. Gluster mount is 4+ times slower for directory indexing performance vs pure NFS to single server, is this as expected? I understand there is a lot more calls involved checking both nodes but I'm just looking for a reality check regarding this. Any suggestions of how I can speed this up? Thanks, ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Documentation on recovering from split-brain situation
Hi, I'm trying to find official documentation that describes the procedure for recovering from a split-brain situation with replicated volumes. I can find various posts on the mailing list that refer to the version 2.x series, but nothing good for 3.x. Can anyone point me in the right direction? Thanks, Toby ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Troubleshooting Unified Object and File Storage in 3.3
On 06/07/2012 03:22 AM, Jason Brooks wrote: I've been testing on CentOS 6.2. The only command from the Admin guide I've run successfully has been: curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass:testing' -k http://127.0.0.1:8080/auth/v1.0. I started out with a centos machine running gluster-swift, which I was connecting to a four node gluster cluster. It wasn't clear to me from the admin guide where I was supposed to mount my gluster volume, You will need to mount the gluster volume at /mnt/gluster-object/account. For the example in admin guide, /mnt/gluster-object/AUTH_test needs to be the mountpoint for your gluster volume. Is there a better source of docs than the admin guide? I'll be happy to help write this up once I get everything working. The admin guide sources are in the glusterfs git repo now and has some updates in release-3.3 branch since the last published version. You can look that up and also contribute patches for that via gerrit. One more thing, the 3.3 admin guide talks about downloading the 3.2 version of gluster-swift and gluster-swift-plugin -- I'd assumed that that was a typo, and I've been working with the packages from 3.3, but maybe I was wrong there -- are the 3.2 packages the right ones? That certainly is a typo. Fixed in release-3.3 doc as part of: http://review.gluster.com/#change,3498 Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] A very special announcement from Gluster.org
Hi Philip, Did you try installing libssl from source to meet the dependency? -sac Sent from my iPhone On 02-Jun-2012, at 13:57, Philip flip...@googlemail.com wrote: It is still not possible to install the 3.3 deb on a stable release of debian because squeeze has no libssl1.0.0. 2012/5/31 John Mark Walker johnm...@redhat.com Today, we’re announcing the next generation of GlusterFS, version 3.3. The release has been a year in the making and marks several firsts: the first post-acquisition release under Red Hat, our first major act as an openly-governed projectand our first foray beyond NAS. We’ve also taken our first steps towards merging big data and unstructured data storage, giving users and developers new ways of managing their data scalability challenges. GlusterFS is an open source, fully distributed storage solution for the world’s ever-increasing volume of unstructured data. It is a software-only, highly available, scale-out, centrally managed storage pool that can be backed by POSIX filesystems that support extended attributes, such as Ext3/4, XFS, BTRFS and many more. This release provides many of the most commonly requested features including proactive self-healing, quorum enforcement, and granular locking for self-healing, as well as many additional bug fixes and enhancements. Some of the more noteworthy features include: Unified File and Object storage – Blending OpenStack’s Object Storage API with GlusterFS provides simultaneous read and write access to data as files or as objects. HDFS compatibility – Gives Hadoop administrators the ability to run MapReduce jobs on unstructured data on GlusterFS and access the data with well-known tools and shell scripts. Proactive self-healing – GlusterFS volumes will now automatically restore file integrity after a replica recovers from failure. Granular locking – Allows large files to be accessed even during self-healing, a feature that is particularly important for VM images. Replication improvements – With quorum enforcement you can be confident that your data has been written in at least the configured number of places before the file operation returns, allowing a user-configurable adjustment to fault tolerance vs performance. Visit http://www.gluster.org to download. Packages are available for most distributions, including Fedora, Debian, RHEL, Ubuntu and CentOS. Get involved! Join us on #gluster on freenode, join our mailing list, ‘like’ our Facebook page, follow us on Twitter, or check out our LinkedIn group. GlusterFS is an open source project sponsored by Red Hat®, who uses it in its line of Red Hat Storage products. (this post published at http://www.gluster.org/2012/05/introducing-glusterfs-3-3/ ) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
Hi Filipe, I have built the RDMA packages and can be found in: http://download.gluster.com/pub/gluster/glusterfs/LATEST/Debian/5.0.3/ -sac - Original Message - From: Filipe Roque flip.ro...@gmail.com To: gluster-users@gluster.org Sent: Wednesday, June 6, 2012 5:02:51 PM Subject: Re: [Gluster-users] 3.3 rdma in .deb package RDMA is definitely still supported - although we may have simply rearranged how the packaging is done. You should be able to use it as before. But if the provided debian package doesn't include RDMA, how can I use it as before? Only option I see is compiling myself. flip -- Rádio Zero www.radiozero.pt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Is glusterd required on clients?
On Tuesday 22 May 2012 10:54 AM, Toby Corkindale wrote: On Linux clients, using the FUSE method of mounting volumes, do you need glusterd to be running? I don't *think* so, but want to check. You don't need glusterd to be running on the clients. It is a management daemon required only on servers if you are doing operations using the gluster cli -sac ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
Hi Filipe, On Wednesday 06 June 2012 05:02 PM, Filipe Roque wrote: RDMA is definitely still supported - although we may have simply rearranged how the packaging is done. You should be able to use it as before. But if the provided debian package doesn't include RDMA, how can I use it as before? Only option I see is compiling myself. Currently, as we are not doing extensive RDMA tests on debian/Ubuntu machines, we have disabled RDMA in the packages. However, as you mentioned you have to compile from source if you need RDMA support. -sachidananda. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
On 06/06/2012 05:06 PM, Sachidananda Urs wrote: Hi Filipe, On Wednesday 06 June 2012 05:02 PM, Filipe Roque wrote: RDMA is definitely still supported - although we may have simply rearranged how the packaging is done. You should be able to use it as before. But if the provided debian package doesn't include RDMA, how can I use it as before? Only option I see is compiling myself. Currently, as we are not doing extensive RDMA tests on debian/Ubuntu machines, we have disabled RDMA in the packages. However, as you mentioned you have to compile from source if you need RDMA support. Please ignore this message. A moderation mistake pushed this message out. .deb RDMA packages are now available in the download site. Thanks, Vijay ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
On 07/06/12 01:10, Sachidananda URS wrote: Hi Filipe, I have built the RDMA packages and can be found in: http://download.gluster.com/pub/gluster/glusterfs/LATEST/Debian/5.0.3/ Why is the .deb in a Debian 5.0.3 directory, when it can't be installed on anything earlier than version 7? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
Agreed - the directory structure needs to be fixed. -JM - Original Message - On 07/06/12 01:10, Sachidananda URS wrote: Hi Filipe, I have built the RDMA packages and can be found in: http://download.gluster.com/pub/gluster/glusterfs/LATEST/Debian/5.0.3/ Why is the .deb in a Debian 5.0.3 directory, when it can't be installed on anything earlier than version 7? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.3 rdma in .deb package
- Original Message - Please ignore this message. A moderation mistake pushed this message out. .deb RDMA packages are now available in the download site. Thanks, Vijay and Sac. -JM ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
