Re: [Gluster-users] glusterfs client waiting on SYN_SENT to connect...
Ah the two different IPs are because I was changing my IPs for this mailing list and I guess I forgot that one. :) Will try added a static route. Also going to snoop traffic and see if the gluster client is actually getting to the server or being blocked by the firewall. Ill letcha all know what I find. Thanks for the ideas. Liam On Dec 3, 2010 6:32 PM, wrote: > On Fri, Dec 03, 2010 at 04:25:18PM -0800, Liam Slusser wrote: >> [r...@client~]# netstat -pan|grep glus >> tcp 0 1 10.8.10.107:1000 10.8.11.102:6996 SYN_SENT 3385/glusterfs >> >> from the gluster client log: >> >> However, the port is obviously open... >> >> [r...@client~]# telnet 10.8.11.102 6996 >> Trying 10.2.56.102... >> Connected to glusterserverb (10.8.11.102). >> Escape character is '^]'. >> ^] >> telnet> close >> Connection closed. > > Looking further... why is your telnet trying 10.2.56.102 when you > clearly specified 10.8.11.102? Also, what happens if you do a > specific route for the 10.8.11.0/24 block thru the appropriate gw > without relying on the default gw to route for you? In this way > you dont end up in a situation where the client is mistakenly > trying to go over the wrong interface. The telnet maybe switching > to an alternate interface to see if it gets thru? > > Mohan > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs client waiting on SYN_SENT to connect...
On Fri, Dec 03, 2010 at 04:25:18PM -0800, Liam Slusser wrote: > [r...@client~]# netstat -pan|grep glus > tcp 0 1 10.8.10.107:1000 10.8.11.102:6996 SYN_SENT 3385/glusterfs > > from the gluster client log: > > However, the port is obviously open... > > [r...@client~]# telnet 10.8.11.102 6996 > Trying 10.2.56.102... > Connected to glusterserverb (10.8.11.102). > Escape character is '^]'. > ^] > telnet> close > Connection closed. Looking further... why is your telnet trying 10.2.56.102 when you clearly specified 10.8.11.102? Also, what happens if you do a specific route for the 10.8.11.0/24 block thru the appropriate gw without relying on the default gw to route for you? In this way you dont end up in a situation where the client is mistakenly trying to go over the wrong interface. The telnet maybe switching to an alternate interface to see if it gets thru? Mohan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs client waiting on SYN_SENT to connect...
Telnet never fails. Gluster client consistently fails however. Server is using bonded NICs but as far as i can tell they're configured correctly, both links are up and passing traffic. On Fri, Dec 3, 2010 at 6:15 PM, wrote: > On Fri, Dec 03, 2010 at 06:03:19PM -0800, Liam Slusser wrote: >> > This almost seems like a networking/firewall issue... ?Do you have >> > any trunks setup between the switch that the client and/or server >> > are on and the router? ?Perhaps one of those trunk legs is down >> > causing random packets to get blackholed? >> >> I thought the exact same thing...but like i said i can telnet to the >> host/port without any issue. And there is no other issues on the >> network that would indicate any not working correctly. And all the >> other clients on the same network/switch are working fine. Its only >> when a client crashes... > > Consistently? If random telnets fail then that would explain your > random SYN_SENT state stuck sockets. Is the client or server using > bonded nics? > > Mohan > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs client waiting on SYN_SENT to connect...
On Fri, Dec 03, 2010 at 06:03:19PM -0800, Liam Slusser wrote: > > This almost seems like a networking/firewall issue... ?Do you have > > any trunks setup between the switch that the client and/or server > > are on and the router? ?Perhaps one of those trunk legs is down > > causing random packets to get blackholed? > > I thought the exact same thing...but like i said i can telnet to the > host/port without any issue. And there is no other issues on the > network that would indicate any not working correctly. And all the > other clients on the same network/switch are working fine. Its only > when a client crashes... Consistently? If random telnets fail then that would explain your random SYN_SENT state stuck sockets. Is the client or server using bonded nics? Mohan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] glusterfs client waiting on SYN_SENT to connect...
I thought the exact same thing...but like i said i can telnet to the host/port without any issue. And there is no other issues on the network that would indicate any not working correctly. And all the other clients on the same network/switch are working fine. Its only when a client crashes... liam On Fri, Dec 3, 2010 at 4:34 PM, wrote: >> I've run into a weird problem. I have a few client boxes that >> occasionally crash due to a non-gluster related problem. But once the >> box comes back up i cannot get the Gluster client to reconnect to the >> bricks. > > This almost seems like a networking/firewall issue... Do you have > any trunks setup between the switch that the client and/or server > are on and the router? Perhaps one of those trunk legs is down > causing random packets to get blackholed? > > Mohan > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] glusterfs client waiting on SYN_SENT to connect...
Hey all, I've run into a weird problem. I have a few client boxes that occasionally crash due to a non-gluster related problem. But once the box comes back up i cannot get the Gluster client to reconnect to the bricks. Centos 5 64bit and Gluster 2.0.9 df shows: df: `/mnt/mymount': Transport endpoint is not connected [r...@client~]# netstat -pan|grep glus tcp0 1 10.8.10.107:100010.8.11.102:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:100110.8.11.102:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:998 10.8.11.102:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:996 10.8.11.102:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:100310.8.11.101:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:100210.8.11.101:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:997 10.8.11.101:6996 SYN_SENT3385/glusterfs tcp0 1 10.8.10.107:999 10.8.11.101:6996 SYN_SENT3385/glusterfs from the gluster client log: +--+ [2010-12-03 15:48:28] W [glusterfsd.c:526:_log_if_option_is_invalid] readahead: option 'page-size' is not recognized [2010-12-03 15:48:28] N [glusterfsd.c:1306:main] glusterfs: Successfully started [2010-12-03 15:48:29] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 2: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:48:30] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 3: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:48:31] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 4: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:48:31] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 5: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:48:32] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 6: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick1a: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick1a: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick2a: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick2a: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick1b: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick1b: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick2b: connection to failed (Connection timed out) [2010-12-03 15:51:37] E [socket.c:745:socket_connect_finish] brick2b: connection to failed (Connection timed out) [2010-12-03 15:59:46] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 7: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:59:47] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 8: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:59:54] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 9: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:59:55] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 10: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:59:55] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 11: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:59:55] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 12: ERR => -1 (Transport endpoint is not connected) [2010-12-03 15:59:56] W [fuse-bridge.c:1892:fuse_statfs_cbk] glusterfs-fuse: 13: ERR => -1 (Transport endpoint is not connected) However, the port is obviously open... [r...@client~]# telnet 10.8.11.102 6996 Trying 10.2.56.102... Connected to glusterserverb (10.8.11.102). Escape character is '^]'. ^] telnet> close Connection closed. The gluster server log doesnt see ANY connection attempts from the client however it DOES see my telnet tcp attempts. I'm using IP addresses in all my configuration files - no names. I do have a Juniper firewall between the two servers that is doing stateful firewalling and i've set it up for the connections to never timeout - and ive never had a problem once it finally connects. And i can create a new connection with telnet but not the client... Anybody seen anything like this before? Ideas? thanks, liam ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] random file not founds (enoent) in a distribute+replicate setup
Hi In a distribute+replicate setup, I'm randomly seeing cases where a file can't be seen by processes. A stat of the file using the full path results in an ENOENT (No such file or directory). However if I cd to the directory that it's supposed to be in, and then do an ls it appears and is fine from there on. Any ideas as to what might be happening here? This is 3.0.4 btw, I'm working to get it updated at the very least to 3.0.6 soonish but need to get this other cruft cleaned up first, etc. # ls -al 3/9/0/9/5/263259093 ls: cannot access 3/9/0/9/5/263259093: No such file or directory # ls -al 3/9/0/9/5 ... thousands of files spewed here ... # ls -al 3/9/0/9/5/263259093 -rw-r--r-- 1 2000 2000 2780068 Nov 12 23:15 3/9/0/9/5/263259093 If a backend server goes offline and comes back on later, would that cause this issue? I thought that stat'ing a file would result in the self heal and eventual consistency across all replicated pairs? The configs are standard glusterfs-volgen generated raid1 configs so no read-subvolume directives in there anywhere... Thanks. Mohan ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working
I confirm the same (bad) behavior with my install of 3.1.1 on CentOS 5.5, with secondary group membership not having access. Craig Miller On 3-Dec-10, at 2:06 PM, Mike Hanby wrote: The release notes for 3.1.1 don't mention bug 2045, however the bug itself indicates that it's fixed with a target of 3.1.1 did it actually make it into the build? -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org ] On Behalf Of Mike Hanby Sent: Friday, December 03, 2010 3:02 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working After upgrading my servers to 3.1.1 I'm still experiencing the issue with secondary groups not working with the client using NFS: In this scenario root owns the directory and the admins non primary group (of which my id is a member) is the group: $ groups mikeh admins $ ls -ld /research/project/ drwxrwx--- 10 root admins 73728 Nov 22 12:13 /gluster/project/ $ ls -l /gluster/project/ ls: /gluster/project/: Permission denied $ mount|grep research nas-01.mydom:/research-storage on /gluster type nfs (rw,addr=192.168.1.11) Both Gluster servers and the client are CentOS 5 x86_64 systems. The servers had glusterfs-core / glusterfs-fuse 3.0.1-1 updated to 3.1.1-1 via the RPMs. The storage is a simple distributed file system, no striping or replication. Following the update I rebooted both servers and remounted the NFS file system on the client. I'll add this info to the bug as well. Mike -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org ] On Behalf Of Mike Hanby Sent: Thursday, November 11, 2010 1:53 PM To: 'Shehjar Tikoo' Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Thanks, that looks to be the same issue. I'll add my comments to the bug. Mike -Original Message- From: Shehjar Tikoo [mailto:shehj...@gluster.com] Sent: Thursday, November 11, 2010 12:51 PM To: Mike Hanby Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Hi, It might be related to a bug filed at: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2045 If you please update it there or file a new one, I'll take a look. Thanks. - Original Message - From: "Mike Hanby" To: gluster-users@gluster.org Sent: Friday, November 12, 2010 12:00:23 AM Subject: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Howdy, I have a GlusterFS 3.1 volume being mounted on a client using NFS. From the client I created a directory under the mount point and set the permissions to root:groupa 750 My user account is a member of groupa on the client, yet I am unable to list the contents of the directory: $ ls -l /gfs/dir1 ls: /gfs/dir1/: Permission denied $ ls -ld /gfs/dir1 rwxr-x--- 9 root groupa 73728 Nov 9 09:44 /gfs/dir1/ $ groups myuser groupa I am able to list the directory as the user root. If I change the group ownership to my primary group, myuser, then I can successfully list the contents of the directory. $ sudo chgrp myuser /gfs/dir1 $ ls -ld /gfs/dir1 rwxr-x--- 9 root myuser 73728 Nov 9 09:44 /gfs/dir1/ $ ls -l /gfs/dir1 drwxr-xr-x 5 root root 73728 Mar 26 2010 testdir1 drwxr-x--- 4 root root 73728 Apr 8 2010 testdir2 drwxr-x--- 2 root root 73728 Aug 4 21:23 testdir3 The volume is being exported using the builtin GlusterFS NFS server. The servers and client are all CentOS 5.5 x86_64 boxes. Thanks for any suggestions, Mike = Mike Hanby mha...@uab.edu UAB School of Engineering Information Systems Specialist II IT HPCS / Research Computing ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] GlusterFS - Abysmal Speed on Fresh Install
Hi there, I'm new to the GlusterFS project - looks superb and very promising! I'm running into massive performance issues, however. I'm using the stock default configuration that GlusterFS put in place when I created the volume - it seems to reference io-cache, quick-read, etc in the volume files, which makes me think it's already pulling in these optimizations. This is a replicate setup - I've tried with 2, 3, and 4 nodes, and performance remains awful on any of them - network communication seems fine, with average speeds at 9-15MB/sec. # gluster --version glusterfs 3.1.1 built on Nov 29 2010 10:07:45 Repository revision: v3.1.1 With no other activity on the filesystems on any of my nodes: # time ls -al total 10072 drwx--x--x 10 root root 4096 Dec 3 16:50 ./ drwxr-xr-x 3 root root 4096 Dec 3 14:19 ../ -rw-r--r-- 1 root root 1024 Dec 3 16:49 testfile real0m1.347s user0m0.000s sys0m0.000s # time cp testfile testfile2 real0m11.254s user0m0.000s sys0m0.000s # time diff testfile* real0m5.792s user0m0.004s sys0m0.000s Read speed is marginally faster than write speed, but still horrible - e.g. if Apache is serving content off of a glusterfs mountpoint, it times out 95% of the time before it can read the files. I'm using mount -t glusterfs, with default mount options. Can anyone point me in the right direction to getting things nice and speedy here? I'd appreciate any feedback or help! I can provide any configuration files necessary, or even root login access to the box(es) via private e-mail if you want to poke around (these are just test boxes presently). Thanks, --Chris chris.michae...@uk2group.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working
The release notes for 3.1.1 don't mention bug 2045, however the bug itself indicates that it's fixed with a target of 3.1.1 did it actually make it into the build? -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Mike Hanby Sent: Friday, December 03, 2010 3:02 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working After upgrading my servers to 3.1.1 I'm still experiencing the issue with secondary groups not working with the client using NFS: In this scenario root owns the directory and the admins non primary group (of which my id is a member) is the group: $ groups mikeh admins $ ls -ld /research/project/ drwxrwx--- 10 root admins 73728 Nov 22 12:13 /gluster/project/ $ ls -l /gluster/project/ ls: /gluster/project/: Permission denied $ mount|grep research nas-01.mydom:/research-storage on /gluster type nfs (rw,addr=192.168.1.11) Both Gluster servers and the client are CentOS 5 x86_64 systems. The servers had glusterfs-core / glusterfs-fuse 3.0.1-1 updated to 3.1.1-1 via the RPMs. The storage is a simple distributed file system, no striping or replication. Following the update I rebooted both servers and remounted the NFS file system on the client. I'll add this info to the bug as well. Mike -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Mike Hanby Sent: Thursday, November 11, 2010 1:53 PM To: 'Shehjar Tikoo' Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Thanks, that looks to be the same issue. I'll add my comments to the bug. Mike -Original Message- From: Shehjar Tikoo [mailto:shehj...@gluster.com] Sent: Thursday, November 11, 2010 12:51 PM To: Mike Hanby Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Hi, It might be related to a bug filed at: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2045 If you please update it there or file a new one, I'll take a look. Thanks. - Original Message - > From: "Mike Hanby" > To: gluster-users@gluster.org > Sent: Friday, November 12, 2010 12:00:23 AM > Subject: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working > Howdy, > > I have a GlusterFS 3.1 volume being mounted on a client using NFS. > From the client I created a directory under the mount point and set > the permissions to root:groupa 750 > > My user account is a member of groupa on the client, yet I am unable > to list the contents of the directory: > > $ ls -l /gfs/dir1 > ls: /gfs/dir1/: Permission denied > > $ ls -ld /gfs/dir1 > rwxr-x--- 9 root groupa 73728 Nov 9 09:44 /gfs/dir1/ > > $ groups > myuser groupa > > I am able to list the directory as the user root. If I change the > group ownership to my primary group, myuser, then I can successfully > list the contents of the directory. > > $ sudo chgrp myuser /gfs/dir1 > $ ls -ld /gfs/dir1 > rwxr-x--- 9 root myuser 73728 Nov 9 09:44 /gfs/dir1/ > > $ ls -l /gfs/dir1 > drwxr-xr-x 5 root root 73728 Mar 26 2010 testdir1 > drwxr-x--- 4 root root 73728 Apr 8 2010 testdir2 > drwxr-x--- 2 root root 73728 Aug 4 21:23 testdir3 > > The volume is being exported using the builtin GlusterFS NFS server. > The servers and client are all CentOS 5.5 x86_64 boxes. > > Thanks for any suggestions, > > Mike > > = > Mike Hanby > mha...@uab.edu > UAB School of Engineering > Information Systems Specialist II > IT HPCS / Research Computing > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working
After upgrading my servers to 3.1.1 I'm still experiencing the issue with secondary groups not working with the client using NFS: In this scenario root owns the directory and the admins non primary group (of which my id is a member) is the group: $ groups mikeh admins $ ls -ld /research/project/ drwxrwx--- 10 root admins 73728 Nov 22 12:13 /gluster/project/ $ ls -l /gluster/project/ ls: /gluster/project/: Permission denied $ mount|grep research nas-01.mydom:/research-storage on /gluster type nfs (rw,addr=192.168.1.11) Both Gluster servers and the client are CentOS 5 x86_64 systems. The servers had glusterfs-core / glusterfs-fuse 3.0.1-1 updated to 3.1.1-1 via the RPMs. The storage is a simple distributed file system, no striping or replication. Following the update I rebooted both servers and remounted the NFS file system on the client. I'll add this info to the bug as well. Mike -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Mike Hanby Sent: Thursday, November 11, 2010 1:53 PM To: 'Shehjar Tikoo' Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Thanks, that looks to be the same issue. I'll add my comments to the bug. Mike -Original Message- From: Shehjar Tikoo [mailto:shehj...@gluster.com] Sent: Thursday, November 11, 2010 12:51 PM To: Mike Hanby Cc: gluster-users@gluster.org Subject: Re: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working Hi, It might be related to a bug filed at: http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2045 If you please update it there or file a new one, I'll take a look. Thanks. - Original Message - > From: "Mike Hanby" > To: gluster-users@gluster.org > Sent: Friday, November 12, 2010 12:00:23 AM > Subject: [Gluster-users] NFS Mounted GlusterFS, secondary groups not working > Howdy, > > I have a GlusterFS 3.1 volume being mounted on a client using NFS. > From the client I created a directory under the mount point and set > the permissions to root:groupa 750 > > My user account is a member of groupa on the client, yet I am unable > to list the contents of the directory: > > $ ls -l /gfs/dir1 > ls: /gfs/dir1/: Permission denied > > $ ls -ld /gfs/dir1 > rwxr-x--- 9 root groupa 73728 Nov 9 09:44 /gfs/dir1/ > > $ groups > myuser groupa > > I am able to list the directory as the user root. If I change the > group ownership to my primary group, myuser, then I can successfully > list the contents of the directory. > > $ sudo chgrp myuser /gfs/dir1 > $ ls -ld /gfs/dir1 > rwxr-x--- 9 root myuser 73728 Nov 9 09:44 /gfs/dir1/ > > $ ls -l /gfs/dir1 > drwxr-xr-x 5 root root 73728 Mar 26 2010 testdir1 > drwxr-x--- 4 root root 73728 Apr 8 2010 testdir2 > drwxr-x--- 2 root root 73728 Aug 4 21:23 testdir3 > > The volume is being exported using the builtin GlusterFS NFS server. > The servers and client are all CentOS 5.5 x86_64 boxes. > > Thanks for any suggestions, > > Mike > > = > Mike Hanby > mha...@uab.edu > UAB School of Engineering > Information Systems Specialist II > IT HPCS / Research Computing > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] 3.1.1 crashing under moderate load
I'm running GlusterFS 3.1.1, CentOS5.5 servers, CentOS5.4 clients, RDMA transport, native/fuse access. I have a directory which is shared on the gluster. In fact, it is a clone of /lib from one of the clients, shared so all can see it. I have a script which does find lib -type f -print0 | xargs -0 sum | md5sum If I run this on my clients one at a time, they all yield the same md5sum: for h in <>; do ssh $host script; done If I run this on my clients concurrently, up to roughly 25 at a time they still yield the same md5sum. for h in <>; do ssh $host script& done Beyond that the gluster share often, but not always, fails. The errors vary. - sometimes I get "sum: xxx.so not found" - sometimes I get the wrong checksum without any error message - sometimes the job simply hangs until I kill it Some of the server logs show messages like these from the time of the failures (other servers show nothing from around that time): [2010-12-03 10:03:06.34328] E [rdma.c:4442:rdma_event_handler] rpc-transport/rdma: rdma.RaidData-server: pollin received on tcp socket (peer: 10.54.255.240:1022) after handshake is complete [2010-12-03 10:03:06.34363] E [rpcsvc.c:1548:rpcsvc_submit_generic] rpc-service: failed to submit message (XID: 0x55e82, Program: GlusterFS-3.1.0, ProgVers: 310, Proc: 12) to rpc-transport (rdma.RaidData-server) [2010-12-03 10:03:06.34377] E [server.c:137:server_submit_reply] : Reply submission failed [2010-12-03 10:03:06.34464] E [rpcsvc.c:1548:rpcsvc_submit_generic] rpc-service: failed to submit message (XID: 0x55e83, Program: GlusterFS-3.1.0, ProgVers: 310, Proc: 12) to rpc-transport (rdma.RaidData-server) [2010-12-03 10:03:06.34520] E [server.c:137:server_submit_reply] : Reply submission failed On a client which had a failure I see messages like: [2010-12-03 10:03:06.21290] E [rdma.c:4442:rdma_event_handler] rpc-transport/rdma: RaidData-client-1: pollin received on tcp socket (peer: 10.54.50.101:24009) after handshake is complete [2010-12-03 10:03:06.21776] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x3814a0f769] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x3814a0ef1e] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3814a0ee8e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2010-12-03 10:03:06.20492 [2010-12-03 10:03:06.21821] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x3814a0f769] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x3814a0ef1e] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3814a0ee8e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2010-12-03 10:03:06.20529 [2010-12-03 10:03:06.26827] I [client-handshake.c:993:select_server_supported_programs] RaidData-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2010-12-03 10:03:06.27029] I [client-handshake.c:829:client_setvolume_cbk] RaidData-client-1: Connected to 10.54.50.101:24009, attached to remote volume '/data'. [2010-12-03 10:03:06.27067] I [client-handshake.c:698:client_post_handshake] RaidData-client-1: 2 fds open - Delaying child_up until they are re-opened Anyone else seen anything like this and/or have suggestions about options I can set to work around this? .. Lana (lana.de...@gmail.com) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Who's using Fedora in production on Glusterfs storage servers?
Thanks very much Mark. I am going to put up a prototype using Fedora - being able to test btrfs as well is a nice plus. For now, we apparently aren't going to be able to use Fedora in production because HP doesn't support it for their command line hardware reporting and configuration utilities (hpasmcli, hpacucli, etc). James Burnash, Unix Engineering -Original Message- From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Mark "Naoki" Rogers Sent: Wednesday, December 01, 2010 10:20 PM To: gluster-users@gluster.org Subject: Re: [Gluster-users] Who's using Fedora in production on Glusterfs storage servers? Hi James, I'm using 3.1.1 on six bricks in dist+replicate all running F14+BTRFS, the clients are on fedora12/13/14. I build the RPMs from source on a F14 machine. The cluster is running entirely on GbE (with some 10Gb lines going in shortly), no RDMA/infiniband so I can't help there. It's gone through a series of looped benchmarks for a while now (from 3.1.0 through a few qa releases) and have so far pushed/pulled over 110TB through it - I'm happy in the stability but not /entirely/ sure of the performance just yet, just started up more testing under 3.1.1. But back to your main question there really isn't enough difference between the near-term releases of Fedora for it to make a huge difference either way. I do think you're better off using the latest Fedora release than an older one that will be end of life soon (f12 tomorrow). Being able to patch/maintain your system is more important than an, often very arbitrary, vendor support list which is usually just an outcome of what people have had time to look into, rather than any measured reason a newer OS isn't supported. Besides the only thing you ever have to /really/ care about is the kernel and glibc major versions, so if it compiles you're pretty much ok (ldd it, that's all it needs). On 12/02/2010 01:45 AM, Burnash, James wrote: > How many people on the list are using Fedora 12 (or 13) in production for > Glusterfs storage servers? I know that Gluster Platform uses Fedora 12 as its > OS - I was thinking of building my new glusterfs storage servers using > Fedora, and was wondering whether Fedora 13 was tested by Gluster for v 3.1.1 > and what other people's experiences were. > > One of the reasons for my interest was so that I could use ext4 as the > backend file store, instead of ext3. > > Thanks, > > James Burnash, Unix Engineering ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Incomprehensive errors
Hi there, I have a lot (one each second) of these errors : [2010-12-03 16:33:53.600610] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 248602: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:53.658075] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 248696: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:53.685461] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 248748: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:53.992316] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 248905: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:54.22034] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 248960: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:54.287419] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 249502: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:54.744890] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 249514: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:56.148194] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 250798: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:56.186352] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 250892: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:57.307090] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 251415: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:57.978952] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 252582: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) [2010-12-03 16:33:58.596328] W [fuse-bridge.c:2506:fuse_getxattr] glusterfs-fuse: 252715: GETXATTR (null)/41872720 (security.capability) (fuse_loc_fill() failed) What does it mean? Thanks for your answers. Regards. Samuel Hassine ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
That makes sense. What software adjustments should I make to ensure there won't be any problems? Thank you. On Fri, Dec 3, 2010 at 7:53 AM, Raghavendra G wrote: > From the logs its evident that the reason for completion queue creation > failure is that the number of completion queue elements (in a completion > queue) we had requested in ibv_create_cq, (1024 * send_count) is less than > the maximum supported by the ib hardware (max_cqe = 131071). > > - Original Message - > From: "Jeremy Stout" > To: "Raghavendra G" > Cc: gluster-users@gluster.org > Sent: Friday, December 3, 2010 4:20:04 PM > Subject: Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1 > > I patched the source code and rebuilt GlusterFS. Here are the full logs: > Server: > [2010-12-03 07:08:55.945804] I [glusterd.c:275:init] management: Using > /etc/glusterd as working directory > [2010-12-03 07:08:55.947692] E [rdma.c:2047:rdma_create_cq] > rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq = > 65408, max_cqe = 131071, max_mr = 131056 > [2010-12-03 07:08:55.953226] E [rdma.c:2079:rdma_create_cq] > rpc-transport/rdma: rdma.management: creation of send_cq failed > [2010-12-03 07:08:55.953509] E [rdma.c:3785:rdma_get_device] > rpc-transport/rdma: rdma.management: could not create CQ > [2010-12-03 07:08:55.953582] E [rdma.c:3971:rdma_init] > rpc-transport/rdma: could not create rdma device for mthca0 > [2010-12-03 07:08:55.953668] E [rdma.c:4803:init] rdma.management: > Failed to initialize IB Device > [2010-12-03 07:08:55.953691] E > [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma' > initialization failed > [2010-12-03 07:08:55.953780] I [glusterd.c:96:glusterd_uuid_init] > glusterd: generated UUID: 4eb47ca7-227c-49c4-97bd-25ac177b2f6a > Given volfile: > +--+ > 1: volume management > 2: type mgmt/glusterd > 3: option working-directory /etc/glusterd > 4: option transport-type socket,rdma > 5: option transport.socket.keepalive-time 10 > 6: option transport.socket.keepalive-interval 2 > 7: end-volume > 8: > > +--+ > [2010-12-03 07:09:10.244790] I > [glusterd-handler.c:785:glusterd_handle_create_volume] glusterd: > Received create volume req > [2010-12-03 07:09:10.247646] I [glusterd-utils.c:232:glusterd_lock] > glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a > [2010-12-03 07:09:10.247678] I > [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired > local lock > [2010-12-03 07:09:10.247708] I > [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock > req to 0 peers > [2010-12-03 07:09:10.248038] I > [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req > to 0 peers > [2010-12-03 07:09:10.251970] I > [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req > to 0 peers > [2010-12-03 07:09:10.252020] I > [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent > unlock req to 0 peers > [2010-12-03 07:09:10.252036] I > [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared > local lock > [2010-12-03 07:09:22.11649] I > [glusterd-handler.c:936:glusterd_handle_cli_start_volume] glusterd: > Received start vol reqfor volume testdir > [2010-12-03 07:09:22.11724] I [glusterd-utils.c:232:glusterd_lock] > glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a > [2010-12-03 07:09:22.11734] I > [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired > local lock > [2010-12-03 07:09:22.11761] I > [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock > req to 0 peers > [2010-12-03 07:09:22.12120] I > [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req > to 0 peers > [2010-12-03 07:09:22.184403] I > [glusterd-utils.c:971:glusterd_volume_start_glusterfs] : About to > start glusterfs for brick pgh-submit-1:/mnt/gluster > [2010-12-03 07:09:22.229143] I > [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req > to 0 peers > [2010-12-03 07:09:22.229198] I > [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent > unlock req to 0 peers > [2010-12-03 07:09:22.229218] I > [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared > local lock > [2010-12-03 07:09:22.240157] I > [glusterd-pmap.c:281:pmap_registry_remove] pmap: removing brick (null) > on port 24009 > > > Client: > [2010-12-03 07:09:00.82784] W [io-stats.c:1644:init] testdir: dangling > volume. check volfile > [2010-12-03 07:09:00.82824] W [dict.c:1204:data_to_str] dict: @data=(nil) > [2010-12-03 07:09:00.82836] W [dict.c:1204:data_to_str] dict: @data=(nil) > [2010-12-03 07:09:00.85980] E [rdma.c:2047:rdma_create_cq] > rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq = > 65408, max_cqe = 131071, max_mr = 131056 > [2010-12-03 07:09:00.92883] E [rdma.c:2079:rdma_create_cq] > rpc-transp
Re: [Gluster-users] performance stops at 1Gb
Do both read and write throughput peak at 1Gbit/s? What is the block size used for performing I/O? Can you get the output of - 1. dd if=/dev/zero of=/mnt/stripe/file bs=1M count=1K 2. dd if=/mnt/stripe/file of=/dev/null bs=1M count=1K Just one instance of dd is enough as the client network interface (10Gbit/s) has enough juice to saturate 4x1Gbit servers. Avati On Fri, Dec 3, 2010 at 6:06 PM, Gotwalt, P. wrote: > Craig, > > Using multiple parallel bonnie++ benchmarks (4,8,16) does use several > files. These file are 1GB each, and we take care there will be at least > 32 of them. As we have multiple processes (4,8,16 bonnie++s) and each > uses several files, we spread the io over different storage bricks. I > can see this when monitoring network and disk activity on the bricks. > For example: when bonnie++ does block read/writes on a striped (4 > bricks) volume I notice that the load of the client (network throughput) > is evenly spread over the 4 nodes. These nodes have enough cpu, memory, > network and disk resources left! The accoumulated throughput doesn't get > over the 1 Gb. > The 10Gb nic at the client is set to fixed 10Gb, full duplex, All the > nics on the storage bricks are 1Gb, fixed, full duplex. The 10Gb client > (dual quadcore, 16GB) has plenty of resources to run 16 bonnie++s > parallel. We should be able to get more than this 1Gb throughput, > especially with a striped volume. > > What kind of benchmarks do you run? And with what kind of setup? > > Peter > > > > > Peter - > > Using Gluster the performance of any single file is going to be > > limited to the performance of the server on which it exists, or in the > > > case of a striped volume of the server on which the segment of the > file > > you are accessing exists. If you were able to start 4 processes, > > accessing different parts of the striped file, or lots of different > > files in a distribute cluster you would see your performance increase > > significantly. > > > Thanks, > > > Craig > > > --> > > Craig Carl > > Senior Systems Engineer > > Gluster > > > > > > On 11/26/2010 07:57 AM, Gotwalt, P. wrote: > > > Hi All, > > > > > > I am doing some tests with gluster (3.1) and have a problem of not > > > getting higher throughput than 1 Gb (yes bit!) with 4 storage > bricks. > > > My setup: > > > > > > 4 storage bricks (dualcore, 4GB mem) each with 3 sata 1Tb disks, > > > connected to a switch with 1 Gb nics. In my tests I only use 1 SATA > > > disk as a volume, per brick. > > > 1 client (2xquad core, 16 GB mem) with a 10Gb nic to the same switch > as > > > the bricks. > > > > > > When using striped of distributed configurations, with all 4 bricks > > > configured to act as a server, the performance will never be higher > than > > > just below 1 Gb! I tested with 4, 8 and 16 parallel bonnie++ runs. > > > > > > The idea is that parallel bonnie's create enough files to get > > > distributed over the storage bricks. And all this bonnie's wil > deliver > > > enough throughput to fill up this 10Gb line. I expect the throughput > to > > > be maximum 4Gb because that's the maximum the 4 storage bricks > together > > > can produce. > > > > > > I also tested the throughput of the network with iperf3 and got: > > > - 5Gb to a second temporary client on another switch 200 Km from my > > > site, connected with a 5Gb fiber > > > - 908-920 Mb to the interfaces of the bricks. > > > So the network seems ok. > > > > > > Can someone advise me on why I don't get 4Gb? Or can someone advise > me > > > on a better setup with the equipment I have? > > > > > > > > > Peter Gotwalt > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
From the logs its evident that the reason for completion queue creation failure is that the number of completion queue elements (in a completion queue) we had requested in ibv_create_cq, (1024 * send_count) is less than the maximum supported by the ib hardware (max_cqe = 131071). - Original Message - From: "Jeremy Stout" To: "Raghavendra G" Cc: gluster-users@gluster.org Sent: Friday, December 3, 2010 4:20:04 PM Subject: Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1 I patched the source code and rebuilt GlusterFS. Here are the full logs: Server: [2010-12-03 07:08:55.945804] I [glusterd.c:275:init] management: Using /etc/glusterd as working directory [2010-12-03 07:08:55.947692] E [rdma.c:2047:rdma_create_cq] rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq = 65408, max_cqe = 131071, max_mr = 131056 [2010-12-03 07:08:55.953226] E [rdma.c:2079:rdma_create_cq] rpc-transport/rdma: rdma.management: creation of send_cq failed [2010-12-03 07:08:55.953509] E [rdma.c:3785:rdma_get_device] rpc-transport/rdma: rdma.management: could not create CQ [2010-12-03 07:08:55.953582] E [rdma.c:3971:rdma_init] rpc-transport/rdma: could not create rdma device for mthca0 [2010-12-03 07:08:55.953668] E [rdma.c:4803:init] rdma.management: Failed to initialize IB Device [2010-12-03 07:08:55.953691] E [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma' initialization failed [2010-12-03 07:08:55.953780] I [glusterd.c:96:glusterd_uuid_init] glusterd: generated UUID: 4eb47ca7-227c-49c4-97bd-25ac177b2f6a Given volfile: +--+ 1: volume management 2: type mgmt/glusterd 3: option working-directory /etc/glusterd 4: option transport-type socket,rdma 5: option transport.socket.keepalive-time 10 6: option transport.socket.keepalive-interval 2 7: end-volume 8: +--+ [2010-12-03 07:09:10.244790] I [glusterd-handler.c:785:glusterd_handle_create_volume] glusterd: Received create volume req [2010-12-03 07:09:10.247646] I [glusterd-utils.c:232:glusterd_lock] glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a [2010-12-03 07:09:10.247678] I [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired local lock [2010-12-03 07:09:10.247708] I [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock req to 0 peers [2010-12-03 07:09:10.248038] I [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:10.251970] I [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:10.252020] I [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 0 peers [2010-12-03 07:09:10.252036] I [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared local lock [2010-12-03 07:09:22.11649] I [glusterd-handler.c:936:glusterd_handle_cli_start_volume] glusterd: Received start vol reqfor volume testdir [2010-12-03 07:09:22.11724] I [glusterd-utils.c:232:glusterd_lock] glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a [2010-12-03 07:09:22.11734] I [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired local lock [2010-12-03 07:09:22.11761] I [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock req to 0 peers [2010-12-03 07:09:22.12120] I [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:22.184403] I [glusterd-utils.c:971:glusterd_volume_start_glusterfs] : About to start glusterfs for brick pgh-submit-1:/mnt/gluster [2010-12-03 07:09:22.229143] I [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:22.229198] I [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 0 peers [2010-12-03 07:09:22.229218] I [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared local lock [2010-12-03 07:09:22.240157] I [glusterd-pmap.c:281:pmap_registry_remove] pmap: removing brick (null) on port 24009 Client: [2010-12-03 07:09:00.82784] W [io-stats.c:1644:init] testdir: dangling volume. check volfile [2010-12-03 07:09:00.82824] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-12-03 07:09:00.82836] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-12-03 07:09:00.85980] E [rdma.c:2047:rdma_create_cq] rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq = 65408, max_cqe = 131071, max_mr = 131056 [2010-12-03 07:09:00.92883] E [rdma.c:2079:rdma_create_cq] rpc-transport/rdma: testdir-client-0: creation of send_cq failed [2010-12-03 07:09:00.93156] E [rdma.c:3785:rdma_get_device] rpc-transport/rdma: testdir-client-0: could not create CQ [2010-12-03 07:09:00.93224] E [rdma.c:3971:rdma_init] rpc-transport/rdma: could not create rdma device for mthca0 [2010-12-03 07:09:00.93313] E [rdma.c:4803:init] testdir-client-0: Failed to init
Re: [Gluster-users] performance stops at 1Gb
Craig, Using multiple parallel bonnie++ benchmarks (4,8,16) does use several files. These file are 1GB each, and we take care there will be at least 32 of them. As we have multiple processes (4,8,16 bonnie++s) and each uses several files, we spread the io over different storage bricks. I can see this when monitoring network and disk activity on the bricks. For example: when bonnie++ does block read/writes on a striped (4 bricks) volume I notice that the load of the client (network throughput) is evenly spread over the 4 nodes. These nodes have enough cpu, memory, network and disk resources left! The accoumulated throughput doesn't get over the 1 Gb. The 10Gb nic at the client is set to fixed 10Gb, full duplex, All the nics on the storage bricks are 1Gb, fixed, full duplex. The 10Gb client (dual quadcore, 16GB) has plenty of resources to run 16 bonnie++s parallel. We should be able to get more than this 1Gb throughput, especially with a striped volume. What kind of benchmarks do you run? And with what kind of setup? Peter > Peter - > Using Gluster the performance of any single file is going to be > limited to the performance of the server on which it exists, or in the > case of a striped volume of the server on which the segment of the file > you are accessing exists. If you were able to start 4 processes, > accessing different parts of the striped file, or lots of different > files in a distribute cluster you would see your performance increase > significantly. > Thanks, > Craig > --> > Craig Carl > Senior Systems Engineer > Gluster > > > On 11/26/2010 07:57 AM, Gotwalt, P. wrote: > > Hi All, > > > > I am doing some tests with gluster (3.1) and have a problem of not > > getting higher throughput than 1 Gb (yes bit!) with 4 storage bricks. > > My setup: > > > > 4 storage bricks (dualcore, 4GB mem) each with 3 sata 1Tb disks, > > connected to a switch with 1 Gb nics. In my tests I only use 1 SATA > > disk as a volume, per brick. > > 1 client (2xquad core, 16 GB mem) with a 10Gb nic to the same switch as > > the bricks. > > > > When using striped of distributed configurations, with all 4 bricks > > configured to act as a server, the performance will never be higher than > > just below 1 Gb! I tested with 4, 8 and 16 parallel bonnie++ runs. > > > > The idea is that parallel bonnie's create enough files to get > > distributed over the storage bricks. And all this bonnie's wil deliver > > enough throughput to fill up this 10Gb line. I expect the throughput to > > be maximum 4Gb because that's the maximum the 4 storage bricks together > > can produce. > > > > I also tested the throughput of the network with iperf3 and got: > > - 5Gb to a second temporary client on another switch 200 Km from my > > site, connected with a 5Gb fiber > > - 908-920 Mb to the interfaces of the bricks. > > So the network seems ok. > > > > Can someone advise me on why I don't get 4Gb? Or can someone advise me > > on a better setup with the equipment I have? > > > > > > Peter Gotwalt ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Start a new volume with pre-existing directories
Hi! >Can you send the output of - > >`gluster volume info all` >`gluster peer status` > >from a gluster storage server and > >`mount` from the client? Certainly -- r...@ekpfs8:~# gluster volume info all Volume Name: heal_me Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 192.168.101.246:/storage/8 Brick2: 192.168.101.247:/storage/7 -- r...@ekpfs8:~# gluster peer status Number of Peers: 1 Hostname: 192.168.101.247 Uuid: b36ce6e3-fa14-4d7e-bc4a-170a59a6f4f5 State: Peer in Cluster (Connected) -- [r...@ekpbelle ~]# mount [ ... ] glusterfs#192.168.101.246:/heal_me on /storage/gluster type fuse (rw,allow_other,default_permissions,max_read=131072) -- Regards, Daniel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
I patched the source code and rebuilt GlusterFS. Here are the full logs: Server: [2010-12-03 07:08:55.945804] I [glusterd.c:275:init] management: Using /etc/glusterd as working directory [2010-12-03 07:08:55.947692] E [rdma.c:2047:rdma_create_cq] rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq = 65408, max_cqe = 131071, max_mr = 131056 [2010-12-03 07:08:55.953226] E [rdma.c:2079:rdma_create_cq] rpc-transport/rdma: rdma.management: creation of send_cq failed [2010-12-03 07:08:55.953509] E [rdma.c:3785:rdma_get_device] rpc-transport/rdma: rdma.management: could not create CQ [2010-12-03 07:08:55.953582] E [rdma.c:3971:rdma_init] rpc-transport/rdma: could not create rdma device for mthca0 [2010-12-03 07:08:55.953668] E [rdma.c:4803:init] rdma.management: Failed to initialize IB Device [2010-12-03 07:08:55.953691] E [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma' initialization failed [2010-12-03 07:08:55.953780] I [glusterd.c:96:glusterd_uuid_init] glusterd: generated UUID: 4eb47ca7-227c-49c4-97bd-25ac177b2f6a Given volfile: +--+ 1: volume management 2: type mgmt/glusterd 3: option working-directory /etc/glusterd 4: option transport-type socket,rdma 5: option transport.socket.keepalive-time 10 6: option transport.socket.keepalive-interval 2 7: end-volume 8: +--+ [2010-12-03 07:09:10.244790] I [glusterd-handler.c:785:glusterd_handle_create_volume] glusterd: Received create volume req [2010-12-03 07:09:10.247646] I [glusterd-utils.c:232:glusterd_lock] glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a [2010-12-03 07:09:10.247678] I [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired local lock [2010-12-03 07:09:10.247708] I [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock req to 0 peers [2010-12-03 07:09:10.248038] I [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:10.251970] I [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:10.252020] I [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 0 peers [2010-12-03 07:09:10.252036] I [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared local lock [2010-12-03 07:09:22.11649] I [glusterd-handler.c:936:glusterd_handle_cli_start_volume] glusterd: Received start vol reqfor volume testdir [2010-12-03 07:09:22.11724] I [glusterd-utils.c:232:glusterd_lock] glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a [2010-12-03 07:09:22.11734] I [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired local lock [2010-12-03 07:09:22.11761] I [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock req to 0 peers [2010-12-03 07:09:22.12120] I [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:22.184403] I [glusterd-utils.c:971:glusterd_volume_start_glusterfs] : About to start glusterfs for brick pgh-submit-1:/mnt/gluster [2010-12-03 07:09:22.229143] I [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req to 0 peers [2010-12-03 07:09:22.229198] I [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 0 peers [2010-12-03 07:09:22.229218] I [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared local lock [2010-12-03 07:09:22.240157] I [glusterd-pmap.c:281:pmap_registry_remove] pmap: removing brick (null) on port 24009 Client: [2010-12-03 07:09:00.82784] W [io-stats.c:1644:init] testdir: dangling volume. check volfile [2010-12-03 07:09:00.82824] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-12-03 07:09:00.82836] W [dict.c:1204:data_to_str] dict: @data=(nil) [2010-12-03 07:09:00.85980] E [rdma.c:2047:rdma_create_cq] rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq = 65408, max_cqe = 131071, max_mr = 131056 [2010-12-03 07:09:00.92883] E [rdma.c:2079:rdma_create_cq] rpc-transport/rdma: testdir-client-0: creation of send_cq failed [2010-12-03 07:09:00.93156] E [rdma.c:3785:rdma_get_device] rpc-transport/rdma: testdir-client-0: could not create CQ [2010-12-03 07:09:00.93224] E [rdma.c:3971:rdma_init] rpc-transport/rdma: could not create rdma device for mthca0 [2010-12-03 07:09:00.93313] E [rdma.c:4803:init] testdir-client-0: Failed to initialize IB Device [2010-12-03 07:09:00.93332] E [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma' initialization failed Given volfile: +--+ 1: volume testdir-client-0 2: type protocol/client 3: option remote-host submit-1 4: option remote-subvolume /mnt/gluster 5: option transport-type rdma 6: end-volume 7: 8: volume testdir-write-behind 9: type performance/write-be
Re: [Gluster-users] Anormal Gluster shutdown
Release 3.0.x has the same behavior what you describe here, which is not really an issue. Please verify if some other component in your system changed (kernel or fuse kernel module) which has changed the behavior for you. Avati On Thu, Dec 2, 2010 at 7:29 PM, Samuel Hassine wrote: > Hi all, > > > > GlusterFS partition automatically shutdown when umounting a binded mount > point > > with "-f" option (without it works). > > > > How to reproduce: > > > > mounted Gluster partition on /gluster (any config): > > > > df: localhost:/gluster4.5T 100G 4.4T 3% /gluster > > mount: localhost:/gluster on /gluster type fuse.glusterfs > > (rw,allow_other,default_permissions,max_read=131072) > > > > commands: > > > > mkdir /test > > mount -n --bind /gluster /test > > ls /test (verify you have the Gluster) > > > > and: > > > > umount -f /test > > > > ===> > > > > df: `/gluster': Transport endpoint is not connected > > [2010-12-02 14:48:56.38309] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: > > unmounting /gluster > > [2010-12-02 14:48:56.38364] I [glusterfsd.c:672:cleanup_and_exit] > glusterfsd: > > shutting down > > > > Before 3.1.x I did not have this bug. > > > > Regards. > > Sam > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Start a new volume with pre-existing directories
Can you send the output of - `gluster volume info all` `gluster peer status` from a gluster storage server and `mount` from the client? Craig On 12/03/2010 02:50 AM, Daniel Zander wrote: Dear Craig, I'm afraid I wasn't able to start a self heal the way you suggested. I tested the following: -- WHAT I DID -- I created on fs7:/storage/7 user_7_1 user_7_2 user_7_3 and on fs8:/storage/8 user_8_1 user_8_2 user_8_3 and filled all of the directories with some small files and subdirectories. Then, on fs8: gluster volume create heal_me transport tcp 192.168.101.246:/storage/8 192.168.101.247:/storage/7 Then on fs8 and afterwards on fs7: mount -t glusterfs localhost:/heal_me /tempmount/ cd /tempmount find . | xargs stat >>/dev/null 2>&1 umount /tempmount All went well, no error messages or anything. The output of `find . | xargs stat` is probably too long to post it here, but there were no error messages or anything else that would look suspicious to me. --- RESULTS --- ls fs8:storage/8 user_8_1 user_8_2 user_8_3 ls fs7:/storage/7 user_7_1 user_7_2 user_7_3 user_8_1 user_8_2 user_8_3 ls client:/storage/gluster user_8_1 user_8_2 user_8_3 ls fs7:/tempmount user_8_1 user_8_2 user_8_3 ls fs8:/tempmount user_8_1 user_8_2 user_8_3 Unmounting and remounting has no effect. Servers are both Ubuntu Server 10.4, client is CentOS 5, 64bits all around. Thanks and regards, Daniel On 12/03/2010 10:10 AM, Craig Carl wrote: Daniel - If you want to export existing data you will need to run the self heal process so extended attributes can get written. While this should work without any issues it isn't an officially supported process, please make sure you have complete and up to date backups. After you have setup and started the Gluster volume mount it locally on one of the servers using `mount -t glusterfs localhost:/ /`. CD into the root of the mount point and run `find . | xargs stat >>/dev/null 2>&1` to start a self heal. Also the command you used to create the volume should not have worked, it is missing a volume name - gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8, typo maybe? Please let us know how it goes, and please let me know if you have any other questions. Thanks, Craig --> Craig Carl Senior Systems Engineer; Gluster, Inc. Cell - (408) 829-9953 (California, USA) Office - (408) 770-1884 Gtalk - craig.c...@gmail.com Twitter - @gluster http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ On 12/02/2010 11:38 PM, Daniel Zander wrote: Dear all, at our institute, we currently have 6 file servers, each one of them individually mounted via NFS on ~ 20 clients. The structure on the servers and the clients is the following: /storage/1/ (NFS export from FS1) /storage/2/ (NFS export from FS2) etc ... Recently, we decided that we would like to migrate this to glusterFS, so that we can have one big storage directory on the clients. Let's call it /gluster/ I tried to set up a gluster volume with two empty fileservers and it worked without any problems. I could easily mount it on a client and use it (using the native glusterFS mount). If we now want to migrate the entire institute, it would be very convenient, if existing folders could be easily included into a new volume. I tried to do this, but I did not succeed. Here's a short description of what I tried: Existing folders: on fs7: /storage/7/user_1,user_2 on fs8: /storage/8/user_3,user_4 gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8 I hoped to see on the client: /gluster/user_1 /gluster/user_2 /gluster/user_3 /gluster/user_4 The creation was successful, the volume could be started and mounted. On the client, however, I could only find (via "ls /gluster") the directories user_1 and user_2. But when I tried "cd /gluster/user_3", it succeeded! Now "ls /gluster" showed me user_1, user_2 and user_3. Unfortunately, user_3's subdirectories and files were still invisible, but with the above mentioned trick, I could make them visible. This is however not an option, as there are too much users and too complicated file structures to do this manually. It anyhow seems like Voodoo to me. Is it possible to include all of the existing directories in the new glusterFS volume? If yes: how? Thank you in advance for your efforts, Regards, Daniel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-
Re: [Gluster-users] Start a new volume with pre-existing directories
Dear Craig, I'm afraid I wasn't able to start a self heal the way you suggested. I tested the following: -- WHAT I DID -- I created on fs7:/storage/7 user_7_1 user_7_2 user_7_3 and on fs8:/storage/8 user_8_1 user_8_2 user_8_3 and filled all of the directories with some small files and subdirectories. Then, on fs8: gluster volume create heal_me transport tcp 192.168.101.246:/storage/8 192.168.101.247:/storage/7 Then on fs8 and afterwards on fs7: mount -t glusterfs localhost:/heal_me /tempmount/ cd /tempmount find . | xargs stat >>/dev/null 2>&1 umount /tempmount All went well, no error messages or anything. The output of `find . | xargs stat` is probably too long to post it here, but there were no error messages or anything else that would look suspicious to me. --- RESULTS --- ls fs8:storage/8 user_8_1 user_8_2 user_8_3 ls fs7:/storage/7 user_7_1 user_7_2 user_7_3 user_8_1 user_8_2 user_8_3 ls client:/storage/gluster user_8_1 user_8_2 user_8_3 ls fs7:/tempmount user_8_1 user_8_2 user_8_3 ls fs8:/tempmount user_8_1 user_8_2 user_8_3 Unmounting and remounting has no effect. Servers are both Ubuntu Server 10.4, client is CentOS 5, 64bits all around. Thanks and regards, Daniel On 12/03/2010 10:10 AM, Craig Carl wrote: Daniel - If you want to export existing data you will need to run the self heal process so extended attributes can get written. While this should work without any issues it isn't an officially supported process, please make sure you have complete and up to date backups. After you have setup and started the Gluster volume mount it locally on one of the servers using `mount -t glusterfs localhost:/ /`. CD into the root of the mount point and run `find . | xargs stat >>/dev/null 2>&1` to start a self heal. Also the command you used to create the volume should not have worked, it is missing a volume name - gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8, typo maybe? Please let us know how it goes, and please let me know if you have any other questions. Thanks, Craig --> Craig Carl Senior Systems Engineer; Gluster, Inc. Cell - (408) 829-9953 (California, USA) Office - (408) 770-1884 Gtalk - craig.c...@gmail.com Twitter - @gluster http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ On 12/02/2010 11:38 PM, Daniel Zander wrote: Dear all, at our institute, we currently have 6 file servers, each one of them individually mounted via NFS on ~ 20 clients. The structure on the servers and the clients is the following: /storage/1/ (NFS export from FS1) /storage/2/ (NFS export from FS2) etc ... Recently, we decided that we would like to migrate this to glusterFS, so that we can have one big storage directory on the clients. Let's call it /gluster/ I tried to set up a gluster volume with two empty fileservers and it worked without any problems. I could easily mount it on a client and use it (using the native glusterFS mount). If we now want to migrate the entire institute, it would be very convenient, if existing folders could be easily included into a new volume. I tried to do this, but I did not succeed. Here's a short description of what I tried: Existing folders: on fs7: /storage/7/user_1,user_2 on fs8: /storage/8/user_3,user_4 gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8 I hoped to see on the client: /gluster/user_1 /gluster/user_2 /gluster/user_3 /gluster/user_4 The creation was successful, the volume could be started and mounted. On the client, however, I could only find (via "ls /gluster") the directories user_1 and user_2. But when I tried "cd /gluster/user_3", it succeeded! Now "ls /gluster" showed me user_1, user_2 and user_3. Unfortunately, user_3's subdirectories and files were still invisible, but with the above mentioned trick, I could make them visible. This is however not an option, as there are too much users and too complicated file structures to do this manually. It anyhow seems like Voodoo to me. Is it possible to include all of the existing directories in the new glusterFS volume? If yes: how? Thank you in advance for your efforts, Regards, Daniel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Anormal Gluster shutdown
Samuel - I was able to recreate the failure and have updated the bug you filed. Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster On 12/03/2010 01:24 AM, Samuel Hassine wrote: Craig, I am using Debian Lenny (Proxmox 1.7) r...@on-003:/# uname -a Linux on-003 2.6.32-3-pve #1 SMP Fri Sep 17 17:56:13 CEST 2010 x86_64 GNU/Linux On all Gluster nodes and gluster clients. For hardware, it is sata disks with an LVM partition of 2.2To, in distributed-replicated Gluster. I tested this morning and the problem is still here. Regards. Sam -Message d'origine- De : gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] De la part de Craig Carl Envoyé : vendredi 3 décembre 2010 09:02 À : gluster-users@gluster.org Objet : Re: [Gluster-users] Anormal Gluster shutdown Samuel - I can't reproduce this issue locally, can you send me operating system and hardware details for both the Gluster servers and the client? Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster On 12/02/2010 05:59 AM, Samuel Hassine wrote: Hi all, GlusterFS partition automatically shutdown when umounting a binded mount point with "-f" option (without it works). How to reproduce: mounted Gluster partition on /gluster (any config): df: localhost:/gluster4.5T 100G 4.4T 3% /gluster mount: localhost:/gluster on /gluster type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072) commands: mkdir /test mount -n --bind /gluster /test ls /test (verify you have the Gluster) and: umount -f /test ===> df: `/gluster': Transport endpoint is not connected [2010-12-02 14:48:56.38309] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: unmounting /gluster [2010-12-02 14:48:56.38364] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down Before 3.1.x I did not have this bug. Regards. Sam ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Start a new volume with pre-existing directories
Hi! > Also the command you used to create the volume should not have worked, > it is missing a volume name - gluster volume create transport > tcp fs7:/storage/7, fs8:/storage/8, typo maybe? Yes, typo. Sorry ... Unfortunately, we do not have the storage capacity for a complete backup. If we should decide to take the risk, I will let you know how it goes. Thanks for your help, Daniel On 12/03/2010 10:10 AM, Craig Carl wrote: Daniel - If you want to export existing data you will need to run the self heal process so extended attributes can get written. While this should work without any issues it isn't an officially supported process, please make sure you have complete and up to date backups. After you have setup and started the Gluster volume mount it locally on one of the servers using `mount -t glusterfs localhost:/ /`. CD into the root of the mount point and run `find . | xargs stat >>/dev/null 2>&1` to start a self heal. Also the command you used to create the volume should not have worked, it is missing a volume name - gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8, typo maybe? Please let us know how it goes, and please let me know if you have any other questions. Thanks, Craig --> Craig Carl Senior Systems Engineer; Gluster, Inc. Cell - (408) 829-9953 (California, USA) Office - (408) 770-1884 Gtalk - craig.c...@gmail.com Twitter - @gluster http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ On 12/02/2010 11:38 PM, Daniel Zander wrote: Dear all, at our institute, we currently have 6 file servers, each one of them individually mounted via NFS on ~ 20 clients. The structure on the servers and the clients is the following: /storage/1/ (NFS export from FS1) /storage/2/ (NFS export from FS2) etc ... Recently, we decided that we would like to migrate this to glusterFS, so that we can have one big storage directory on the clients. Let's call it /gluster/ I tried to set up a gluster volume with two empty fileservers and it worked without any problems. I could easily mount it on a client and use it (using the native glusterFS mount). If we now want to migrate the entire institute, it would be very convenient, if existing folders could be easily included into a new volume. I tried to do this, but I did not succeed. Here's a short description of what I tried: Existing folders: on fs7: /storage/7/user_1,user_2 on fs8: /storage/8/user_3,user_4 gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8 I hoped to see on the client: /gluster/user_1 /gluster/user_2 /gluster/user_3 /gluster/user_4 The creation was successful, the volume could be started and mounted. On the client, however, I could only find (via "ls /gluster") the directories user_1 and user_2. But when I tried "cd /gluster/user_3", it succeeded! Now "ls /gluster" showed me user_1, user_2 and user_3. Unfortunately, user_3's subdirectories and files were still invisible, but with the above mentioned trick, I could make them visible. This is however not an option, as there are too much users and too complicated file structures to do this manually. It anyhow seems like Voodoo to me. Is it possible to include all of the existing directories in the new glusterFS volume? If yes: how? Thank you in advance for your efforts, Regards, Daniel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Anormal Gluster shutdown
Craig, I am using Debian Lenny (Proxmox 1.7) r...@on-003:/# uname -a Linux on-003 2.6.32-3-pve #1 SMP Fri Sep 17 17:56:13 CEST 2010 x86_64 GNU/Linux On all Gluster nodes and gluster clients. For hardware, it is sata disks with an LVM partition of 2.2To, in distributed-replicated Gluster. I tested this morning and the problem is still here. Regards. Sam -Message d'origine- De : gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] De la part de Craig Carl Envoyé : vendredi 3 décembre 2010 09:02 À : gluster-users@gluster.org Objet : Re: [Gluster-users] Anormal Gluster shutdown Samuel - I can't reproduce this issue locally, can you send me operating system and hardware details for both the Gluster servers and the client? Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster On 12/02/2010 05:59 AM, Samuel Hassine wrote: > Hi all, > > > > GlusterFS partition automatically shutdown when umounting a binded > mount point > > with "-f" option (without it works). > > > > How to reproduce: > > > > mounted Gluster partition on /gluster (any config): > > > > df: localhost:/gluster4.5T 100G 4.4T 3% /gluster > > mount: localhost:/gluster on /gluster type fuse.glusterfs > > (rw,allow_other,default_permissions,max_read=131072) > > > > commands: > > > > mkdir /test > > mount -n --bind /gluster /test > > ls /test (verify you have the Gluster) > > > > and: > > > > umount -f /test > > > > ===> > > > > df: `/gluster': Transport endpoint is not connected > > [2010-12-02 14:48:56.38309] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: > > unmounting /gluster > > [2010-12-02 14:48:56.38364] I [glusterfsd.c:672:cleanup_and_exit] > glusterfsd: > > shutting down > > > > Before 3.1.x I did not have this bug. > > > > Regards. > > Sam > > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Storage Platform 3.1.1?
Christian - We're working on it everyday but we don't have a release date yet. As soon as we have a date we will send a note to the list. Thanks for your interest in Gluster and please let me know if you have any other questions. Thanks again, Craig --> Craig Carl Senior Systems Engineer; Gluster, Inc. Cell - (408) 829-9953 (California, USA) Office - (408) 770-1884 Gtalk - craig.c...@gmail.com Twitter - @gluster http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ On 12/03/2010 12:48 AM, Christian Fischer wrote: Sorry to bother you. Is there a planned release date for Storage Platform 3.1.1? Thanks Christian ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Start a new volume with pre-existing directories
Daniel - If you want to export existing data you will need to run the self heal process so extended attributes can get written. While this should work without any issues it isn't an officially supported process, please make sure you have complete and up to date backups. After you have setup and started the Gluster volume mount it locally on one of the servers using `mount -t glusterfs localhost:/ /`. CD into the root of the mount point and run `find . | xargs stat >>/dev/null 2>&1` to start a self heal. Also the command you used to create the volume should not have worked, it is missing a volume name - gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8, typo maybe? Please let us know how it goes, and please let me know if you have any other questions. Thanks, Craig --> Craig Carl Senior Systems Engineer; Gluster, Inc. Cell - (408) 829-9953 (California, USA) Office - (408) 770-1884 Gtalk - craig.c...@gmail.com Twitter - @gluster http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ On 12/02/2010 11:38 PM, Daniel Zander wrote: Dear all, at our institute, we currently have 6 file servers, each one of them individually mounted via NFS on ~ 20 clients. The structure on the servers and the clients is the following: /storage/1/ (NFS export from FS1) /storage/2/ (NFS export from FS2) etc ... Recently, we decided that we would like to migrate this to glusterFS, so that we can have one big storage directory on the clients. Let's call it /gluster/ I tried to set up a gluster volume with two empty fileservers and it worked without any problems. I could easily mount it on a client and use it (using the native glusterFS mount). If we now want to migrate the entire institute, it would be very convenient, if existing folders could be easily included into a new volume. I tried to do this, but I did not succeed. Here's a short description of what I tried: Existing folders: on fs7: /storage/7/user_1,user_2 on fs8: /storage/8/user_3,user_4 gluster volume create transport tcp fs7:/storage/7, fs8:/storage/8 I hoped to see on the client: /gluster/user_1 /gluster/user_2 /gluster/user_3 /gluster/user_4 The creation was successful, the volume could be started and mounted. On the client, however, I could only find (via "ls /gluster") the directories user_1 and user_2. But when I tried "cd /gluster/user_3", it succeeded! Now "ls /gluster" showed me user_1, user_2 and user_3. Unfortunately, user_3's subdirectories and files were still invisible, but with the above mentioned trick, I could make them visible. This is however not an option, as there are too much users and too complicated file structures to do this manually. It anyhow seems like Voodoo to me. Is it possible to include all of the existing directories in the new glusterFS volume? If yes: how? Thank you in advance for your efforts, Regards, Daniel ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Who's using Fedora in production on Glusterfs storage servers?
Hi James, I'm using 3.1.1 on six bricks in dist+replicate all running F14+BTRFS, the clients are on fedora12/13/14. I build the RPMs from source on a F14 machine. The cluster is running entirely on GbE (with some 10Gb lines going in shortly), no RDMA/infiniband so I can't help there. It's gone through a series of looped benchmarks for a while now (from 3.1.0 through a few qa releases) and have so far pushed/pulled over 110TB through it - I'm happy in the stability but not /entirely/ sure of the performance just yet, just started up more testing under 3.1.1. But back to your main question there really isn't enough difference between the near-term releases of Fedora for it to make a huge difference either way. I do think you're better off using the latest Fedora release than an older one that will be end of life soon (f12 tomorrow). Being able to patch/maintain your system is more important than an, often very arbitrary, vendor support list which is usually just an outcome of what people have had time to look into, rather than any measured reason a newer OS isn't supported. Besides the only thing you ever have to /really/ care about is the kernel and glibc major versions, so if it compiles you're pretty much ok (ldd it, that's all it needs). On 12/02/2010 01:45 AM, Burnash, James wrote: How many people on the list are using Fedora 12 (or 13) in production for Glusterfs storage servers? I know that Gluster Platform uses Fedora 12 as its OS - I was thinking of building my new glusterfs storage servers using Fedora, and was wondering whether Fedora 13 was tested by Gluster for v 3.1.1 and what other people's experiences were. One of the reasons for my interest was so that I could use ext4 as the backend file store, instead of ext3. Thanks, James Burnash, Unix Engineering ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Who's using Fedora in production on Glusterfs storage servers?
Hi James, I'm using 3.1.1 on six bricks in dist+replicate all running F14+BTRFS, the clients are on fedora12/13/14. I build the RPMs from source on a F14 machine. The cluster is running entirely on GbE (with some 10Gb lines going in shortly), no RDMA/infiniband so I can't help there. It's gone through a series of looped benchmarks for a while now (from 3.1.0 through a few qa releases) and have so far pushed/pulled over 110TB through it - I'm happy in the stability but not /entirely/ sure of the performance just yet, just started up more testing under 3.1.1. But back to your main question there really isn't enough difference between the near-term releases of Fedora for it to make a huge difference either way. I do think you're better off using the latest Fedora release than an older one that will be end of life soon (f12 tomorrow). Being able to patch/maintain your system is more important than an, often very arbitrary, vendor support list which is usually just an outcome of what people have had time to look into, rather than any measured reason a newer OS isn't supported. Besides the only thing you ever have to /really/ care about is the kernel and glibc major versions, so if it compiles you're pretty much ok (ldd it, that's all it needs). On 12/02/2010 01:45 AM, Burnash, James wrote: How many people on the list are using Fedora 12 (or 13) in production for Glusterfs storage servers? I know that Gluster Platform uses Fedora 12 as its OS - I was thinking of building my new glusterfs storage servers using Fedora, and was wondering whether Fedora 13 was tested by Gluster for v 3.1.1 and what other people's experiences were. One of the reasons for my interest was so that I could use ext4 as the backend file store, instead of ext3. Thanks, James Burnash, Unix Engineering ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Who's using Fedora in production on Glusterfs storage servers?
I'm using 3.1.1 on six bricks running F14+BTRFS. Been running looped benchmarks for a while and pushed/pulled over 110TB so far. I'm convinced it's stable but not entirely sure of the performance just yet. On 12/02/2010 01:45 AM, Burnash, James wrote: How many people on the list are using Fedora 12 (or 13) in production for Glusterfs storage servers? I know that Gluster Platform uses Fedora 12 as its OS - I was thinking of building my new glusterfs storage servers using Fedora, and was wondering whether Fedora 13 was tested by Gluster for v 3.1.1 and what other people's experiences were. One of the reasons for my interest was so that I could use ext4 as the backend file store, instead of ext3. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Storage Platform 3.1.1?
Sorry to bother you. Is there a planned release date for Storage Platform 3.1.1? Thanks Christian ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Anormal Gluster shutdown
Samuel - I can't reproduce this issue locally, can you send me operating system and hardware details for both the Gluster servers and the client? Thanks, Craig --> Craig Carl Senior Systems Engineer Gluster On 12/02/2010 05:59 AM, Samuel Hassine wrote: Hi all, GlusterFS partition automatically shutdown when umounting a binded mount point with "-f" option (without it works). How to reproduce: mounted Gluster partition on /gluster (any config): df: localhost:/gluster4.5T 100G 4.4T 3% /gluster mount: localhost:/gluster on /gluster type fuse.glusterfs (rw,allow_other,default_permissions,max_read=131072) commands: mkdir /test mount -n --bind /gluster /test ls /test (verify you have the Gluster) and: umount -f /test ===> df: `/gluster': Transport endpoint is not connected [2010-12-02 14:48:56.38309] I [fuse-bridge.c:3138:fuse_thread_proc] fuse: unmounting /gluster [2010-12-02 14:48:56.38364] I [glusterfsd.c:672:cleanup_and_exit] glusterfsd: shutting down Before 3.1.x I did not have this bug. Regards. Sam ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
