[Gluster-users] 'Transport endpoint not connected'
This should be a pretty easy issue to reproduce, at least it seems to happen to me very often. (gluster-3.2.5) After storage backend(s) have been rebooted, the client mounts are often broken until you unmount and remount. Example from this morning: I had rebooted storage servers to upgrade them to ubuntu 12.04. Now at the client side: $ ls /gluster/scratch ls: cannot access /gluster/scratch: Transport endpoint is not connected $ ls /gluster/scratch3 dbbuild DBS $ sudo umount /gluster/scratch $ sudo mount /gluster/scratch $ ls /gluster/scratch dbbuild $ Note that /gluster/scratch is a distributed volume (spread across servers 'storage2' and 'storage3'), whereas /gluster/scratch3 is a single brick (server 'storage3' only). So *some* of the mounts do seem to automatically reconnect - not all are affected. But in future, I think it would be good if the FUSE client could automatically attempt to reconnect under whatever circumstance causes 'Transport endpoint is not connected'; clearly it *can* reconnect if forced. Thanks, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 'Transport endpoint not connected'
On 05/04/2012 01:05 PM, Brian Candler wrote: This should be a pretty easy issue to reproduce, at least it seems to happen to me very often. (gluster-3.2.5) After storage backend(s) have been rebooted, the client mounts are often broken until you unmount and remount. Example from this morning: I had rebooted storage servers to upgrade them to ubuntu 12.04. Now at the client side: $ ls /gluster/scratch ls: cannot access /gluster/scratch: Transport endpoint is not connected $ ls /gluster/scratch3 dbbuild DBS $ sudo umount /gluster/scratch $ sudo mount /gluster/scratch $ ls /gluster/scratch dbbuild $ Note that /gluster/scratch is a distributed volume (spread across servers 'storage2' and 'storage3'), whereas /gluster/scratch3 is a single brick (server 'storage3' only). So *some* of the mounts do seem to automatically reconnect - not all are affected. But in future, I think it would be good if the FUSE client could automatically attempt to reconnect under whatever circumstance causes 'Transport endpoint is not connected'; clearly it *can* reconnect if forced. Are you sure the clients are not automatically remounted within 10 seconds of servers coming up? This was working fine from the time we had networking code written. Internally, there is a timer thread which makes sure we automatically reconnect after 10seconds. Please see if you can repeat the operations 2-3 times before doing a umount/mount, it should have gotten reconnected. If not, please file a bug report with the glusterfs logs (of the client process). Regards, Amar ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] 'Transport endpoint not connected'
On Fri, May 04, 2012 at 01:27:35PM +0530, Amar Tumballi wrote: Are you sure the clients are not automatically remounted within 10 seconds of servers coming up? This was working fine from the time we had networking code written. Internally, there is a timer thread which makes sure we automatically reconnect after 10seconds. Please see if you can repeat the operations 2-3 times before doing a umount/mount, it should have gotten reconnected. OK, I'll do that next time. Thanks. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] FD stay open
Amar, Thx for answering us. Is there any other solutions with 3.2.X to avoid this issue except restarting gluster daemon ? And also, when 3.3.x will be in a stable version ? Thx Anthony Message: 1 Date: Mon, 30 Apr 2012 21:44:50 +0530 From: Amar Tumballi ama...@redhat.com Subject: Re: [Gluster-users] Gluster-users Digest, Vol 48, Issue 43 To: Gerald Brandt g...@majentis.com Cc: gluster-users@gluster.org, anthony garnier sokar6...@hotmail.com Message-ID: 4f9eba7a.5080...@redhat.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed You are having the exact same problem I am. So far, no response from anyone at Gluster/RedHat as to what is happening or if this is a known issue. Hi Gerald/Anthony, This issue is not easy to handle with 3.2.x version of gluster's NFS server. This issue is being addressed with 3.3.x branch (ie current master branch). Please try 3.3.0beta3+ or qa36+ for testing the behavior. This happens because NFS process works on FH (file handles), and for that we needed to keep a fd-ref till NFS client has reference to filehandle. With 3.3.0, we changed some of the internal way how we handle NFS FHs, so this problem should not happen in 3.3.0 release. Regards, Amar ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Increase disk space
Let's assume a two node replicated cluster with 12 disks each. Every disks is a standalone FS with no raid. Gluster is replicating 12 bricks (one for each disks). How can I increase disk space by replacing disks in every brick? Can I safely remove one disk per time, replace it with the bigger one, resize the filesystem and then repeat the same operation on the other server? What happens when the first server has a FS bigger then the remote one? ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] How to set backup volume server in mount options
On 05/03/12 18:35, Toby Corkindale wrote: Hi, I saw in the 3.3 changelog that now it is possible to set a secondary server to retrieve the volume information from, when mounting a volume via the native client. My brick servers are all on the same subnet. So I actually have a VIP controlled by VRRP (keepalived) that I use as the volfile server. -- Mr. Flibble King of the Potato People ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Weird folder in my home directory
These look like files from a dbench run. Had you run that? Avati On May 4, 2012 12:28 PM, Mateusz mat.jonc...@gmail.com wrote: Hello, Some time ago I saw a folder called clients in my home directory. I didn't remember creating it manually. It appears to contain some example data for gluster: mateusz@mateusz-laptop:~/**clients$ find | sort . ./client0 ./client0/~dmtmp ./client0/~dmtmp/ACCESS ./client0/~dmtmp/ACCESS/**FASTENER.MDB ./client0/~dmtmp/COREL ./client0/~dmtmp/COREL/**ARTISTIC.ACL ./client0/~dmtmp/COREL/**CDRBARS.CFG ./client0/~dmtmp/COREL/**CDRROLS.CFG ./client0/~dmtmp/COREL/**CORELDRW.CDT [...] ./client7/~dmtmp/WORDPRO/**LWPSAV0.TMP ./client7/~dmtmp/WORDPRO/**NEWS1_1.LWP ./client7/~dmtmp/WORDPRO/**RESULTS.XLS ./client7/~dmtmp/WORD/TIPS.DOC ./client7/filler.000 ./client7/filler.001 ./client7/filler.002 ./client7/filler.003 ./client7/filler.004 The files contain all nulls. I was not able to find in the web any archive that contains them. All I found were some documentation articles and bug reports related to gluster. I am about to do a clean reinstall of my system anyway, but before I would like to investigate this further. I have not installed glusterfs or any files related to it. If and how could I have created these files inadvertently? Greetings, Mateusz __**_ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/**mailman/listinfo/gluster-usershttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] Gluster client can't connect to Gluster volume
Hi, All: I've built a Gluster-based storage cluster on a pair of CentOS 5.7 (i386) VM's. The nodes are using Gluster 3.2.6 (from source) and the host is using Gluster 3.0.0 (from the Mageia package repositories): [eric@node1 ~]$ sudo /usr/local/sbin/gluster --version glusterfs 3.2.6 built on May 3 2012 15:53:02 [eric@localhost ~]$ rpm -qa | grep glusterfs glusterfs-common-3.0.0-2.mga1 glusterfs-client-3.0.0-2.mga1 glusterfs-server-3.0.0-2.mga1 libglusterfs0-3.0.0-2.mga1 None of the systems (i.e., neither the two storage nodes nor the client) can connect to Port 6996 of the cluster (node1.example.com node2.example.com) but the two storage nodes can mount the shared volume using the Gluster helper and/or NFS: [eric@node1 ~]$ sudo /sbin/lsmod | grep fuse [eric@node1 ~]$ sudo /sbin/modprobe fuse [eric@node1 ~]$ sudo /sbin/lsmod | grep fuse fuse 49237 0 [eric@node1 ~]$ sudo mount -t glusterfs node1:/mirror-1 /mnt [eric@node1 ~]$ sudo grep gluster /etc/mtab glusterfs#node1:/mirror-1 /mnt fuse rw,allow_other,default_permissions,max_read=131072 0 0 ...but the host system is only able to connect using NFS: [eric@localhost ~]$ sudo glusterfs --debug -f /tmp/glusterfs.vol /mnt [2012-05-04 19:09:09] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file /tmp/glusterfs.vol Version : glusterfs 3.0.0 built on Apr 10 2011 19:12:54 git: 2.0.1-886-g8379edd Starting Time: 2012-05-04 19:09:09 Command line : glusterfs --debug -f /tmp/glusterfs.vol /mnt PID : 30159 System name : Linux Nodename : localhost.localdomain Kernel Release : 2.6.38.8-desktop586-10.mga Hardware Identifier: i686 Given volfile: +--+ 1: volume mirror-1 2: type protocol/client 3: option transport-type tcp 4: option remote-host node1.example.com 5: option remote-subvolume mirror-1 6: end-volume +--+ [2012-05-04 19:09:09] D [glusterfsd.c:1335:main] glusterfs: running in pid 30159 [2012-05-04 19:09:09] D [client-protocol.c:6581:init] mirror-1: defaulting frame-timeout to 30mins [2012-05-04 19:09:09] D [client-protocol.c:6592:init] mirror-1: defaulting ping-timeout to 42 [2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so [2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] N [glusterfsd.c:1361:main] glusterfs: Successfully started [2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: connection to failed (Connection refused) [2012-05-04 19:09:09] D [fuse-bridge.c:3079:fuse_thread_proc] fuse: pthread_cond_timedout returned non zero value ret: 0 errno: 0 [2012-05-04 19:09:09] N [fuse-bridge.c:2931:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16 [2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: connection to failed (Connection refused) I've read through the Troubleshooting section of the Gluster Administration Guide and the Gluster User Guide but can't seem to resolve the problem. (See my post on the Mageia Forum for all the troubleshooting details: https://forums.mageia.org/en/viewtopic.php?f=7t=2358p=17517) What might be causing this? TIA, Eric Pretorious Truckee, CA https://forums.mageia.org/en/viewtopic.php?f=7t=2358p=17517 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster client can't connect to Gluster volume
Hi Do you have selinux enable? Envoyé de mon iPhone Le 2012-05-05 à 00:27, Eric epretori...@yahoo.com a écrit : Hi, All: I've built a Gluster-based storage cluster on a pair of CentOS 5.7 (i386) VM's. The nodes are using Gluster 3.2.6 (from source) and the host is using Gluster 3.0.0 (from the Mageia package repositories): [eric@node1 ~]$ sudo /usr/local/sbin/gluster --version glusterfs 3.2.6 built on May 3 2012 15:53:02 [eric@localhost ~]$ rpm -qa | grep glusterfs glusterfs-common-3.0.0-2.mga1 glusterfs-client-3.0.0-2.mga1 glusterfs-server-3.0.0-2.mga1 libglusterfs0-3.0.0-2.mga1 None of the systems (i.e., neither the two storage nodes nor the client) can connect to Port 6996 of the cluster (node1.example.com node2.example.com) but the two storage nodes can mount the shared volume using the Gluster helper and/or NFS: [eric@node1 ~]$ sudo /sbin/lsmod | grep fuse [eric@node1 ~]$ sudo /sbin/modprobe fuse [eric@node1 ~]$ sudo /sbin/lsmod | grep fuse fuse 49237 0 [eric@node1 ~]$ sudo mount -t glusterfs node1:/mirror-1 /mnt [eric@node1 ~]$ sudo grep gluster /etc/mtab glusterfs#node1:/mirror-1 /mnt fuse rw,allow_other,default_permissions,max_read=131072 0 0 ...but the host system is only able to connect using NFS: [eric@localhost ~]$ sudo glusterfs --debug -f /tmp/glusterfs.vol /mnt [2012-05-04 19:09:09] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file /tmp/glusterfs.vol Version : glusterfs 3.0.0 built on Apr 10 2011 19:12:54 git: 2.0.1-886-g8379edd Starting Time: 2012-05-04 19:09:09 Command line : glusterfs --debug -f /tmp/glusterfs.vol /mnt PID : 30159 System name : Linux Nodename : localhost.localdomain Kernel Release : 2.6.38.8-desktop586-10.mga Hardware Identifier: i686 Given volfile: +--+ 1: volume mirror-1 2: type protocol/client 3: option transport-type tcp 4: option remote-host node1.example.com 5: option remote-subvolume mirror-1 6: end-volume +--+ [2012-05-04 19:09:09] D [glusterfsd.c:1335:main] glusterfs: running in pid 30159 [2012-05-04 19:09:09] D [client-protocol.c:6581:init] mirror-1: defaulting frame-timeout to 30mins [2012-05-04 19:09:09] D [client-protocol.c:6592:init] mirror-1: defaulting ping-timeout to 42 [2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so [2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2012-05-04 19:09:09] N [glusterfsd.c:1361:main] glusterfs: Successfully started [2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: connection to failed (Connection refused) [2012-05-04 19:09:09] D [fuse-bridge.c:3079:fuse_thread_proc] fuse: pthread_cond_timedout returned non zero value ret: 0 errno: 0 [2012-05-04 19:09:09] N [fuse-bridge.c:2931:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16 [2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: connection to failed (Connection refused) I've read through the Troubleshooting section of the Gluster Administration Guide and the Gluster User Guide but can't seem to resolve the problem. (See my post on the Mageia Forum for all the troubleshooting details: https://forums.mageia.org/en/viewtopic.php?f=7t=2358p=17517) What might be causing this? TIA, Eric Pretorious Truckee, CA https://forums.mageia.org/en/viewtopic.php?f=7t=2358p=17517 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
