Re: [Gluster-users] How to set iptables in glusterfs 3.2.7 ?

[David Coulson]

On 4/10/13 8:28 AM, Jian Lee wrote:


# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Thu Apr 11 00:09:23 2013
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [21:1996]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
Start by removing the line above. That makes all of your gluster rules 
below useless.

-A INPUT -p tcp -m state --state NEW -m tcp --dport 24007:24047 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 1000:1100 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 111 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 111 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 38465:38485 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Thu Apr 11 00:09:23 2013



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] NFS timeout w/ large writes

[David Coulson]

We have a Gluster 3.2.5 environment using NFS mounts, which in general 
is stable. However, we've identified an issue where the NFS server goes 
out to lunch when we do a large (>200mb) write to one of the mounts. 
Unfortunately there is next to nothing in the nfs.log file, other than 
it complaining a brick didn't respond in the timeout interval. The NFS 
server gets to the point where the only way to recover is to reboot the 
box (our gluster nodes mount the volumes using gluster NFS over loopback).


This is the config of the volume which failed this morning - Not sure if 
it is a tuning issue, or a bug. If nothing else, is there a way to 
improve the debugging of the gluster nfs daemon?


[root@dresproddns02 glusterfs]# gluster volume info svn
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: rhesproddns01:/gluster/svn
Brick2: rhesproddns02:/gluster/svn
Brick3: dresproddns01:/gluster/svn
Brick4: dresproddns02:/gluster/svn
Options Reconfigured:
nfs.rpc-auth-allow: 127.0.0.1
performance.client-io-threads: 1
performance.flush-behind: on
network.ping-timeout: 5
performance.stat-prefetch: on
nfs.disable: off
nfs.register-with-portmap: on
auth.allow: 10.250.53.*,10.252.248.*,169.254.*,127.0.0.1
performance.cache-size: 256Mb
performance.write-behind-window-size: 128Mb
performance.io-cache: on
performance.io-thread-count: 64
performance.quick-read: on



___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Client hangs on boot

[David Coulson]

Try making it:

mseas-data:/gdata   /gdata   glusterfs 
defaults,_netdev0 0


Otherwise it'll try to mount too early in the startup sequence.

David

On 12/3/12 8:21 PM, Pat Haley wrote:


Hi,

We have a compute cluster running CentOS 6.2 (installed via
Rocks 6.0) which mounts a gluster filesystem (gluster 3.3.1).
To mount the gluster filesystem we included the following
line in /etc/fstab

mseas-data:/gdata   /gdata   glusterfs defaults
0 0


The issue we're seeing now is that if we reboot the client with
this line present in /etc/fstab, the boot process hangs at
installing local filesystem.  If we comment this line out
of fstab we can boot successfully.  We can then uncomment
this line and mount the gluster filesystem.  Further, if we
maintain 2 other copyies of /etc/fstab (say /etc/fstab_boot
and /etc/fstab_gluster where neither fstab nor fstab_boot
have the gluster line, but fstab_gluster does) then adding
the following lines to /etc/rc.d/rc.d.local allows us to
boot and mount gluster at boot

/bin/cp -f /etc/fstab_gluster /etc/fstab
mount -a
/bin/cp -f /etc/fstab_boot /etc/fstab

This seem like an awful kludge.  Can someone suggest a
cleaner (and presumably more robust) solution?

Thanks.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:  pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Cannot mount gluster volume

[David Coulson]

Didn't you have some interfaces which had IPs assigned, but were not 
connected to anything that was causing issues with your bricks not 
communicating correctly? Last week, or week before.


Maybe that was some other person from MIT?

On 12/1/12 10:57 AM, Patrick Haley wrote:

Hi,

I am sorry, I don't understand the question "Did your unused interfaces
come back online?"

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email: 
pha...@mit.edu
Center for Ocean Engineering Phone:(617) 253-6824
Dept. of Mechanical Engineering Fax:(617) 253-8125
MIT, Room 5-213  
http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301


________
From: David Coulson [da...@davidcoulson.net]
Sent: Friday, November 30, 2012 8:42 PM
To: Patrick Haley
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Cannot mount gluster volume

Did your unused interfaces come back online?

Sent from my iPad

On Nov 30, 2012, at 8:19 PM, Pat Haley  wrote:


Hi,

I have some additional information.  I have just installed gluster on
a second client and tried to mount the same volume.  The first client
appeared to mount the gluster volume but many directories appear as
broken lines and many files absent.  The second client will not
mount at all, giving the following error message:

ERROR: Mount point does not exist.
Usage:  mount.glusterfs : -o  

Options:
man 8 mount.glusterfs

I don't know if this points the was to the problem or to debugging
the problem?

Thanks for any help you can give

Pat



Hi,
We recently installed glusterfs 3.3.1.  We have a 3 brick gluster system running
that was being successfully mounted earlier.  Yesterday we experienced a
power outage and now after rebooting our systems, we are unable to mount
this gluster file system.  On the gluster client, a df -h command shows 41TB
out of 55TB, while an ls command shows broken links for directories and
missing files.
 From the gluster server
# gluster --version
glusterfs 3.3.1 built on Oct 11 2012 22:01:05
# gluster volume info
Volume Name: gdata
Type: Distribute
Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: gluster-0-0:/mseas-data-0-0
Brick2: gluster-0-1:/mseas-data-0-1
Brick3: gluster-data:/data
[root@mseas-data ~]# ps -ef | grep gluster
root  2783 1  0 Nov29 ?00:01:15 /usr/sbin/glusterd -p 
/var/run/glusterd.pid
root  2899 1  0 Nov29 ?00:00:00 /usr/sbin/glusterfsd -s 
localhost --volfile-id gdata.gluster-data.data -p 
/var/lib/glusterd/vols/gdata/run/gluster-data-data.pid -S 
/tmp/e3eac7ce95e786a3d909b8fc65ed2059.socket --brick-name /data -l 
/var/log/glusterfs/bricks/data.log --xlator-option 
*-posix.glusterd-uuid=22f1102a-08e6-482d-ad23-d8e063cf32ed --brick-port 24009 
--xlator-option gdata-server.listen-port=24009
root  2905 1  0 Nov29 ?00:01:22 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/glusterfs/nfs.log -S /tmp/d5c892de43c28a1ee7481b780245b789.socket
root 12329 12291  0 16:28 pts/100:00:00 grep gluster
[root@nas-0-0 ~]# ps -ef | grep gluster
root  2266 1  0 Nov29 ?00:00:21 /usr/sbin/glusterd -p 
/var/run/glusterd.pid
root  2461 1  0 Nov29 ?00:00:00 /usr/sbin/glusterfsd -s 
localhost --volfile-id gdata.gluster-0-0.mseas-data-0-0 -p 
/var/lib/glusterd/vols/gdata/run/gluster-0-0-mseas-data-0-0.pid -S 
/tmp/f4391baa4f7cce7823362c6e0ad3d6de.socket --brick-name /mseas-data-0-0 -l 
/var/log/glusterfs/bricks/mseas-data-0-0.log --xlator-option 
*-posix.glusterd-uuid=3619440a-4ca3-4151-b62e-d4d6bf2e0c03 --brick-port 24009 
--xlator-option gdata-server.listen-port=24009
root  2468 1  0 Nov29 ?00:00:22 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/glusterfs/nfs.log -S /tmp/8f87e178e9707e4694ee7a2543c66db9.socket
root 10007  9972  0 16:12 pts/200:00:00 grep gluster
[root@nas-0-1 ~]# ps -ef | grep gluster
root  2380 1  0 Nov29 ?00:00:09 /usr/sbin/glusterd -p 
/var/run/glusterd.pid
root  2550 1  0 Nov29 ?00:00:00 /usr/sbin/glusterfsd -s 
localhost --volfile-id gdata.gluster-0-1.mseas-data-0-1 -p 
/var/lib/glusterd/vols/gdata/run/gluster-0-1-mseas-data-0-1.pid -S 
/tmp/a69a2b6a0de4366e53fc1584095eeedf.socket --brick-name /mseas-data-0-1 -l 
/var/log/glusterfs/bricks/mseas-data-0-1.log --xlator-option 
*-posix.glusterd-uuid=393fc4a6-1573-4564-971e-1b1aec434167 --brick-port 24009 
--xlator-option gdata-server.listen-port=24009
root  2556 1  0 Nov29 ?00:00:09 /usr/sbin/glusterfs -s 
localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
/var/log/

Re: [Gluster-users] Cannot mount gluster volume

[David Coulson]

Did your unused interfaces come back online?

Sent from my iPad

On Nov 30, 2012, at 8:19 PM, Pat Haley  wrote:

> 
> Hi,
> 
> I have some additional information.  I have just installed gluster on
> a second client and tried to mount the same volume.  The first client
> appeared to mount the gluster volume but many directories appear as
> broken lines and many files absent.  The second client will not
> mount at all, giving the following error message:
> 
> ERROR: Mount point does not exist.
> Usage:  mount.glusterfs : -o  
> 
> Options:
> man 8 mount.glusterfs
> 
> I don't know if this points the was to the problem or to debugging
> the problem?
> 
> Thanks for any help you can give
> 
> Pat
> 
> 
>> Hi,
>> We recently installed glusterfs 3.3.1.  We have a 3 brick gluster system 
>> running
>> that was being successfully mounted earlier.  Yesterday we experienced a
>> power outage and now after rebooting our systems, we are unable to mount
>> this gluster file system.  On the gluster client, a df -h command shows 41TB
>> out of 55TB, while an ls command shows broken links for directories and
>> missing files.
>> From the gluster server
>> # gluster --version
>> glusterfs 3.3.1 built on Oct 11 2012 22:01:05
>> # gluster volume info
>> Volume Name: gdata
>> Type: Distribute
>> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>> Status: Started
>> Number of Bricks: 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster-0-0:/mseas-data-0-0
>> Brick2: gluster-0-1:/mseas-data-0-1
>> Brick3: gluster-data:/data
>> [root@mseas-data ~]# ps -ef | grep gluster
>> root  2783 1  0 Nov29 ?00:01:15 /usr/sbin/glusterd -p 
>> /var/run/glusterd.pid
>> root  2899 1  0 Nov29 ?00:00:00 /usr/sbin/glusterfsd -s 
>> localhost --volfile-id gdata.gluster-data.data -p 
>> /var/lib/glusterd/vols/gdata/run/gluster-data-data.pid -S 
>> /tmp/e3eac7ce95e786a3d909b8fc65ed2059.socket --brick-name /data -l 
>> /var/log/glusterfs/bricks/data.log --xlator-option 
>> *-posix.glusterd-uuid=22f1102a-08e6-482d-ad23-d8e063cf32ed --brick-port 
>> 24009 --xlator-option gdata-server.listen-port=24009
>> root  2905 1  0 Nov29 ?00:01:22 /usr/sbin/glusterfs -s 
>> localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
>> /var/log/glusterfs/nfs.log -S /tmp/d5c892de43c28a1ee7481b780245b789.socket
>> root 12329 12291  0 16:28 pts/100:00:00 grep gluster
>> [root@nas-0-0 ~]# ps -ef | grep gluster
>> root  2266 1  0 Nov29 ?00:00:21 /usr/sbin/glusterd -p 
>> /var/run/glusterd.pid
>> root  2461 1  0 Nov29 ?00:00:00 /usr/sbin/glusterfsd -s 
>> localhost --volfile-id gdata.gluster-0-0.mseas-data-0-0 -p 
>> /var/lib/glusterd/vols/gdata/run/gluster-0-0-mseas-data-0-0.pid -S 
>> /tmp/f4391baa4f7cce7823362c6e0ad3d6de.socket --brick-name /mseas-data-0-0 -l 
>> /var/log/glusterfs/bricks/mseas-data-0-0.log --xlator-option 
>> *-posix.glusterd-uuid=3619440a-4ca3-4151-b62e-d4d6bf2e0c03 --brick-port 
>> 24009 --xlator-option gdata-server.listen-port=24009
>> root  2468 1  0 Nov29 ?00:00:22 /usr/sbin/glusterfs -s 
>> localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
>> /var/log/glusterfs/nfs.log -S /tmp/8f87e178e9707e4694ee7a2543c66db9.socket
>> root 10007  9972  0 16:12 pts/200:00:00 grep gluster
>> [root@nas-0-1 ~]# ps -ef | grep gluster
>> root  2380 1  0 Nov29 ?00:00:09 /usr/sbin/glusterd -p 
>> /var/run/glusterd.pid
>> root  2550 1  0 Nov29 ?00:00:00 /usr/sbin/glusterfsd -s 
>> localhost --volfile-id gdata.gluster-0-1.mseas-data-0-1 -p 
>> /var/lib/glusterd/vols/gdata/run/gluster-0-1-mseas-data-0-1.pid -S 
>> /tmp/a69a2b6a0de4366e53fc1584095eeedf.socket --brick-name /mseas-data-0-1 -l 
>> /var/log/glusterfs/bricks/mseas-data-0-1.log --xlator-option 
>> *-posix.glusterd-uuid=393fc4a6-1573-4564-971e-1b1aec434167 --brick-port 
>> 24009 --xlator-option gdata-server.listen-port=24009
>> root  2556 1  0 Nov29 ?00:00:09 /usr/sbin/glusterfs -s 
>> localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l 
>> /var/log/glusterfs/nfs.log -S /tmp/6054da6605d9f9d1c1e99252f1d235a6.socket
>> root  6922  6888  0 16:14 pts/300:00:00 grep gluster
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley  Email: 
>> pha...@mit.edu
>> Center for Ocean Engineering Phone:(617) 253-6824
>> Dept. of Mechanical Engineering Fax:(617) 253-8125
>> MIT, Room 5-213  
>> http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA  02139-4301
>> 
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> -- 
>

Re: [Gluster-users] FW: cant mount gluster volume

[David Coulson]

ABLISHED)
> glusterd  22582  root  134u IPv4 664881 
> TCP 18.38.1.23:24007->10.1.255.180:1023 (ESTABLISHED)
> glusterd  22582  root  135u IPv4 664882 
> TCP 18.38.1.23:24007->10.1.255.204:1023 (ESTABLISHED)
> glusterd  22582  root  136u IPv4 664883 
> TCP 18.38.1.23:24007->10.1.255.225:1023 (ESTABLISHED)
> glusterd  22582  root  137u IPv4 664884 
> TCP 18.38.1.23:24007->10.1.255.227:1023 (ESTABLISHED)
> glusterd  22582  root  138u IPv4 664886 
> TCP 18.38.1.23:24007->10.1.255.246:1023 (ESTABLISHED)
> glusterd  22582  root  139u IPv4 664898 
> TCP 18.38.1.23:24007->10.1.255.245:1023 (ESTABLISHED)
> glusterd  22582  root  141u     IPv4 666862 
> TCP 10.0.0.2:24007->10.1.1.10:1023 (ESTABLISHED)
> glusterd  22582  root  142u IPv4 670639 
> TCP 10.0.0.2:24007->10.1.1.11:1023 (ESTABLISHED)
> glusterfs 22603  root7u IPv4 664633 
> TCP 127.0.0.1:1021->127.0.0.1:24007 (ESTABLISHED)
> glusterfs 22609  root7u IPv4 664643 
> TCP 127.0.0.1:1020->127.0.0.1:24007 (ESTABLISHED)
> 
> [root@mseas-data ~]# lsof -n | grep 24009
> glusterfs 22603  root   10u IPv4 664658 
> TCP *:24009 (LISTEN)
> glusterfs 22603  root   13u IPv4 665090 
> TCP 10.0.0.2:24009->10.0.0.2:1022 (ESTABLISHED)
> glusterfs 22603  root   14u IPv4 667011 
> TCP 10.0.0.2:24009->10.1.1.10:1021 (ESTABLISHED)
> glusterfs 22603  root   15u IPv4 670783 
> TCP 10.0.0.2:24009->10.1.1.11:1021 (ESTABLISHED)
> glusterfs 22609  root   18u IPv4 665089 
> TCP 10.0.0.2:1022->10.0.0.2:24009 (ESTABLISHED)
> [root@mseas-data ~]# lsof -n | grep 24010
> [root@mseas-data ~]# lsof -n | grep 24011
> [root@mseas-data ~]#
> 
> 
> 
> 
> 
> 
> From: David Coulson [da...@davidcoulson.net]
> Sent: Wednesday, November 21, 2012 3:20 PM
> To: Steve Postma
> Cc: Eco Willson; gluster-users@gluster.org
> Subject: Re: [Gluster-users] FW: cant mount gluster volume
> 
> I would be concerned about the connections in a SYN_SENT state. Would be
> helpful if this was done with the -n flag so no DNS and we could see the
> real IPs.
> 
> On 11/21/12 2:49 PM, Steve Postma wrote:
>> Eco,
>> they all appear to be using 24007 and 24009, none of them are running on 
>> 24010 or 24011.
>> Steve
>> 
>> [root@nas-0-0<mailto:root@nas-0-0> ~]# lsof | grep 24010
>> [root@nas-0-0<mailto:root@nas-0-0> ~]# lsof | grep 24011
>> [root@nas-0-0<mailto:root@nas-0-0> ~]# lsof | grep 24009
>> glusterfs 3536 root 18u IPv4 143541 0t0 TCP 
>> 10.1.1.10:1022->gluster-data:24009 (ESTABLISHED)
>> [root@nas-0-0<mailto:root@nas-0-0> ~]# lsof | grep 24007
>> glusterd 3515 root 6u IPv4 143469 0t0 TCP nas-0-0:24007->nas-0-0:1022 
>> (ESTABLISHED)
>> glusterd 3515 root 8u IPv4 77801 0t0 TCP *:24007 (LISTEN)
>> glusterd 3515 root 12u IPv4 143805 0t0 TCP 
>> 10.1.1.10:1020->gluster-data:24007 (ESTABLISHED)
>> glusterfs 3536 root 7u IPv4 143468 0t0 TCP nas-0-0:1022->nas-0-0:24007 
>> (ESTABLISHED)
>> glusterfs 3536 root 16u IPv4 399743 0t0 TCP 
>> 10.1.1.10:1023->gluster-0-0:24007 (SYN_SENT)
>> glusterfs 3536 root 17u IPv4 399745 0t0 TCP 
>> 10.1.1.10:1021->gluster-0-1:24007 (SYN_SENT)
>> 
>> 
>> 
>> [root@nas-0-1<mailto:root@nas-0-1> ~]# lsof | grep 24007
>> glusterd 3447 root 6u IPv4 77189 0t0 TCP nas-0-1:24007->nas-0-1:1021 
>> (ESTABLISHED)
>> glusterd 3447 root 8u IPv4 11540 0t0 TCP *:24007 (LISTEN)
>> glusterd 3447 root 10u IPv4 317363 0t0 TCP 10.1.1.11:1022->gluster-0-0:24007 
>> (SYN_SENT)
>> glusterd 3447 root 12u IPv4 77499 0t0 TCP 10.1.1.11:1023->gluster-data:24007 
>> (ESTABLISHED)
>> glusterfs 3468 root 7u IPv4 77188 0t0 TCP nas-0-1:1021->nas-0-1:24007 
>> (ESTABLISHED)
>> glusterfs 3468 root 17u IPv4 317361 0t0 TCP 
>> 10.1.1.11:1019->gluster-0-1:24007 (SYN_SENT)
>> [root@nas-0-1<mailto:root@nas-0-1> ~]# lsof | grep 24009
>> glusterfs 3468 root 18u IPv4 77259 0t0 TCP 
>> 10.1.1.11:1021->gluster-data:24009 (ESTABLISHED)
>> [root@nas-0-1<mailto:root@nas-0-1> ~]# lsof | grep 24010
>> [root@nas-0-1<mailto:root@nas-0-1> ~]# lsof | grep 24011
>>

Re: [Gluster-users] FW: cant mount gluster volume

[David Coulson]

I would be concerned about the connections in a SYN_SENT state. Would be 
helpful if this was done with the -n flag so no DNS and we could see the 
real IPs.


On 11/21/12 2:49 PM, Steve Postma wrote:

Eco,
they all appear to be using 24007 and 24009, none of them are running on 24010 
or 24011.
Steve

[root@nas-0-0 ~]# lsof | grep 24010
[root@nas-0-0 ~]# lsof | grep 24011
[root@nas-0-0 ~]# lsof | grep 24009
glusterfs 3536  root   18u IPv4 143541  0t0TCP 
10.1.1.10:1022->gluster-data:24009 (ESTABLISHED)
[root@nas-0-0 ~]# lsof | grep 24007
glusterd  3515  root6u IPv4 143469  0t0TCP 
nas-0-0:24007->nas-0-0:1022 (ESTABLISHED)
glusterd  3515  root8u IPv4  77801  0t0TCP 
*:24007 (LISTEN)
glusterd  3515  root   12u IPv4 143805  0t0TCP 
10.1.1.10:1020->gluster-data:24007 (ESTABLISHED)
glusterfs 3536  root7u IPv4 143468  0t0TCP 
nas-0-0:1022->nas-0-0:24007 (ESTABLISHED)
glusterfs 3536  root   16u IPv4 399743  0t0TCP 
10.1.1.10:1023->gluster-0-0:24007 (SYN_SENT)
glusterfs 3536  root   17u IPv4 399745  0t0TCP 
10.1.1.10:1021->gluster-0-1:24007 (SYN_SENT)



[root@nas-0-1 ~]# lsof | grep 24007
glusterd  3447  root6u IPv4  77189  0t0TCP 
nas-0-1:24007->nas-0-1:1021 (ESTABLISHED)
glusterd  3447  root8u IPv4  11540  0t0TCP 
*:24007 (LISTEN)
glusterd  3447  root   10u IPv4 317363  0t0TCP 
10.1.1.11:1022->gluster-0-0:24007 (SYN_SENT)
glusterd  3447  root   12u IPv4  77499  0t0TCP 
10.1.1.11:1023->gluster-data:24007 (ESTABLISHED)
glusterfs 3468  root7u IPv4  77188  0t0TCP 
nas-0-1:1021->nas-0-1:24007 (ESTABLISHED)
glusterfs 3468  root   17u IPv4 317361  0t0TCP 
10.1.1.11:1019->gluster-0-1:24007 (SYN_SENT)
[root@nas-0-1 ~]# lsof | grep 24009
glusterfs 3468  root   18u IPv4  77259  0t0TCP 
10.1.1.11:1021->gluster-data:24009 (ESTABLISHED)
[root@nas-0-1 ~]# lsof | grep 24010
[root@nas-0-1 ~]# lsof | grep 24011

glusterfs  4301  root   16u IPv4 586766  TCP 
10.1.1.2:1021->gluster-0-0:24007 (SYN_SENT)
glusterfs  4301  root   17u IPv4 586768  TCP 
10.1.1.2:1020->gluster-0-1:24007 (SYN_SENT)
glusterfs 17526  root8u IPv4 205563  TCP 
mseas-data.mit.edu:1015->mseas-data.mit.edu:24007 (ESTABLISHED)
[root@mseas-data ~]# lsof | grep 24009
glusterfs  4008  root   10u IPv4  77692  
TCP *:24009 (LISTEN)
glusterfs  4008  root   13u IPv4 148473  TCP 
gluster-data:24009->gluster-data:1018 (ESTABLISHED)
glusterfs  4008  root   14u IPv4  82251  TCP 
gluster-data:24009->10.1.1.10:1022 (ESTABLISHED)
glusterfs  4008  root   15u IPv4  82440  TCP 
gluster-data:24009->10.1.1.11:1021 (ESTABLISHED)
glusterfs  4008  root   16u IPv4 205600  TCP 
gluster-data:24009->gluster-data:1023 (ESTABLISHED)
glusterfs  4008  root   17u IPv4 218671  TCP 
10.1.1.2:24009->10.1.1.1:1018 (ESTABLISHED)
glusterfs  4301  root   18u IPv4 148472  TCP 
gluster-data:1018->gluster-data:24009 (ESTABLISHED)
glusterfs 17526  root   12u IPv4 205599  TCP 
gluster-data:1023->gluster-data:24009 (ESTABLISHED)
[root@mseas-data ~]# lsof | grep 24010
[root@mseas-data ~]# lsof | grep 24011
[root@mseas-data ~]#




From: gluster-users-boun...@gluster.org [gluster-users-boun...@gluster.org] on 
behalf of Steve Postma [spos...@ztechnet.com]
Sent: Wednesday, November 21, 2012 10:19 AM
To: Eco Willson; gluster-users@gluster.org
Subject: Re: [Gluster-users] FW: cant mount gluster volume

Your right Eco
I am only able to telnet on port 24007, ports 24009, 24010 and 24011 are all 
connection refused . Iptables is not running on any of the machines


mseas-data 24007, 24009 are open, 24010 and 24011 closed
nas-0-0 24007 open, 24009,24010 and 24011 closed
nas-0-1 24007 open, 24009,24010 and 24011 closed



Steve

From: gluster-users-boun...@gluster.org 
[gluster-users-boun...@gluster.org] on behalf of 
Eco Willson [ewill...@redhat.com]
Sent: Tuesday, November 20, 2012 6:32 PM
To: gluster-users@gluster.org
Subject: Re: [Gluster-users] FW: cant mount gluster volume

Steve,
On 11/20/2012 02:43 PM, Steve Postma wrote:

Hi Eco,
I believe you a

Re: [Gluster-users] Inviting comments on my plans

[David Coulson]

On 11/18/12 7:53 PM, Whit Blauvelt wrote:

Red Hat does not support upgrades between major versions. Thus Cent OS and
Scientific don't either. That's a major part of why I generally run Ubuntu
or Debian instead, except for users who are really wedded to the Red Hat
way.
I work in an Enterprise environment, and in general once a RHEL release 
goes out of support it is time to replace the hardware anyway - We 
usually stick to a 4-5yr max lifecycle for hardware, which fits well 
with the RedHat 7yr support model. Assuming you are running non-RHEL 
software, an upgrade between major versions has to be a complete mess, 
if not impossible.


Back to the original posters query - What's your business requirements? 
I've learned that if you try to build a solution by putting technology 
first it never seems to work well. We don't even know if Gluster is 
appropriate for your needs.


David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Setting up a new gluster volume using a block device

[David Coulson]

Your gluster brick must be a directory, not a block device. The 
filesystem that directory is located on must supported xattr.


David

On 7/19/12 1:16 PM, Sallee, Stephen (Jake) wrote:


I am new to gluster so please be a bit patient with me.

I am trying to setup a gluster volume with the bricks being /dev/sdb1 
(or /dev/sdb, I have tried both) ... etc. however I get an error 
message and the operation fails. I do not have the error message 
(sorry!) the only brick I have available for testing is tied up at the 
moment so I cant rerun the test to get the exact error.  If I can I 
will post it later.


However if I mount the partition I am able to create the volume no 
problem.  I have read the admin guide several times but cannot find 
any reference to using a block device only a mounted device.


Is it even possible, or am I crazy for even trying?

Jake Sallee

Godfather of Bandwidth

System Engineer

University of Mary Hardin-Baylor

900 College St.

Belton TX. 76513

Fone: 254-295-4658

Phax: 254-295-4221

HTTP://WWW.UMHB.EDU



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?

[David Coulson]

Was that introduced by the same person who thought that binding to 
sequential ports down from 1024 was a good idea?


Considering how hard RedHat was pushing Gluster at the Summit a week or 
two ago, it seems like they're making it hard for people to really 
implement it in any capacity other than their Storage Appliance product.


Luckily I don't need locking yet, but I suppose RedHat will be happy 
when I do since I'll need to buy more GFS2 Add-Ons for my environment :-)


David

On 7/13/12 7:49 AM, Rajesh Amaravathi wrote:

Actually, if you want to mount *any* nfs volumes(of Gluster) OR
exports (of kernel-nfs-server), you cannot do it with locking on
a system where a glusterfs(nfs process) is running(since 3.3.0).
However, if its ok to mount without locking, then you should be
able to do it on localhost.

Regards,
Rajesh Amaravathi,
Software Engineer, GlusterFS
RedHat Inc.

- Original Message -----
From: "David Coulson" 
To: "Tomasz Chmielewski" 
Cc: "Rajesh Amaravathi" , "Gluster General Discussion List" 

Sent: Friday, July 13, 2012 3:16:38 PM
Subject: Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable 
or not?


On 7/13/12 5:29 AM, Tomasz Chmielewski wrote:

Killing the option to use NFS mounts on localhost is certainly quite
the opposite to my performance needs!


He was saying you can't run kernel NFS server and gluster NFS server at
the same time, on the same host. There is nothing stopping you from
mounting localhost:/volume on all your boxes. That is exactly how our
3.2.5 and 3.3.0 environments access volumes for the performance reasons
you identified.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] NFS mounts with glusterd on localhost - reliable or not?

[David Coulson]

On 7/13/12 5:29 AM, Tomasz Chmielewski wrote:


Killing the option to use NFS mounts on localhost is certainly quite 
the opposite to my performance needs!




He was saying you can't run kernel NFS server and gluster NFS server at 
the same time, on the same host. There is nothing stopping you from 
mounting localhost:/volume on all your boxes. That is exactly how our 
3.2.5 and 3.3.0 environments access volumes for the performance reasons 
you identified.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Inconsistency of disk free on replica volume.

[David Coulson]

I've a simple 2-way replica volume, however it's capacity utilization is 
really inconsistent. I realize du and df aren't the same thing, but I'm 
confused how the brick and the NFS mount are not showing the same amount 
of capacity available. Underlying filesystem is XFS, and gluster volume 
is mounted using gluster NFS daemon - Note that volume is mounted on 
systems which run the bricks too.


[root@rhesproddns01 ~]# gluster volume info openfire

Volume Name: openfire
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: rhesproddns01:/gluster/openfire
Brick2: rhesproddns02:/gluster/openfire
Options Reconfigured:
nfs.rpc-auth-allow: 127.0.0.1
performance.write-behind-window-size: 128Mb
performance.cache-size: 256Mb
auth.allow: 10.250.53.*,10.252.248.*,169.254.*,127.0.0.1
network.ping-timeout: 5
performance.stat-prefetch: on
nfs.register-with-portmap: on
nfs.disable: off
performance.client-io-threads: 1
performance.io-cache: on
performance.io-thread-count: 64
performance.quick-read: on
[root@rhesproddns01 ~]# df -h /opt/openfire
FilesystemSize  Used Avail Use% Mounted on
localhost:/openfire   960M  881M   80M  92% /opt/openfire

[root@rhesproddns01 ~]# du -sh /opt/openfire
174M/opt/openfire

[root@rhesproddns01 ~]# df -h /gluster/openfire
FilesystemSize  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-gluster_openfire
  960M  756M  205M  79% /gluster/openfire

[root@rhesproddns01 ~]# du -sh /gluster/openfire
256M/gluster/openfire

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] about HA infrastructure for hypervisors

[David Coulson]

No

I saw a patch to have it behave like this, but I can't find it right now.

On 6/28/12 6:54 AM, Tim Bell wrote:

Assuming that we use a 3 copy approach across the hypervisors, does Gluster
favour the local copy on the hypervisor if the data is on
distributed/replicated ?

It would be good to avoid the network hop when the data is on the local
disk.

Tim


-Original Message-
From: gluster-users-boun...@gluster.org [mailto:gluster-users-
boun...@gluster.org] On Behalf Of Fernando Frediani (Qube)
Sent: 28 June 2012 11:43
To: 'Nicolas Sebrecht'; 'Thomas Jackson'
Cc: 'gluster-users'
Subject: Re: [Gluster-users] about HA infrastructure for hypervisors

You should indeed to use the same server running as a storage brick as a
KVM host to maximize hardware and power usage. Only thing I am not sure
is if you can limit the amount of host memory Gluster can eat so most of

it

gets reserved for the Virtual Machines.

Fernando

-Original Message-
From: gluster-users-boun...@gluster.org [mailto:gluster-users-
boun...@gluster.org] On Behalf Of Nicolas Sebrecht
Sent: 28 June 2012 10:31
To: Thomas Jackson
Cc: 'gluster-users'
Subject: [Gluster-users] Re: about HA infrastructure for hypervisors

The 28/06/12, Thomas Jackson wrote:


Why don't you have KVM running on the Gluster bricks as well?

Good point. While abtracting we decided to seperate KVM & Gluster but I
can't remember why.
We'll think about that again.


We have a 4 node cluster (each with 4x 300GB 15k SAS drives in
RAID10), 10 gigabit SFP+ Ethernet (with redundant switching). Each
node participates in a distribute+replicate Gluster namespace and runs
KVM. We found this to be the most efficient (and fastest) way to run the

cluster.

This works well for us, although (due to Gluster using fuse) it isn't
as fast as we would like. Currently waiting for the KVM driver that
has been discussed a few times recently, that should make a huge
difference to the performance for us.

Ok! Thanks.

--
Nicolas Sebrecht
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Switching clients to NFS

[David Coulson]

On 6/22/12 7:08 AM, Marcus Bointon wrote:

Sorry, I should have said, I'm using 3.2.5 on 64-bit ubuntu lucid. I assume 
it's ok to have one client mounted with nfs while the other uses native?

We do this all the time - Works fine.

David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Switching clients to NFS

[David Coulson]

On 6/22/12 6:18 AM, Fernando Frediani (Qube) wrote:

I have seen a few people recently saying they are using NFS instead of the 
Native Gluster client. I would imagine that the Gluster client would always be 
better and faster besides the automatic failover, but it makes me wonder what 
sort of problems their as experiencing with the Gluster client.

I ran FUSE mounts for a couple of months then switched to NFS. In 
general I saw an approx 2x improvement in read performance, and writes 
appeared to be a little quicker. Since my environment is 'mostly reads', 
as NFS utilizes the OS filesystem cache I saw a substantial drop in 
network traffic between Gluster nodes.


I was also never able to get FUSE to mount volumes with an explicit 
SELinux context set. Not sure if it's a bug in FUSE on RHEL6, or just 
something broken with FUSE, but it just ignored the secontext= mount 
parameter. NFS works with this, so I was able to run SELinux in 
enforcing mode while using Gluster.


David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Not real confident in 3.3

[David Coulson]

On 6/17/12 8:21 AM, Sean Fulton wrote:
This was a Linux-HA cluster with a floating IP that the clients would 
mount off of whichever server is active. So I set up a two-node 
replicated cluster, which the floating IP and heartbeat, and the 
client mounted the drive over the floating IP. I'm using the NFS 
server built into gluster. So rpcbind and nfslock are running on the 
server, but not nfs. The client writes to the one server with the 
floating IP, and gluster takes care of keeping the volume in sync 
between the two servers. I thought that was the way to do it. 
It has never been clear to me how well you can fail over a NFS mount 
using a floating IP address, especially with the Gluster NFS server. I 
typically install gluster on the client and have it mount 
localhost:/whatever and all the brick routing is handles locally. Not 
practical with a lot of clients, but it's a simpler configuration in a 
smaller environment.


If it makes you feel better, I'm in the process of restoring a 
production SVN repo because Gluster 3.2.5 ate it after a node reboot 
last night. Not had time to dig through the logs in detail, but it seems 
like a self-heal from a 4-way replica did something wrong.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] AutoFS NFS Problems

[David Coulson]

Does 'localhost' work?

On 6/13/12 3:36 AM, Chris LeBlanc wrote:
What I found guys is that I can't mount using the local gluster peer 
address for some odd reason.


I ended up doing this...

on "sld-web-4"
/srv -rw,fstype=nfs,nfsvers=3,mountproto=tcp,noatime,nodiratime 
 sld-web-3:/srv


on "sld-web-3"
/srv -rw,fstype=nfs,nfsvers=3,mountproto=tcp,noatime,nodiratime 
 sld-web-4:/srv


Any reason why you guys can think of?

On Sunday, June 10, 2012 at 10:36 AM, Mailing Lists wrote:


Sorry, sent to fast :-)

Here is what I use in the fstab :

gluster.server.tld:/volume /mnt nfs defaults,_netdev,mountproto=tcp 0 0

Meanwhile, on Debian, the mount is sometime not done at boot and I 
should use mount /mnt. I not sure that is related to GlusterFs 
because I've sometimes the same issue with other NFS servers.


In an openVz container, the mount is done at each boot !

Best regards.

Michel

- Mail original -
De: "Whit Blauvelt" >

À: "Chris LeBlanc" mailto:ch...@blendedby.us>>
Cc: gluster-users@gluster.org 
Envoyé: Dimanche 10 Juin 2012 14:52:28
Objet: Re: [Gluster-users] AutoFS NFS Problems

Don't know if "proto" is a synonym. But in my working auto.fs mounts 
I have

"mountproto=tcp" not proto.

Whit

On Sun, Jun 10, 2012 at 04:33:50AM -0500, Chris LeBlanc wrote:

Hi friends,

I'd like to use autofs but I had mixed luck. It seems as thought it 
mounts
itself instead of the gluster volume which leads me to believe it's 
not using

the correct NFS protocol.

auto.master
/- /etc/auto.srv

auto.srv
/srv -fstype=nfs,nfsvers=3,rw,proto=tcp sld-web-3:/srv

I've tried all sorts of variations for options in auto.srv, tried
adding MOUNT_NFS_DEFAULT_PROTOCOL=3 to auto.master (not sure if 
that's where I
should be putting that), and even saw somewhere to add vers=3 to the 
right of

the auto.master direct mount .

I'm using cluster 3.2.5 and Ubuntu Precise.

Many thanks for your help.

--
Chris LeBlanc



___
Gluster-users mailing list
Gluster-users@gluster.org 
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org 
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org 
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster NFS performance issue upgrading from 3.2.5 to 3.2.6/3.3.0

[David Coulson]

For what it is worth, I had weird performance issues when I moved from 
3.2.5 to 3.3.0 - I saw increased CPU utilization, as well as drastically 
increased network utilization between the nodes with the same workload. 
I could never really quantify the difference, other than I noticed my 
systems mounting the volumes via NFS had more problems when a brick went 
offline in an unclean way (e.g. network disappeared). I have six nodes 
which run 4-way or 2-way replicas and mount the volumes locally, so it's 
a pretty self-contained configuration.


Today I rolled back to 3.2.5 and ran for ~8hours. My traffic dropped 
back to what it looked like previously, and my load dropped to next to 
nothing. I just upgraded to 3.2.7, and at least in the last couple of 
hours network utilization is about the same as 3.3.0. CPU usage is 
actually worse with 3.2.7 than 3.3.0.


Does anyone have a 'good' test of Gluster performance? Most of my 
operations take <10s, so when they take 5.4s avg with 3.3.0 after 100 
runs, and avg 4.6s with 3.2.5 it's hard to tell if it's a meaningful 15% 
or bad statistics. I'd like to understand what is different between 
3.2.5 and 3.3.0 or 3.2.7, but really need a good way to quantify it.


On 6/11/12 10:15 AM, Simon Detheridge wrote:

Hi,

I have a situation where I'm mounting a gluster volume on several web servers 
via NFS. The web servers run Rails applications off the gluster NFS mounts. The 
whole thing is running on EC2.

On 3.2.5, starting a Rails application on the web server was sluggish but 
acceptable. However, after upgrading to 3.2.6 the length of time taken to start 
a Rails application has increased by over 10 times, to something that's not 
really suitable for a production environment. The situation still occurs with 
3.3.0 as well.

If I attach strace to the rails process as it's starting up, I see that it's 
looking for a very large number of nonexistent files. I think this is something 
that Rails does that can't be helped - it checks to see if a file is there for 
many things, and does something accordingly if it does.

Has something changed that could negatively affect the length of time it takes 
to stat a nonexistent file over a NFS mount to a gluster volume, between 3.2.5 
and 3.2.6? Is there any way I can get the old behaviour without downgrading?

-- I don't currently have proof that it's the nonexistent files that's causing 
the problem, but it seems highly likely as performance for the other tasks that 
the servers carry out appears unaffected.

Sorry this is slightly vague. I can run some more tests/benchmarks to try and 
figure out what's going on in more detail, but thought I would ask here first 
in case this is related to a known issue.

Thanks,
Simon




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Existing Data and self mounts ?

[David Coulson]

On 6/4/12 4:05 AM, Jacques du Rand wrote:

HI Guys
This all applies to Gluster3.3

I love gluster but I'm having  some difficulties understanding some 
things.


1.Replication(with existing data):
Two servers in simple single brick replication. ie 1 volume (testvol)
-server1:/data/ && server2:/data/
-server1 has a few millions files in the /data dir
-server2 has a no files in /data dir

So after i created the testvol and started the volume
QUESTION (1): Do i  need to mount  each volume on the servers like so 
? If yes why ?

---> on server1: mount -t gluster 127.0.0.1:/testvol /mnt/gfstest
---> on server2: mount -t gluster 127.0.0.1:/testvol /mnt/gfstest
Only if you want to access the files within the volume on the two 
servers which have the bricks on them.


CLIENT:
Then I mount the client:
mount server-1-ip:/testvol /mnt/gfstest

Question(2) :
I only see files from server2 ???

Probably hit and miss what you see, since your bricks are not consistent.


Question (3)
Whenever I'm writing/updating/working with the files on the SERVER i 
should ALWAYS do it via the (local mount )/mnt/gfstest. I should never 
work with files directly in the bricks /data ??
Correct - Gluster can't keep track of writes if you don't do it through 
the glusterfs mount point.


Question (4.1)
-Whats the best-practise to sync existing "data"  ?
You will need to force a manual self-heal and see if that copies all the 
data over to the other brick.


find /mnt/gfstest -noleaf -print0 | xargs --null stat>/dev/null




Question (4.2)
-Is it safe to create a brick in a directory that already has files in 
it ?


As long as you force a self-heal on it before you use it.









___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] why is glusterfs sometimes listening on port 80?

[David Coulson]

Is there a way to change this behavior? It's particularly frustrating 
having Gluster mount a filesystem before the service starts up, only to 
find it steps on the top end of <1024 ports often - IMAPS and POP3S are 
typical victims at 993 and 995.


Why does not not use ports within the 
/proc/sys/net/ipv4/ip_local_port_range?


On 6/4/12 5:46 AM, Raghavendra Bhat wrote:

Glusterfs client starts binding from port number 1023 and if any port is not 
available, then tries to bind to the lower port. (i.e. suppose 1023 is not 
available because some other process is already using it, then it decrements 
the port to 1022 and tries to bind.) So in this case it has tried all the ports 
starting 1023 and found that those ports are not free (used by other processes) 
and found that port 80 is free and just did bind on it.


Regards,
Raghavendra Bhat



- Original Message -
From: "Tomasz Chmielewski"
To: "Gluster General Discussion List"
Sent: Monday, June 4, 2012 7:02:55 AM
Subject: [Gluster-users] why is glusterfs sometimes listening on port 80?

I've noticed that I'm sometimes not able to start a webserver on machines 
running glusterfs. It turned out that glusterfs, when mounting, is sometimes 
starting to listen on port 80:

root@ca11:~# netstat -tpna|grep :80
tcp0  0 192.168.20.31:80192.168.20.34:24010 ESTABLISHED 
13843/glusterfs


Why does glusterfs do it?

Killing glusterfs and mounting again usually fixes the problem.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] File IO issues during brick unreachable in replica config

[David Coulson]

I've a volume in a 4 way replica configuration running 3.3.0 - Two 
bricks are in one datacenter, two are in the other. We had some sort of 
connectivity issue between the two facilities this morning, and 
applications utilizing gluster mounts (via NFS; in this case only-read 
work load) experienced IO timeouts.


I've a 5s network timeout on the volume, and a 20s timeout on the 
application - I'd expect even if it went through 3 bricks before it 
found a good one for a read, it would take 10s.


What is the expectation for a read which occurs when a brick is in the 
process of failing? Should the IO fail, or should it be re-routed to an 
available brick? I don't see anything specific in nfs.log indicating a 
particular read failed, just that the bricks went up/down.


Info is below - Let me know if there are other logs I need to look at.

[root@dresproddns02 glusterfs]# gluster volume info svn

Volume Name: svn
Type: Replicate
Volume ID: fabe320d-5ef2-4f35-9720-eab617e13dde
Status: Started
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: rhesproddns01:/gluster/svn
Brick2: rhesproddns02:/gluster/svn
Brick3: dresproddns01:/gluster/svn
Brick4: dresproddns02:/gluster/svn
Options Reconfigured:
performance.write-behind-window-size: 128Mb
performance.cache-size: 256Mb
auth.allow: 10.250.53.*,10.252.248.*,169.254.*,127.0.0.1
nfs.register-with-portmap: on
nfs.disable: off
performance.stat-prefetch: 1
network.ping-timeout: 5
performance.flush-behind: on
performance.client-io-threads: 1
nfs.rpc-auth-allow: 127.0.0.1

nfs.log output is here:
http://pastebin.com/CNmP4s32
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] broken after IPs changed

[David Coulson]

You probably want to blow away your brick filesystem and start clean - 
There will be xattr information that is confusing Gluster.


Best practice is to use DNS to support peers, rather than IP addresses.

On 6/1/12 12:27 AM, Костырев Александр Алексеевич wrote:


Hello!

I fired up gluster v.3.2.5 with this:

gluster peer probe 10.0.1.131

gluster volume create vms replica 2 transport tcp 10.0.1.130:/mnt/ld0 
10.0.1.131:/mnt/ld3 10.0.1.130:/mnt/ld1 10.0.1.131:/mnt/ld4 
10.0.1.130:/mnt/ld2 10.0.1.131:/mnt/ld5


gluster volume start vms

mkdir /mnt/gluster

added in /etc/fstab

127.0.0.1:/vms /mnt/gluster glusterfs defaults,_netdev 0 0

mount /mnt/gluster

everything was awesome

than for some reason I had to change IPs of my servers

from

10.0.1.130 -> 10.0.1.50

10.0.1.131 -> 10.0.1.51

I’ve decided to

stop my volume,

delete it,

stop glusterd,

erase gluster software (and all configs) with yum (Centos 6.2 x86_64)

and then recreate that volume again with:

gluster peer probe 10.0.1.51

gluster volume create vms replica 2 transport tcp 10.0.1.50:/mnt/ld0 
10.0.1.51:/mnt/ld3 10.0.1.50:/mnt/ld1 10.0.1.51:/mnt/ld4 
10.0.1.50:/mnt/ld2 10.0.1.51:/mnt/ld5


but  it said:

Operation failed

I googled for awhile and found out that new 3.3.0 has arrived, so I 
updated to it and start whole thing over again


Now It says:

/mnt/ld0 or a prefix of it is already part of a volume

Any help ?

Thanks in advance



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] A very special announcement from Gluster.org

[David Coulson]

On 6/1/12 8:14 AM, Kaleb S. KEITHLEY wrote:



If by "'official' gluster packages" you mean the glusterfs rpms in the 
fedora/epel yum repo, and your 3.2.5 was built from source or using 
rpms from somewhere else, including e.g. gluster.org, then your 
experience is not unexpected.

I used the el6 rpms from gluster.org


The rpms in the fedora/epel yum repo follow the fedora file system 
layout standard and as you note, you needed to move your .vol files to 
the correct location in this situation.
Not sure that really matters. The point is the 3.3.0 RPMs try to copy 
/etc/glusterd to /var/lib/glusterd during the upgrade process, and that 
doesn't work if /etc and /var/lib are not on the same filesystem. It 
wouldn't be so bad if it didn't even try to move the in the first place, 
but it does and it doesn't always work.


Or did I miss something?

David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Increased brick 'activity' when moving 3.2.5->3.3.0

[David Coulson]

For what it is worth, here is a graph showing bandwidth utilization 
between two of my four Gluster nodes following 3.2.5->3.3.0 upgrade. 
There were no other changes to the environment or work load, other than 
the Gluster update.


http://i.imgur.com/VnZPl.png

I'm going to try to reproduce it on another environment this weekend to 
get more solid numbers.


David

On 6/1/12 12:28 PM, David Coulson wrote:

I upgraded my 3.2.5 environment to 3.3.0 this morning. I'm seeing an approx 4x 
increase in network activity since the upgrade. tcpdump indicates a volume 
which is pretty much 100% reads has a lot of tcp activity between the nodes. 
Since it is mounted NFS, I was expecting that for a 'nearly all read' workload 
it would utilize the OS cache (this is what I experienced when we moved from 
Fuse to NFS under 3.2.5).

Is there a volume tweak or a change to NFS I need to make to utilize OS 
caching? We're running on RHEL6. Here is my mount output for this volume.

localhost:/ldirectord on /etc/ldirectord type nfs 
(rw,noatime,nodiratime,tcp,vers=3,nolock,addr=127.0.0.1)

Thanks-
David

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Increased brick 'activity' when moving 3.2.5->3.3.0

[David Coulson]

I upgraded my 3.2.5 environment to 3.3.0 this morning. I'm seeing an approx 4x 
increase in network activity since the upgrade. tcpdump indicates a volume 
which is pretty much 100% reads has a lot of tcp activity between the nodes. 
Since it is mounted NFS, I was expecting that for a 'nearly all read' workload 
it would utilize the OS cache (this is what I experienced when we moved from 
Fuse to NFS under 3.2.5).

Is there a volume tweak or a change to NFS I need to make to utilize OS 
caching? We're running on RHEL6. Here is my mount output for this volume.

localhost:/ldirectord on /etc/ldirectord type nfs 
(rw,noatime,nodiratime,tcp,vers=3,nolock,addr=127.0.0.1)

Thanks-
David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] A very special announcement from Gluster.org

[David Coulson]

I experienced the following going from both 3.2.5 and 3.2.6 (using 
'official' gluster packages) on RHEL6.


[root@rhesproddns02 ~]# rpm -Uvh glusterfs-*3.3.0*
Preparing...### 
[100%]
   1:glusterfs  ### 
[ 33%]
   2:glusterfs-fuse ### 
[ 67%]
   3:glusterfs-server   ### 
[100%]
mv: inter-device move failed: `/etc/glusterd' to `/var/lib/glusterd'; 
unable to remove target: Is a directory
glusterd: symbol lookup error: glusterd: undefined symbol: 
xdr_gf_event_notify_rsp
warning: %post(glusterfs-server-3.3.0-1.el6.x86_64) scriptlet failed, 
exit status 127


I copied /etc/glusterd/* to /var/lib/glusterd/ and it seems to work. Is 
there some other issue I should expect to hit, or is the rpm just broken 
in a weird way?


On 5/31/12 2:55 PM, John Mark Walker wrote:
See this post - 
http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3/


Will publish that on gluster.org very soon.

-JM




Is there a migration guide from 3.2.5 to 3.3 available?

On 5/31/12 12:33 PM, John Mark Walker wrote:

Today, we’re announcing the next generation of GlusterFS
, version 3.3. The release has been a
year in the making and marks several firsts: the first
post-acquisition release under Red Hat, our first major act as
an openly-governed project
and our first foray beyond
NAS. We’ve also taken our first steps towards merging big data
and unstructured data storage, giving users and developers new
ways of managing their data scalability challenges.

GlusterFS is an open source, fully distributed storage
solution for the world’s ever-increasing volume of
unstructured data. It is a software-only, highly available,
scale-out, centrally managed storage pool that can be backed
by POSIX filesystems that support extended attributes, such as
Ext3/4, XFS, BTRFS and many more.

This release provides many of the most commonly requested
features including proactive self-healing, quorum enforcement,
and granular locking for self-healing, as well as many
additional bug fixes and enhancements.

Some of the more noteworthy features include:

  * Unified File and Object storage – Blending OpenStack’s
Object Storage API
 with GlusterFS
provides simultaneous read and write access to data as
files or as objects.
  * HDFS compatibility – Gives Hadoop administrators the
ability to run MapReduce jobs on unstructured data on
GlusterFS and access the data with well-known tools and
shell scripts.
  * Proactive self-healing – GlusterFS volumes will now
automatically restore file integrity after a replica
recovers from failure.
  * Granular locking – Allows large files to be accessed even
during self-healing, a feature that is particularly
important for VM images.
  * Replication improvements – With quorum enforcement you can
be confident that  your data has been written in at least
the configured number of places before the file operation
returns, allowing a user-configurable adjustment to fault
tolerance vs performance.

*
*Visit http://www.gluster.org  to
download. Packages are available for most distributions,
including Fedora, Debian, RHEL, Ubuntu and CentOS.

Get involved! Join us on #gluster on freenode, join our
mailing list ,
‘like’ our Facebook page ,
follow us on Twitter , or check
out our LinkedIn group .

GlusterFS is an open source project sponsored by Red Hat
®, who uses it in its line of Red Hat
Storage  products.

(this post published at
http://www.gluster.org/2012/05/introducing-glusterfs-3-3/ )



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@

Re: [Gluster-users] A very special announcement from Gluster.org

[David Coulson]

Is there a migration guide from 3.2.5 to 3.3 available?

On 5/31/12 12:33 PM, John Mark Walker wrote:


Today, we're announcing the next generation of GlusterFS 
, version 3.3. The release has been a year in 
the making and marks several firsts: the first post-acquisition 
release under Red Hat, our first major act as an openly-governed 
project and our first foray beyond 
NAS. We've also taken our first steps towards merging big data and 
unstructured data storage, giving users and developers new ways of 
managing their data scalability challenges.


GlusterFS is an open source, fully distributed storage solution for 
the world's ever-increasing volume of unstructured data. It is a 
software-only, highly available, scale-out, centrally managed storage 
pool that can be backed by POSIX filesystems that support extended 
attributes, such as Ext3/4, XFS, BTRFS and many more.


This release provides many of the most commonly requested features 
including proactive self-healing, quorum enforcement, and granular 
locking for self-healing, as well as many additional bug fixes and 
enhancements.


Some of the more noteworthy features include:

  * Unified File and Object storage -- Blending OpenStack's Object
Storage API  with
GlusterFS provides simultaneous read and write access to data as
files or as objects.
  * HDFS compatibility -- Gives Hadoop administrators the ability to
run MapReduce jobs on unstructured data on GlusterFS and access
the data with well-known tools and shell scripts.
  * Proactive self-healing -- GlusterFS volumes will now automatically
restore file integrity after a replica recovers from failure.
  * Granular locking -- Allows large files to be accessed even during
self-healing, a feature that is particularly important for VM images.
  * Replication improvements -- With quorum enforcement you can be
confident that  your data has been written in at least the
configured number of places before the file operation returns,
allowing a user-configurable adjustment to fault tolerance vs
performance.

*
*Visit http://www.gluster.org  to download. 
Packages are available for most distributions, including Fedora, 
Debian, RHEL, Ubuntu and CentOS.


Get involved! Join us on #gluster on freenode, join our mailing list 
, 'like' our Facebook 
page , follow us on Twitter 
, or check out our LinkedIn group 
.


GlusterFS is an open source project sponsored by Red Hat 
®, who uses it in its line of Red Hat Storage 
 products.


(this post published at 
http://www.gluster.org/2012/05/introducing-glusterfs-3-3/ )




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS on a two-node setup

[David Coulson]

On 5/20/12 5:55 PM, Ramon Diaz-Uriarte wrote:
I might have to look at DRBD more carefully, but I do not think it 
fits my needs: I need both nodes to be working (and thus doing I/O) at 
the same time. These are basically number crunching nodes and data 
needs to be accessible from both nodes (e.g., some jobs will use MPI 
over the CPUs/cores of both nodes ---assuming both nodes are up, of 
course ;-).
DRBD will let you do read/write on both nodes, but it requires a 
clustered filesystem such as GFS2 or OCFS2 on top of it. You are also 
limited to a max of two nodes.


But from the docs and the mailing list I get the impression that
replication has severe performance penalties when writing and some
penalties when reading. And with a two-node setup, it is unclear to me
that, even with replication, if one node fails, gluster will continue to
work (i.e., the other node will continue to work). I've not been able to
find what is the recommended procedure to continue working, with
replicated volumes, when one of the two nodes fails. So that is why I am
wondering what would replication really give me in this case.


Gluster doing replication requires writes to hit both nodes, which may 
slow you down a lot if there is significant latency between the two. I 
run a replicated configuration, and have had nodes down for extended 
periods - Gluster will repair the missing data from the brick on the 
failed node during self-heal, so it is transparent. I've never had to 
shut down applications in order for gluster to fix something first.


David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster client can't connect to Gluster volume

[David Coulson]

There's probably not a whole lot of 'fixing' since the two versions are 
so different.


You can tell the package maintainer to make a 3.2.6 build. That'll help.

On 5/5/12 1:43 PM, Eric wrote:

Thanks, David:

But I'd rather see if I can fix the problem with the Mageia client 
(|glusterfs-client-3.0.0-2.mga1|):


  * This will allow me to file a bug report with the package
maintainer (if necessary).
  * I already know that the systems that have Gluster 3.2.6 from
source [i.e., the storage nodes] are able to mount the volume.
  * I'd rather keep my daily-driver (i.e., the host system) 100% Mageia.

Eric Pretorious
Truckee, CA

--------
*From:* David Coulson 
*To:* Eric 
*Cc:* "gluster-users@gluster.org" 
*Sent:* Saturday, May 5, 2012 9:16 AM
*Subject:* Re: [Gluster-users] Gluster client can't connect to
Gluster volume

That's way old documentation.

Start by installing 3.2.6 on your client and see if it works then.
I don't think anyone expects 3.2 and 3.0 to work correctly.

On 5/5/12 12:09 PM, Eric wrote:

Thanks, David:

Yes...

  * iptables has been disabled on all three systems.
  * SELinux is set to permissive on the two systems that employ
it - the two CentOS nodes.
  * Port #6996 is referenced in the Troubleshooting section of
the Gluster User Guide

<http://www.gluster.org/community/documentation/index.php/User_Guide#Troubleshooting>.

FWIW: All of this except the SELinux question is already
documented in my post on the Mageia Forum
<https://forums.mageia.org/en/viewtopic.php?f=7&t=2358&p=17517>.

Eric Pretorious
Truckee, CA

--------
*From:* David Coulson 
<mailto:da...@davidcoulson.net>
*To:* Eric 
<mailto:epretori...@yahoo.com>
*Cc:* "gluster-users@gluster.org"
<mailto:gluster-users@gluster.org>
 <mailto:gluster-users@gluster.org>
*Sent:* Saturday, May 5, 2012 5:44 AM
*Subject:* Re: [Gluster-users] Gluster client can't connect
to Gluster volume

Do you have any firewall rules enabled? I'd start by
disabling iptables (or at least setting everything to ACCEPT)
and as someone else suggested setting selinux to
permissive/disabled.

Why are your nodes and client using different versions of
Gluster? Why not just use the 3.2.6 version for everything?
Also, I'm not sure where port 6996 comes from - Gluster uses
24007 for it's core communications and ports above that for
individual bricks.

David

On 5/5/12 12:27 AM, Eric wrote:

Hi, All:

I've built a Gluster-based storage cluster on a pair of
CentOS 5.7 (i386) VM's. The nodes are using Gluster 3.2.6
(from source) and the host is using Gluster 3.0.0 (from the
Mageia package repositories):

|[eric@node1 ~]$ sudo /usr/local/sbin/gluster --version
glusterfs 3.2.6 built on May  3 2012 15:53:02

||[eric@localhost ~]$ rpm -qa | grep glusterfs|
|glusterfs-common-3.0.0-2.mga1|
|glusterfs-client-3.0.0-2.mga1|
|glusterfs-server-3.0.0-2.mga1|
|libglusterfs0-3.0.0-2.mga1|

None of the systems (i.e., neither the two storage nodes nor
the client) can connect to Port 6996 of the cluster
(node1.example.com <http://node1.example.com> &
node2.example.com <http://node2.example.com>) but the two
storage nodes can mount the shared volume using the Gluster
helper and/or NFS:

|[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse

[eric@node1 ~]$ sudo /sbin/modprobe fuse

[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse
fuse   49237  0

[eric@node1 ~]$ sudo mount -t glusterfs node1:/mirror-1 /mnt

[eric@node1 ~]$ sudo grep gluster /etc/mtab
glusterfs#node1:/mirror-1 /mnt fuse
rw,allow_other,default_permissions,max_read=131072 0 0|

...but the host system is only able to connect using NFS:

|[eric@localhost ~]$ sudo glusterfs --debug -f
/tmp/glusterfs.vol /mnt
[2012-05-04 19:09:09] D [glusterfsd.c:424:_get_specfp]
glusterfs: loading volume file /tmp/glusterfs.vol


Version  : glusterfs 3.0.0 built on Apr 10 2011 19:12:54
git: 2.0.1-886-g8379edd
Starting Time: 2012-05-04 19:09:09
Command line : glusterfs --debug -f /tmp/glusterfs.vol /mnt
PID  : 30159
System name  : Linu

Re: [Gluster-users] Gluster client can't connect to Gluster volume

[David Coulson]

That's way old documentation.

Start by installing 3.2.6 on your client and see if it works then. I 
don't think anyone expects 3.2 and 3.0 to work correctly.


On 5/5/12 12:09 PM, Eric wrote:

Thanks, David:

Yes...

  * iptables has been disabled on all three systems.
  * SELinux is set to permissive on the two systems that employ it -
the two CentOS nodes.
  * Port #6996 is referenced in the Troubleshooting section of the
Gluster User Guide

<http://www.gluster.org/community/documentation/index.php/User_Guide#Troubleshooting>.

FWIW: All of this except the SELinux question is already documented in 
my post on the Mageia Forum 
<https://forums.mageia.org/en/viewtopic.php?f=7&t=2358&p=17517>.


Eric Pretorious
Truckee, CA

--------
*From:* David Coulson 
*To:* Eric 
*Cc:* "gluster-users@gluster.org" 
*Sent:* Saturday, May 5, 2012 5:44 AM
*Subject:* Re: [Gluster-users] Gluster client can't connect to
Gluster volume

Do you have any firewall rules enabled? I'd start by disabling
iptables (or at least setting everything to ACCEPT) and as someone
else suggested setting selinux to permissive/disabled.

Why are your nodes and client using different versions of Gluster?
Why not just use the 3.2.6 version for everything? Also, I'm not
sure where port 6996 comes from - Gluster uses 24007 for it's core
communications and ports above that for individual bricks.

David

On 5/5/12 12:27 AM, Eric wrote:

Hi, All:

I've built a Gluster-based storage cluster on a pair of CentOS
5.7 (i386) VM's. The nodes are using Gluster 3.2.6 (from source)
and the host is using Gluster 3.0.0 (from the Mageia package
repositories):

|[eric@node1 ~]$ sudo /usr/local/sbin/gluster --version
glusterfs 3.2.6 built on May  3 2012 15:53:02

||[eric@localhost ~]$ rpm -qa | grep glusterfs|
|glusterfs-common-3.0.0-2.mga1|
|glusterfs-client-3.0.0-2.mga1|
|glusterfs-server-3.0.0-2.mga1|
|libglusterfs0-3.0.0-2.mga1|

None of the systems (i.e., neither the two storage nodes nor the
client) can connect to Port 6996 of the cluster
(node1.example.com <http://node1.example.com> & node2.example.com
<http://node2.example.com>) but the two storage nodes can mount
the shared volume using the Gluster helper and/or NFS:

|[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse

[eric@node1 ~]$ sudo /sbin/modprobe fuse

[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse
fuse   49237  0

[eric@node1 ~]$ sudo mount -t glusterfs node1:/mirror-1 /mnt

[eric@node1 ~]$ sudo grep gluster /etc/mtab
glusterfs#node1:/mirror-1 /mnt fuse
rw,allow_other,default_permissions,max_read=131072 0 0|

...but the host system is only able to connect using NFS:

|[eric@localhost ~]$ sudo glusterfs --debug -f /tmp/glusterfs.vol
/mnt
[2012-05-04 19:09:09] D [glusterfsd.c:424:_get_specfp] glusterfs:
loading volume file /tmp/glusterfs.vol


Version  : glusterfs 3.0.0 built on Apr 10 2011 19:12:54
git: 2.0.1-886-g8379edd
Starting Time: 2012-05-04 19:09:09
Command line : glusterfs --debug -f /tmp/glusterfs.vol /mnt
PID  : 30159
System name  : Linux
Nodename : localhost.localdomain
Kernel Release : 2.6.38.8-desktop586-10.mga
Hardware Identifier: i686

Given volfile:

+--+
  1: volume mirror-1
  2:  type protocol/client
  3:  option transport-type tcp
  4:  option remote-host node1.example.com
  5:  option remote-subvolume mirror-1
  6: end-volume

+--+
[2012-05-04 19:09:09] D [glusterfsd.c:1335:main] glusterfs:
running in pid 30159
[2012-05-04 19:09:09] D [client-protocol.c:6581:init] mirror-1:
defaulting frame-timeout to 30mins
[2012-05-04 19:09:09] D [client-protocol.c:6592:init] mirror-1:
defaulting ping-timeout to 42
[2012-05-04 19:09:09] D [transport.c:145:transport_load]
transport: attempt to load file
/usr/lib/glusterfs/3.0.0/transport/socket.so
[2012-05-04 19:09:09] D [transport.c:145:transport_load]
transport: attempt to load file
/usr/lib/glusterfs/3.0.0/transport/socket.so
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1:
got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1:
got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1:
got GF_EVENT_PARENT_UP, attempting connect on transport
[2012-

Re: [Gluster-users] Gluster client can't connect to Gluster volume

[David Coulson]

Do you have any firewall rules enabled? I'd start by disabling iptables 
(or at least setting everything to ACCEPT) and as someone else suggested 
setting selinux to permissive/disabled.


Why are your nodes and client using different versions of Gluster? Why 
not just use the 3.2.6 version for everything? Also, I'm not sure where 
port 6996 comes from - Gluster uses 24007 for it's core communications 
and ports above that for individual bricks.


David

On 5/5/12 12:27 AM, Eric wrote:

Hi, All:

I've built a Gluster-based storage cluster on a pair of CentOS 5.7 
(i386) VM's. The nodes are using Gluster 3.2.6 (from source) and the 
host is using Gluster 3.0.0 (from the Mageia package repositories):


|[eric@node1 ~]$ sudo /usr/local/sbin/gluster --version
glusterfs 3.2.6 built on May  3 2012 15:53:02

||[eric@localhost ~]$ rpm -qa | grep glusterfs|
|glusterfs-common-3.0.0-2.mga1|
|glusterfs-client-3.0.0-2.mga1|
|glusterfs-server-3.0.0-2.mga1|
|libglusterfs0-3.0.0-2.mga1|

None of the systems (i.e., neither the two storage nodes nor the 
client) can connect to Port 6996 of the cluster (node1.example.com & 
node2.example.com) but the two storage nodes can mount the shared 
volume using the Gluster helper and/or NFS:


|[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse

[eric@node1 ~]$ sudo /sbin/modprobe fuse

[eric@node1 ~]$ sudo /sbin/lsmod | grep fuse
fuse   49237  0

[eric@node1 ~]$ sudo mount -t glusterfs node1:/mirror-1 /mnt

[eric@node1 ~]$ sudo grep gluster /etc/mtab
glusterfs#node1:/mirror-1 /mnt fuse 
rw,allow_other,default_permissions,max_read=131072 0 0|


...but the host system is only able to connect using NFS:

|[eric@localhost ~]$ sudo glusterfs --debug -f /tmp/glusterfs.vol /mnt
[2012-05-04 19:09:09] D [glusterfsd.c:424:_get_specfp] glusterfs: 
loading volume file /tmp/glusterfs.vol


Version  : glusterfs 3.0.0 built on Apr 10 2011 19:12:54
git: 2.0.1-886-g8379edd
Starting Time: 2012-05-04 19:09:09
Command line : glusterfs --debug -f /tmp/glusterfs.vol /mnt
PID  : 30159
System name  : Linux
Nodename : localhost.localdomain
Kernel Release : 2.6.38.8-desktop586-10.mga
Hardware Identifier: i686

Given volfile:
+--+
  1: volume mirror-1
  2:  type protocol/client
  3:  option transport-type tcp
  4:  option remote-host node1.example.com
  5:  option remote-subvolume mirror-1
  6: end-volume
+--+
[2012-05-04 19:09:09] D [glusterfsd.c:1335:main] glusterfs: running in 
pid 30159
[2012-05-04 19:09:09] D [client-protocol.c:6581:init] mirror-1: 
defaulting frame-timeout to 30mins
[2012-05-04 19:09:09] D [client-protocol.c:6592:init] mirror-1: 
defaulting ping-timeout to 42
[2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: 
attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so
[2012-05-04 19:09:09] D [transport.c:145:transport_load] transport: 
attempt to load file /usr/lib/glusterfs/3.0.0/transport/socket.so
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] D [client-protocol.c:7005:notify] mirror-1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2012-05-04 19:09:09] N [glusterfsd.c:1361:main] glusterfs: 
Successfully started
[2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: 
connection to  failed (Connection refused)
[2012-05-04 19:09:09] D [fuse-bridge.c:3079:fuse_thread_proc] fuse:  
pthread_cond_timedout returned non zero value ret: 0 errno: 0
[2012-05-04 19:09:09] N [fuse-bridge.c:2931:fuse_init] glusterfs-fuse: 
FUSE inited with protocol versions: glusterfs 7.13 kernel 7.16
[2012-05-04 19:09:09] E [socket.c:760:socket_connect_finish] mirror-1: 
connection to  failed (Connection refused)|


I've read through the Troubleshooting section of the Gluster 
Administration Guide 
 
and the Gluster User Guide 
but 
can't seem to resolve the problem. (See my post on the Mageia Forum 
 for 
all the troubleshooting details: 
https://forums.mageia.org/en/viewtopic.php?f=7&t=2358&p=17517)


What might be causing this?

TIA,
Eric Pretorious
Truckee, CA

https://forums.mageia.org/en/viewtopic.php?f=7&t=2358&p=17517


___
Gluster-users mailing list
Glus

Re: [Gluster-users] multi-interface/IP peers

[David Coulson]

Gluster relies on DNS and/or /etc/hosts to determine the IP for a 
particular cluster member. You can have gluster utilize a different IP 
for *new* connections by updating DNS or /etc/hosts to point the cluster 
peer name to a new IP.


On 4/21/12 7:31 AM, lejeczek wrote:

helo everybody

this I'd imagine must be common scenario, where a peer, or more peers 
are multi if/IPs nodes

for an instance
what happens if a cheaper route to a peer is made available after some 
time a volume has been up?
how one introduces this change to the volume? how one tells the 
gluster to use different IP whereas everything else remains unchanged?


cheers
lejeczek




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Setting up a 2 sites replication with access to files from both sites

[David Coulson]

I am not aware of a mechanism where you can force a read from only one brick, 
but write to both, short of having reads done out of a totally separate 
directory structure.

Someone can correct me if this is wrong, but that is how my replica 
configuration behaves w/ 3.2.5.

David



On Apr 6, 2012, at 10:24 AM, François Legal wrote:

> Hello,
> 
>  
> This is my first post to the list.
> 
>  
> I have 2 distant sites linked through a 10MBits MPLS.
> 
> To improve user experience, we would like to setup a replicated glusterfs on 
> the 2 sites, such that a user on site A would access the files from site A 
> replicated server, and a user on site B would access the files from site B 
> replicated server.
> 
> So I was thinking of a dual server setup, each being client and server for 
> its site, so that read access to files would be fast, and write access would 
> be as fast as MPLS allows it to be.
> 
> The files would then be shared using samba to the users.
> 
>  
> In this situation, I don't know if I should prefer client side AFR or server 
> side AFR (I guess server side AFR would make the files on both sites 
> unavailable if MPLS fails).
> 
>  
> Anyway, I gave a test drive on this, and seem to be having locking issues 
> (users on both side can modify the same file without anything preventing 
> this).
> 
>  
> Can anybody comment on this.
> 
>  
> Thanks in advance.
> 
>  
> François
> 
>  
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow performance from simple tar -x && rm -r benchmark

[David Coulson]

Weird - Actually slower than fuse. Does the 'nolock' nfs mount option 
make a difference?


On 3/21/12 1:22 PM, Bryan Whitehead wrote:

[root@lab0-v3 ~]# mount -t nfs -o tcp,nfsvers=3 localhost:/images /mnt
[root@lab0-v3 ~]# cd /mnt
[root@lab0-v3 mnt]# time bash -c 'tar xf /root/linux-3.3.tar ; sync ;
rm -rf linux-3.3'

real2m26.698s
user0m0.334s
sys 0m6.943s


On Wed, Mar 21, 2012 at 6:22 AM, David Coulson  wrote:


On 3/20/12 2:47 AM, Bryan Whitehead wrote:

I'm going to start off and say that I misstated, I must have been
doing my *many-file* tests *inside* VM's running on top of glusterfs.
I post a loopback test later this week.


Can you repeat the test using NFS rather than Fuse? I've seen a approx 2x
performance increase with 'small files' using NFS over Fuse, however I'm not
sure if it's just anecdotal, or actually a reality.

David

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow performance from simple tar -x && rm -r benchmark

[David Coulson]

On 3/20/12 2:47 AM, Bryan Whitehead wrote:

I'm going to start off and say that I misstated, I must have been
doing my *many-file* tests *inside* VM's running on top of glusterfs.
I post a loopback test later this week.

Can you repeat the test using NFS rather than Fuse? I've seen a approx 
2x performance increase with 'small files' using NFS over Fuse, however 
I'm not sure if it's just anecdotal, or actually a reality.


David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Usage Case: just not getting the performance I was hoping for

[David Coulson]

Is there a FAQ/document somewhere with optimal mkfs and mount options 
for ext4 and xfs? Is xfs still the 'desired' filesystem for gluster bricks?


On 3/15/12 3:22 AM, Brian Candler wrote:

On Wed, Mar 14, 2012 at 11:09:28PM -0500, D. Dante Lorenso wrote:

get 50-60 MB/s transfer speeds tops when sending large files (>  2GB)
to gluster. When copying a directory of small files, we get<= 1
MB/s performance!

My question is ... is this right?  Is this what I should expect from
Gluster, or is there something we did wrong?  We aren't using super
expensive equipment, granted, but I was really hoping for better
performance than this given that raw drive speeds using dd show that
we can write at 125+ MB/s to each "brick" 2TB disk.

Unfortunately I don't have any experience with replicated volumes, but the
raw glusterfs protocol is very fast: a single brick which is a 12-disk raid0
stripe can give 500MB/sec easily over 10G ethernet without any tuning.

I would expect a distributed volume to work fine too, as it just sends each
request to one of N nodes.

Striped volumes are unfortunately broken on top of XFS at the moment:
http://oss.sgi.com/archives/xfs/2012-03/msg00161.html

Replicated volumes, from what I've read, need to touch both servers even for
read operations (for the self-healing functionality), and that could be a
major bottleneck.

But there are a few basic things to check:

(1) Are you using XFS for the underlying filesystems? If so, did you mount
them with the "inode64" mount option?  Without this, XFS performance sucks
really badly for filesystems>1TB

Without inode64, even untarring files into a single directory will make XFS
distribute them between AGs, rather than allocating contiguous space for
them.

This is a major trip-up and there is currently talk of changing the default
to be inode64.

(2) I have this in /etc/rc.local:

for i in /sys/block/sd*/bdi/read_ahead_kb; do echo 1024>"$i"; done
for i in /sys/block/sd*/queue/max_sectors_kb; do echo 1024>"$i"; done


If I can't get gluster to work, our fail-back plan is to convert
these 8 servers into iSCSI targets and mount the storage onto a
Win2008 head and continue sharing to the network as before.
Personally, I would rather us continue moving toward CentOS 6.2 with
Samba and Gluster, but I can't justify the change unless I can
deliver the performance.

Optimising replicated volumes I can't help with.

However if you make a simple RAID10 array on each server, and then join the
servers into a distributed gluster volume, I think it will rock.  What you
lose is the high-availability, i.e.  if one server fails a proportion of
your data becomes unavailable until you fix it - but that's no worse than
your iSCSI proposal (unless you are doing something complex, like drbd
replication between pairs of nodes and HA failover of the iSCSI target)

BTW, Linux md RAID10 with 'far' layout is really cool; for reads it performs
like a RAID0 stripe, and it reduces head seeking for random access.

Regards,

Brian.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Question regarding NFS mount points

[David Coulson]

I ended up using the 'nolock' option with NFS - Even with only one 
client mounted, I had issues with locking.


On 3/14/12 5:26 PM, Sean Fulton wrote:
We have a four-node, replicated cluster. When using the native gluster 
client, we use the local server as the mount point (ie., mount 
localhost:/glustervolume /dir). For NFS, can we do the same, or should 
all of the clients mount from the same server to ensure proper 
locking?? I've tried using the localhost for NFS but we seemed to have 
lock contention.


Recommendations would be appreciated.

sean


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Usage Case: just not getting the performance I was hoping for

[David Coulson]

On 3/15/12 9:46 AM, Sean Fulton wrote:
In a case where four client nodes need equal read/write access to the 
data, is it better to have four Gluster nodes in a replicated 
configuration with each mounting the gluster volume locally, or having 
TWO Gluster server nodes with the four clients mounting the volume 
from the two servers? In other words, the replication would only touch 
two nodes instead of four; would that improve performance.

Yes. two replicated nodes is faster for writes than four.


Also, would NFS be better in this case, mounting from just ONE of the 
server nodes, or using Gluster native client to mount from either of 
the two server nodes. Or can NFS mount from any/all of the gluster 
nodes using the gluster NFS server.
Never tested it, but in theory running the native gluster client should 
be faster than NFS - The client will connect to the two nodes 
individually, rather than sending everything through the NFS server, 
then it distributing reads and paralleling writes. Remember, when you 
use the native client it connects to whichever node you specify to get 
volume information, then connects to each brick in the volume directly - 
If there is not a brick for the volume on the server you tell it to 
connect to, it won't send any IO to it. That is a major distinction 
between native client and NFS.


A better comparison would be NFS to remote node vs. NFS to local node 
(but with no bricks on local node). I use the latter extensively, since 
it's less work than trying to deal with NFS redundancy.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] QA builds for 3.2.6 and 3.3 beta3

[David Coulson]

Is there a change log somewhere for 3.2.6 (or the p3 which is available 
for QA)?


David

On 3/14/12 3:21 PM, John Mark Walker wrote:

Greetings,

There are 2 imminent releases coming soon to a download server near you:

1. GlusterFS 3.2.6 - a maintenance release that fixes some bugs.
2. GlusterFS 3.3 beta 3 - the next iteration of the exciting new hotness that 
will be 3.3

You can find both of these in the "QA builds" server:

http://bits.gluster.com/pub/gluster/glusterfs/


There are source tarballs and binary RPMs at the moment.


Thanks,
John Mark Walker
Gluster Community Guy
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] NFS: server localhost error: fileid changed

[David Coulson]

I recently moved from the fuse client to NFS - Now I'm seeing a bunch of 
this in syslog. Is this something to be concerned about, or is it 
'normal' NFS behavior?


NFS: server localhost error: fileid changed
fsid 0:15: expected fileid 0xd88ba88a97875981, got 0x40e476ef5fdfbe9f

I also see a lot of 'stale file handle' in nfs.log, but the timestamps 
don't correspond.


I'm running gluster 3.2.5 on RHEL6.

David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Unable to start nfs server

[David Coulson]

I poke around at this today and figured it out - Sort of.

When I did 'gluster volume set named nfs.register-with-portmap on' 
originally, I only had rpcbind running on two of my four servers. 
Gluster nfs started up on all four, but obviously only two correctly 
connected with rpcbind/portmap. Seems that if rpcbind is not running 
when you set 'register-with-portmap on', even when you cycle gluster it 
still doesn't work.


So, I started up rpcbind, did a 'register-with-portmap off', followed by 
'register-with-portmap on' and it works now.


When I diffed the before and after nfs-server.vol files, I see this:

[root@dresproddns01 ~]# diff nfs-vol-1 /etc/glusterd/nfs/nfs-server.vol
143c143
< option rpc.register-with-portmap off
---
> option rpc.register-with-portmap on


Apparently if rpcbind is not running, the option does not get enabled 
properly. There is an error in nfs.log, but it's hard to find especially 
if the node you manage the cluster from isn't the node with the issue. 
It isn't clear either that it's broken even if you cycle gluster (and 
even though the gluster volume configuration says 'register-with-portmap 
on'. Does the 'gluster volume set' command have the ability to get 
success/fail information back from each node? It also appears that 
'register-with-portmap' is applied to all volumes, even if you just 
enable it on one - Is there a cluster-wide place to 'set' options?


[2012-03-05 19:51:22.517368] E 
[rpcsvc.c:2771:nfs_rpcsvc_program_register_portmap] 0-nfsrpc: Could not 
register with portmap
[2012-03-05 19:51:22.517420] E 
[rpcsvc.c:2861:nfs_rpcsvc_program_register] 0-nfsrpc: portmap 
registration of program failed
[2012-03-05 19:51:22.517428] E 
[rpcsvc.c:2874:nfs_rpcsvc_program_register] 0-nfsrpc: Program 
registration failed: MOUNT3, Num: 15, Ver: 3, Port: 38465



David


On 3/5/12 2:05 PM, Bryan Whitehead wrote:

Is selinux running? iptables?

Can you http://pastie.org/ the nfs.log in /var/log/glusterfs ?

On Mon, Mar 5, 2012 at 3:59 AM, David Coulson <mailto:da...@davidcoulson.net>> wrote:


Yep.

[root@dresproddns01 ~]# service glusterd stop
Stopping glusterd: [  OK  ]

[root@dresproddns01 ~]# ps ax | grep nfs
 120494 pts/0S+ 0:00 grep nfs

2167119 ?S  0:00 [nfsiod]
[root@dresproddns01 ~]# service rpcbind stop
Stopping rpcbind:  [  OK  ]

[root@dresproddns01 ~]# rpcinfo -p
rpcinfo: can't contact portmapper: RPC: Remote system error - No
such file or directory
[root@dresproddns01 ~]# service rpcbind start
Starting rpcbind:  [  OK  ]
[root@dresproddns01 ~]# service glusterd start
Starting glusterd: [  OK  ]

[root@dresproddns01 ~]# rpcinfo -p
   program vers proto   port  service
104   tcp111  portmapper
103   tcp111  portmapper
102   tcp111  portmapper
104   udp111  portmapper
103   udp111  portmapper
102   udp111  portmapper

Note that I waited a short while between the last two steps. FYI,
this is RHEL6 (the two systems that work are RHEL6 too, so I'm not
sure it matters much).


On 3/5/12 3:27 AM, Bryan Whitehead wrote:

did you start portmap service before you started gluster?

On Sun, Mar 4, 2012 at 11:53 AM, David Coulson
mailto:da...@davidcoulson.net>> wrote:

I've four systems with multiple 4-way replica volumes. I'm
migrating a number of volumes from Fuse to NFS for
performance reasons.

My first two hosts seem to work nicely, but the other two
won't start the NFS services properly. I looked through the
nfs.log, but it doesn't give any indication of why it did not
register with rpcbind. I'm presuming I've got a
misconfiguration on two of the systems, but there isn't a
clear indication of what is not working.

Here is an example from a host which does not work:

[root@dresproddns01 ~]# rpcinfo -p
   program vers proto   port  service
104   tcp111  portmapper
103   tcp111  portmapper
102   tcp111  portmapper
104   udp111  portmapper
103   udp111  portmapper
102   udp111  portmapper
[root@dresproddns01 ~]# ps ax | grep nfs
2167119 ?S  0:00 [nfsiod]
2738268 ?Ssl0:00
/opt/glusterfs/3.2.5/sbin/glusterfs -f
/etc/glusterd/nfs/nfs-server.vol -p
/etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs

Re: [Gluster-users] Unable to start nfs server

[David Coulson]

http://pastie.org/3528212

SELinux/iptables info is below. FYI I started up the standard RedHat nfs server 
(with gluster shutdown), and it started cleanly and correctly bound to 
portmap/rpcbind.

[root@dresproddns02 ~]# getenforce 
Permissive
[root@dresproddns02 ~]# iptables-save 
# Generated by iptables-save v1.4.7 on Mon Mar  5 14:34:27 2012
*filter
:INPUT ACCEPT [51197275114:5822299813362]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [51184812127:5800516863347]
COMMIT
# Completed on Mon Mar  5 14:34:27 2012
# Generated by iptables-save v1.4.7 on Mon Mar  5 14:34:27 2012
*nat
:PREROUTING ACCEPT [663610:53184995]
:POSTROUTING ACCEPT [4471677:292235853]
:OUTPUT ACCEPT [4471677:292235853]
-A PREROUTING -d 172.31.0.1/32 -p udp -m udp --dport 53 -j DNAT 
--to-destination 169.254.202.2 
-A PREROUTING -d 172.31.0.2/32 -p udp -m udp --dport 53 -j DNAT 
--to-destination 169.254.202.2 
COMMIT
# Completed on Mon Mar  5 14:34:27 2012
# Generated by iptables-save v1.4.7 on Mon Mar  5 14:34:27 2012
*mangle
:PREROUTING ACCEPT [330516307:119149868984]
:INPUT ACCEPT [330469748:119144996058]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [319783522:99640884653]
:POSTROUTING ACCEPT [320799566:99775260122]
-A PREROUTING -d 172.31.0.0/32 -i bond0 -p tcp -m tcp --dport 5222 -j MARK 
--set-xmark 0x2bc/0x 
-A PREROUTING -d 172.31.0.0/32 -i bond0 -p tcp -m tcp --dport 5262 -j MARK 
--set-xmark 0x2bd/0x 
-A PREROUTING -d 172.31.0.0/32 -i bond0 -p tcp -m multiport --dports 9091,9091 
-j MARK --set-xmark 0x2be/0x 
-A PREROUTING -d 172.31.0.0/32 -i bond0 -p tcp -m tcp --dport 5222 -j MARK 
--set-xmark 0x2bc/0x 
-A PREROUTING -d 172.31.0.0/32 -i bond0 -p tcp -m tcp --dport 5262 -j MARK 
--set-xmark 0x2bd/0x 
-A PREROUTING -d 172.31.0.0/32 -i bond0 -p tcp -m multiport --dports 9091,9091 
-j MARK --set-xmark 0x2be/0x 
-A PREROUTING -d 172.31.0.64/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 
0x103/0x 
COMMIT
# Completed on Mon Mar  5 14:34:27 2012
[root@dresproddns02 ~]# 



On Mar 5, 2012, at 2:05 PM, Bryan Whitehead wrote:

> Is selinux running? iptables?
> 
> Can you http://pastie.org/ the nfs.log in /var/log/glusterfs ? 
> 
> On Mon, Mar 5, 2012 at 3:59 AM, David Coulson  wrote:
> Yep.
> 
> [root@dresproddns01 ~]# service glusterd stop
> Stopping glusterd: [  OK  ]
> 
> [root@dresproddns01 ~]# ps ax | grep nfs
>  120494 pts/0S+ 0:00 grep nfs
> 
> 2167119 ?S  0:00 [nfsiod]
> [root@dresproddns01 ~]# service rpcbind stop
> Stopping rpcbind:  [  OK  ]
> 
> [root@dresproddns01 ~]# rpcinfo -p
> rpcinfo: can't contact portmapper: RPC: Remote system error - No such file or 
> directory
> [root@dresproddns01 ~]# service rpcbind start
> Starting rpcbind:  [  OK  ]
> [root@dresproddns01 ~]# service glusterd start
> Starting glusterd: [  OK  ]
> 
> [root@dresproddns01 ~]# rpcinfo -p
>program vers proto   port  service
> 104   tcp111  portmapper
> 103   tcp111  portmapper
> 102   tcp111  portmapper
> 104   udp111  portmapper
> 103   udp111  portmapper
> 102   udp111  portmapper
> 
> Note that I waited a short while between the last two steps. FYI, this is 
> RHEL6 (the two systems that work are RHEL6 too, so I'm not sure it matters 
> much).
> 
> 
> On 3/5/12 3:27 AM, Bryan Whitehead wrote:
>> 
>> did you start portmap service before you started gluster?
>> 
>> On Sun, Mar 4, 2012 at 11:53 AM, David Coulson  
>> wrote:
>> I've four systems with multiple 4-way replica volumes. I'm migrating a 
>> number of volumes from Fuse to NFS for performance reasons.
>> 
>> My first two hosts seem to work nicely, but the other two won't start the 
>> NFS services properly. I looked through the nfs.log, but it doesn't give any 
>> indication of why it did not register with rpcbind. I'm presuming I've got a 
>> misconfiguration on two of the systems, but there isn't a clear indication 
>> of what is not working.
>> 
>> Here is an example from a host which does not work:
>> 
>> [root@dresproddns01 ~]# rpcinfo -p
>>program vers proto   port  service
>> 104   tcp111  portmapper
>> 103   tcp111  portmapper
>> 102   tcp111  portmapper
>> 104   udp111  portmapper
>> 103   udp111  portmapper
>> 102   udp111  portmapper
>> [root@dresproddns01 ~]# ps ax | grep nfs
>> 2167119 ?S  0:00 [nfsiod]
>> 

Re: [Gluster-users] Unable to start nfs server

[David Coulson]

Yep.

[root@dresproddns01 ~]# service glusterd stop
Stopping glusterd: [  OK  ]
[root@dresproddns01 ~]# ps ax | grep nfs
 120494 pts/0S+ 0:00 grep nfs
2167119 ?S  0:00 [nfsiod]
[root@dresproddns01 ~]# service rpcbind stop
Stopping rpcbind:  [  OK  ]
[root@dresproddns01 ~]# rpcinfo -p
rpcinfo: can't contact portmapper: RPC: Remote system error - No such 
file or directory

[root@dresproddns01 ~]# service rpcbind start
Starting rpcbind:  [  OK  ]
[root@dresproddns01 ~]# service glusterd start
Starting glusterd: [  OK  ]
[root@dresproddns01 ~]# rpcinfo -p
   program vers proto   port  service
104   tcp111  portmapper
103   tcp111  portmapper
102   tcp111  portmapper
104   udp111  portmapper
103   udp111  portmapper
102   udp111  portmapper

Note that I waited a short while between the last two steps. FYI, this 
is RHEL6 (the two systems that work are RHEL6 too, so I'm not sure it 
matters much).


On 3/5/12 3:27 AM, Bryan Whitehead wrote:

did you start portmap service before you started gluster?

On Sun, Mar 4, 2012 at 11:53 AM, David Coulson <mailto:da...@davidcoulson.net>> wrote:


I've four systems with multiple 4-way replica volumes. I'm
migrating a number of volumes from Fuse to NFS for performance
reasons.

My first two hosts seem to work nicely, but the other two won't
start the NFS services properly. I looked through the nfs.log, but
it doesn't give any indication of why it did not register with
rpcbind. I'm presuming I've got a misconfiguration on two of the
systems, but there isn't a clear indication of what is not working.

Here is an example from a host which does not work:

[root@dresproddns01 ~]# rpcinfo -p
   program vers proto   port  service
104   tcp111  portmapper
103   tcp111  portmapper
102   tcp111  portmapper
104   udp111  portmapper
103   udp111  portmapper
102   udp111  portmapper
[root@dresproddns01 ~]# ps ax | grep nfs
2167119 ?S  0:00 [nfsiod]
2738268 ?Ssl0:00 /opt/glusterfs/3.2.5/sbin/glusterfs
-f /etc/glusterd/nfs/nfs-server.vol -p
/etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
2934228 pts/0S+ 0:00 grep nfs
[root@dresproddns01 ~]# netstat -ntlp | grep 2738268
tcp0  0 0.0.0.0:38465
<http://0.0.0.0:38465>   0.0.0.0:*  
LISTEN  2738268/glusterfs

tcp0  0 0.0.0.0:38466
<http://0.0.0.0:38466>   0.0.0.0:*  
LISTEN  2738268/glusterfs

tcp0  0 0.0.0.0:38467
<http://0.0.0.0:38467>   0.0.0.0:*  
LISTEN  2738268/glusterfs


[root@dresproddns01 ~]# gluster volume info svn

Volume Name: svn
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: rhesproddns01:/gluster/svn
Brick2: rhesproddns02:/gluster/svn
Brick3: dresproddns01:/gluster/svn
Brick4: dresproddns02:/gluster/svn
Options Reconfigured:
performance.client-io-threads: 1
performance.flush-behind: on
network.ping-timeout: 5
performance.stat-prefetch: 1
nfs.disable: off
nfs.register-with-portmap: on
auth.allow: 10.250.53.*,10.252.248.*,169.254.*,127.0.0.1
performance.cache-size: 256Mb
performance.write-behind-window-size: 128Mb

Only obvious difference with a host which does work is this:

[root@rhesproddns01 named]# rpcinfo -p
   program vers proto   port  service
104   tcp111  portmapper
103   tcp111  portmapper
102   tcp111  portmapper
104   udp111  portmapper
103   udp111  portmapper
102   udp111  portmapper
153   tcp  38465  mountd
151   tcp  38466  mountd
133   tcp  38467  nfs


Any ideas where to look for errors?


___
Gluster-users mailing list
Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Unable to start nfs server

[David Coulson]

  
  
I've four systems with multiple 4-way replica volumes. I'm
  migrating a number of volumes from Fuse to NFS for performance
  reasons.
  
  My first two hosts seem to work nicely, but the other two won't
  start the NFS services properly. I looked through the nfs.log, but
  it doesn't give any indication of why it did not register with
  rpcbind. I'm presuming I've got a misconfiguration on two of the
  systems, but there isn't a clear indication of what is not
  working.
  
  Here is an example from a host which does not work:
  
  [root@dresproddns01 ~]# rpcinfo -p
     program vers proto   port  service
      10    4   tcp    111  portmapper
      10    3   tcp    111  portmapper
      10    2   tcp    111  portmapper
      10    4   udp    111  portmapper
      10    3   udp    111  portmapper
      10    2   udp    111  portmapper
  [root@dresproddns01 ~]# ps ax | grep nfs
  2167119 ?    S  0:00 [nfsiod]
  2738268 ?    Ssl    0:00 /opt/glusterfs/3.2.5/sbin/glusterfs
  -f /etc/glusterd/nfs/nfs-server.vol -p
  /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
  2934228 pts/0    S+ 0:00 grep nfs
  [root@dresproddns01 ~]# netstat -ntlp | grep 2738268
  tcp    0  0 0.0.0.0:38465  
  0.0.0.0:*   LISTEN  2738268/glusterfs   
  tcp    0  0 0.0.0.0:38466  
  0.0.0.0:*   LISTEN  2738268/glusterfs   
  tcp    0  0 0.0.0.0:38467  
  0.0.0.0:*   LISTEN  2738268/glusterfs  
  
  [root@dresproddns01 ~]# gluster volume info svn
  
  Volume Name: svn
  Type: Replicate
  Status: Started
  Number of Bricks: 4
  Transport-type: tcp
  Bricks:
  Brick1: rhesproddns01:/gluster/svn
  Brick2: rhesproddns02:/gluster/svn
  Brick3: dresproddns01:/gluster/svn
  Brick4: dresproddns02:/gluster/svn
  Options Reconfigured:
  performance.client-io-threads: 1
  performance.flush-behind: on
  network.ping-timeout: 5
  performance.stat-prefetch: 1
  nfs.disable: off
  nfs.register-with-portmap: on
  auth.allow: 10.250.53.*,10.252.248.*,169.254.*,127.0.0.1
  performance.cache-size: 256Mb
  performance.write-behind-window-size: 128Mb
  
  Only obvious difference with a host which does work is this:
  
  [root@rhesproddns01 named]# rpcinfo -p
     program vers proto   port  service
      10    4   tcp    111  portmapper
      10    3   tcp    111  portmapper
      10    2   tcp    111  portmapper
      10    4   udp    111  portmapper
      10    3   udp    111  portmapper
      10    2   udp    111  portmapper
      15    3   tcp  38465  mountd
      15    1   tcp  38466  mountd
      13    3   tcp  38467  nfs
  
  
  Any ideas where to look for errors?
  

  

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Help with some socket related logwarnings

[David Coulson]

what's in the client logs and the brick logs in /var/log/glusterfs/*log 
and /var/log/glusterfs/bricks/*log


On 2/23/12 4:49 PM, Carl Boberg wrote:

Hello

I have just started to prepare a smallish production setup (nothing 
critical running on it yet).


I have 2 gluster servers with 8 volumes and Im getting a lot of theese 
warnings in the cli.log
[2012-02-23 22:32:15.808271] W 
[rpc-transport.c:606:rpc_transport_load] 0-rpc-transport: missing 
'option transport-type'. defaulting to "socket"


Could be related to the warinings in the etc-glusterfs-glusterd.vol.log
[2012-02-21 17:28:15.279406] W 
[socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
reading from socket failed. Error (Transport endpoint is not 
connected), peer (10.131.139.26:1019 )
[2012-02-21 17:28:15.279521] W 
[socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
reading from socket failed. Error (Transport endpoint is not 
connected), peer (10.131.139.26:1016 )
[2012-02-21 17:28:19.236986] W 
[socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
reading from socket failed. Error (Transport endpoint is not 
connected), peer (10.131.139.25:1013 )
[2012-02-21 17:28:42.660021] W 
[socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
reading from socket failed. Error (Transport endpoint is not 
connected), peer (127.0.0.1:1012 )
[2012-02-21 17:34:58.405575] W 
[socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
reading from socket failed. Error (Transport endpoint is not 
connected), peer (10.131.139.9:1021 )



where 10.131.139.26 and 25 are the 2 servers and 10.131.139.9 is a 
client (other clients ip:s are also showing up)


'gluster peer status' on either server says everything is fine and 
connected and im not experiencing any problems. Im just a little 
nervous about the warnings since they get logged quite often. Would be 
greatful for som insight in the reason for theese warnings.


gluster 3.2.5 rpm install
all clients and servers are RedHat 6.1 using native client

Best regards
---
Carl Boberg
Operations

Memnon Networks AB
Tegnérgatan 34, SE-113 59 Stockholm

Mobile: +46(0)70 467 27 12
www.memnonnetworks.com 



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Would difference in size (and content) of a file on replicated bricks be healed?

[David Coulson]

On 2/5/12 4:12 PM, Ove K. Pettersen wrote:

Hi...

Started playing with gluster. And the heal functions is my "target" 
for testing.


Short description of my test

* 4 replicas on single machine
* glusterfs mounted locally
* Create file on glusterfs-mounted directory: date >data.txt

Usually it is bad to write directly to the brick.


* Append to file on one of the bricks: hostname >>data.txt
* Trigger a self-heal with: stat data.txt
=> Modified file on the brick is still different from the three other 
bricks.

What a surprise :-)


Q1: Is the modified file supposed to get corrected in my case?

Q2: Is my test-case invalid for what gluster is supposed to handle?
Probably, since the file has no gfid attributes, gluster has no clue 
what to do with it. If you delete it, it should self-heal.


I would be very happy to receive an opinion on this.

All the details are put at the bottom


Regards,

Ove






Details and logs

My environment
* single machine with OpenSuse 12.1
* gluster 3.3beta2
* xfs is used for the brick-fs (each one to a separate disk)

Volume Name: d1
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: glusterdev:/b1
Brick2: glusterdev:/b2
Brick3: glusterdev:/b3
Brick4: glusterdev:/b4
Options Reconfigured:
diagnostics.brick-log-level: DEBUG
diagnostics.client-log-level: DEBUG

* My glustervolume is mountet as glusterfs on /d1
glusterdev:/d1   52403200   33024  52370176   1% /d1



1) I put a file on the glusterfs
date >/d1/data.txt

2) Checking the storage on the bricks: ls -l /b1 /b2 /b3 /b4
/b1:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b2:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b3:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b4:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt


Everything is OK.

3) Then append a line on to /b1/data.txt
hostname >>/b1/data.txt

By just appending, I keep the inode and all the extended attributes 
(checked with attr -l on the /b1/data.txt file).



4) Check again: ls -l /b1 /b2 /b3 /b4
/b1:
total 8
-rw-r--r-- 1 root root 40 Feb  5 21:40 data.txt

/b2:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b3:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b4:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

Now we have a mismatch between the bricks. Both in contents and also 
the size of the file.


4) Trigger a self-heal by using stat
cd /d1 ; stat data.txt

  File: `data.txt'
  Size: 29  Blocks: 16 IO Block: 131072 regular file
Device: 1fh/31d Inode: 18446744070822815228  Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2012-02-05 21:40:32.382157697 +0100
Modify: 2012-02-05 21:40:32.388157658 +0100
Change: 2012-02-05 21:40:32.389157657 +0100
 Birth: -


5) stat gives the correct size of 29 which was the size of the file 
stored on the glusterfs.

6) DEBUG-log from the client when running this stat-command below.

7) Check again the bricks: ls -l /b1 /b2 /b3 /b4
/b1:
total 8
-rw-r--r-- 1 root root 40 Feb  5 21:40 data.txt

/b2:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b3:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt

/b4:
total 8
-rw-r--r-- 1 root root 29 Feb  5 21:40 data.txt


8) Brick /b1 still has a bad copy of the file.



In the logs it looks like the difference in size is detected.
It also starts some self-healing actions.
But then for some reason finds out no healing is needed.
Marked a few lines with <  which looks suspect.

===CUT Client Log
[2012-02-05 21:41:01.206102] D 
[client-handshake.c:179:client_start_ping] 0-d1-client-0: returning as 
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:01.206201] D 
[client-handshake.c:179:client_start_ping] 0-d1-client-1: returning as 
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:01.206224] D 
[client-handshake.c:179:client_start_ping] 0-d1-client-2: returning as 
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:01.206241] D 
[client-handshake.c:179:client_start_ping] 0-d1-client-3: returning as 
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:03.279124] D 
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279182] D 
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279202] D 
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279218] D 
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix] 
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279232] D 
[afr-self-heal-common.c:604:afr_find_child_character_type] 
0-d1-replicate-0: child 0 character inno

Re: [Gluster-users] Hanging writes after upgrading "clients" to debian squeeze

[David Coulson]

Can you post the client logs also? There should be a filename 
corresponding to the mountpoint of the gluster volume on the client.


Since you are running a replicate volume, you could try shutting down 
gluster on each of the servers in turn and seeing if the write block 
only occurs on one of them.


Did fuse get updated as part of your debian update? Perhaps you are 
hitting a fuse bug? Do you have another system running the previous 
version of debian you were using to validate it can coonnect/write to 
the gluster volume properly? Did you recompile your gluster fuse 
libraries following the update, so they are built against the version of 
fuse you are running?


On 2/5/12 3:49 PM, Stefan Becker wrote:

- no ip tables involved
- the server is running 3.2.0 as well, of course I could upgrade but this 
probably means some downtime which I can not afford right now
- did not find something in the logs, but there are a lot of files so I might 
miss something, logs on the client or server side?
- some debug flags or verbose logging possible?

the bricks log on the server side just says "client connected" so there is not 
a lot of value in that. On the client side I have the following:

[2012-02-05 19:38:53.324172] I [fuse-bridge.c:3214:fuse_thread_proc] 0-fuse: 
unmounting /home/XXXstorage
[2012-02-05 19:38:53.324221] I [glusterfsd.c:712:cleanup_and_exit] 
0-glusterfsd: shutting down
[2012-02-05 19:38:58.709783] W [write-behind.c:3023:init] 
0-XXXstorage-write-behind: disabling write-behind for first 0 bytes
[2012-02-05 19:38:58.711289] I [client.c:1935:notify] 0-XXXstorage-client-0: 
parent translators are ready, attempting connect on transport
[2012-02-05 19:38:58.711489] I [client.c:1935:notify] 0-XXXstorage-client-1: 
parent translators are ready, attempting connect on transport
Given volfile:
+--+
   1: volume XXXstorage-client-0
   2: type protocol/client
   3: option remote-host 10.10.100.40
   4: option remote-subvolume /brick1
   5: option transport-type tcp
   6: end-volume
   7:
   8: volume XXXstorage-client-1
   9: type protocol/client
  10: option remote-host 10.10.100.41
  11: option remote-subvolume /brick1
  12: option transport-type tcp
  13: end-volume
  14:
  15: volume XXXstorage-replicate-0
  16: type cluster/replicate
  17: subvolumes XXXstorage-client-0 XXXstorage-client-1
  18: end-volume
  19:
  20: volume XXXstorage-write-behind
  21: type performance/write-behind
  22: subvolumes XXXstorage-replicate-0
  23: end-volume
  24:
  25: volume XXXstorage-read-ahead
  26: type performance/read-ahead
  27: subvolumes XXXstorage-write-behind
  28: end-volume
  29:
  30: volume XXXstorage-io-cache
  31: type performance/io-cache
  32: subvolumes XXXstorage-read-ahead
  33: end-volume
  34:
  35: volume XXXstorage-stat-prefetch
  36: type performance/stat-prefetch
  37: subvolumes XXXstorage-io-cache
  38: end-volume
  39:
  40: volume XXXstorage
  41: type debug/io-stats
  42: option latency-measurement off
  43: option count-fop-hits off
  44: subvolumes XXXstorage-stat-prefetch
  45: end-volume

+--+
[2012-02-05 19:38:58.712460] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 
0-XXXstorage-client-1: changing port to 24015 (from 0)
[2012-02-05 19:38:58.712527] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 
0-XXXstorage-client-0: changing port to 24012 (from 0)
[2012-02-05 19:39:02.709882] I 
[client-handshake.c:1080:select_server_supported_programs] 
0-XXXstorage-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version 
(310)
[2012-02-05 19:39:02.710112] I 
[client-handshake.c:1080:select_server_supported_programs] 
0-XXXstorage-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version 
(310)
[2012-02-05 19:39:02.710355] I [client-handshake.c:913:client_setvolume_cbk] 
0-XXXstorage-client-1: Connected to 10.10.100.41:24015, attached to remote 
volume '/brick1'.
[2012-02-05 19:39:02.710395] I [afr-common.c:2514:afr_notify] 
0-XXXstorage-replicate-0: Subvolume 'XXXstorage-client-1' came back up; going 
online.
[2012-02-05 19:39:02.712314] I [fuse-bridge.c:3316:fuse_graph_setup] 0-fuse: 
switched to graph 0
[2012-02-05 19:39:02.712387] I [client-handshake.c:913:client_setvolume_cbk] 
0-XXXstorage-client-0: Connected to 10.10.100.40:24012, attached to remote 
volume '/brick1'.
[2012-02-05 19:39:02.712436] I [fuse-bridge.c:2897:fuse_init] 0-glusterfs-fuse: 
FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2012-02-05 19:39:02.713253] I [afr-common.c:836:afr_fresh_lookup_cbk] 
0-XXXstorage-replicate-0: added root inode

I cannot see any problems. I was tailing a few logs while I issued a write 
which hangs. Nothing gets logged.

-Ursprüngliche Nachricht-
Von: Whit Blauvelt [mailto:whit.glus...@transpect.com]
Gesendet: Sonntag, 5. Februar 201

Re: [Gluster-users] Hanging writes after upgrading "clients" to debian squeeze

[David Coulson]

On 2/5/12 2:09 PM, Stefan Becker wrote:


On the webservers I played around with versions up to 3.2.5, nothing 
helps. On the storage server such an upgrade will not be that easy :)


What version of Gluster are the storage servers running? I don't believe 
there is much work involved in upgrading from 3.2.0 to 3.2.5, and since 
you're got your client side broke it's probably a good time to try it 
anyway.


You'll probably get more productive/useful help if you are running the 
most current stable release of the Gluster code and can still reproduce 
the issue.


100% sure there is nothing in either the client or brick logs on any of 
the systems?


David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Hanging writes after upgrading "clients" to debian squeeze

[David Coulson]

I would start by updating to 3.2.5 and see if you still have the same issue.

On 2/5/12 2:02 PM, Stefan Becker wrote:


Hi,

I am running a simple gluster storage server setup with 2 nodes, one 
volume with two bricks and replication. My "clients" are webservers 
which mount the gluster storage via native client. My storage server 
run debian squeeze for some while now and today I upgraded my 
webservers from debian lenny to debian squeeze. The gluster version I 
am running is 3.2.0, self compiled. After the debian upgrade I can 
still mount my volumes. Reading is fine as well but it hangs on 
writes. I did not touch the storage servers at all. I did not upgrade 
all webservers so I still have some running with debian lenny and 
everything works just fine. The log files are rather empty, at least I 
cannot find anything during the hanging write.


Do you guys have any idea how to fix this?

Regards,

Stefan



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Switch recommendations

[David Coulson]

On 1/30/12 9:12 PM, Harald Stürzebecher wrote:


What happens if one switch fails? Can the remaining equipment do some
useful work until a replacement arrives?
Depending on the answers it might be better to have two or more
smaller stackable switches instead of one big switch, even if the big
one might be cheaper.
Or just run two totally independent switches and either support two 
different L3 networks, or setup bonded interfaces which connected to 
each switch. If you lose a switch, then you can continue to function 
without a service interruption.



I just looked at the Netgear website and googled for prices:
Two Netgear GS724TS stackable switches seem to cost nearly the same as
one GS748TS, both are supposed to have "a 20 Gbps, dual-ring, highly
redundant stacking bus".


The issue with most stacking switches is that you don't have any real 
redundancy without either doubling up on ports so each server is 
multi-homed into multiple switches, or having to manually move ports off 
a dead switch. Typically with a stacking configuration if you lose the 
'master' switch the whole stack has to reboot which causes an outage.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] fc storage examples

[David Coulson]

Just to be clear, you are mounting the same LUN on both servers, or 
providing two different LUNs, one to each box, then running gluster on 
top? What do you mean by 'multipathing'? Usually that indicates you have 
2 or more FC paths to the same LUN or target.


If you are mounting the same LUN, you will need a clustered volume 
manager and filesystem (CLVMD + GFS2 or OCFS2). Gluster would probably 
have a lot of issues working in this configuration, since it doesn't 
know that the two bricks comprising your volume are really the same thing.


If you are mounting two different LUNs, then you can just mkfs them with 
ext4 or xfs and configure your gluster volume. As part of your create 
command, you will need to specify the type of volume (replica, 
distributed, striped, or some combination of the two).


David

On 1/30/12 5:05 AM, Wojciech Giel wrote:

Hello,
I will have two servers with fc storage. storage will be connected 
with two links to both servers

___   ___
| server 1 || server 2 |
----
  |  | |  |
 --
 |storage  |
 --
on servers i will  have multipathing enabled. storage has only one 
volume and this volume will be mounted under /mnt/data. I want to 
enable glusterfs. How should I do it:

is it correct?

gluster volume create transport tcp server1:/mnt/data server2:/mnt/data

manuals don't shows examples with directly connected multi-home 
storage configurations.

Could you give some advice , best practice
thanks
Wojciech



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Large number of 'no gfid found' messages in log

[David Coulson]

I am periodically seeing a high number of these messages in the client 
log - Nothing in the log for the bricks. There appears to be a log entry 
for every file in that directory, including sub-directories. I check 
getfattr on the bricks and they have the gfid set and both replica brick 
gfid's match for each file.


[2012-01-26 15:58:16.590368] W 
[fuse-resolve.c:273:fuse_resolve_deep_cbk] 0-fuse: /plugins/search.jar: 
no gfid found


Is this a bug, or just a log message about something which was 'fixed' 
automatically?


David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] remote operation failed: Stale NFS file handle

[David Coulson]

I've got a pretty simple Gluster 3.2.5 implementation on RHEL6. One 
client keeps complaining about 'Stale NFS file handle', even though I 
have explicitly disabled NFS for all volumes in the environment. I've 
got 7 volumes on the cluster, and this is the only one actively 
complaining about the issue. For what it is worth, the clients and 
servers are the same systems. Any ideas? I saw a few other people 
reporting the same issue with earlier releases of Gluster, but did not 
see a solution or further troubleshooting steps.


[2012-01-24 20:57:22.765190] I [fuse-bridge.c:3339:fuse_graph_setup] 
0-fuse: switched to graph 0
[2012-01-24 20:57:22.765314] I [fuse-bridge.c:2927:fuse_init] 
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 
kernel 7.13
[2012-01-24 20:57:22.765710] I 
[afr-common.c:1520:afr_set_root_inode_on_first_lookup] 
0-openfire-replicate-0: added root inode
[2012-01-24 22:15:44.984871] E 
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-openfire-client-0: remote 
operation failed: Stale NFS file handle
[2012-01-24 22:15:44.985228] E 
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-openfire-client-1: remote 
operation failed: Stale NFS file handle


# gluster volume info openfire

Volume Name: openfire
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: rhesproddns01:/gluster/openfire
Brick2: rhesproddns02:/gluster/openfire
Options Reconfigured:
nfs.register-with-portmap: off
nfs.disable: on

Thanks-
David
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users