Re: [Gluster-users] Permission denied after upgrade and rebalancing

[Jochen Lillich]

Hi Amar,

Thanks for your prompt answer!

On Mon, Oct 17, 2011 at 6:52 PM, Amar Tumballi  wrote:

> Can you try running 'gluster volume rebalance  fix-layout start'
> again?

# gluster volume rebalance glfs fix-layout start
starting rebalance on volume glfs has been successful
# gluster volume rebalance glfs fix-layout status
rebalance step 1: layout fix complete: fixed layout 73
# gluster volume info glfs

Volume Name: glfs
Type: Distribute
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: backup01.topcluster.net:/vol1/glfs
Brick2: backup02.topcluster.net:/vol1/glfs
Brick3: backup03.topcluster.net:/vol1/glfs

On backup01, where I executed the fix-layout, I don't find any
suspicious log entries, only the "lookup-unhashed" option that
probably is necessary for this maintenance job.

> Also, are you using ACLs? can the accessing of directories/files work as
> 'root' user?

No, we don't use ACLs. I've been using the root user all the time, but
even then the usual "find ... stat" self-healing command triggers the
"permission denied" errors.

And it still does... :-(

Thanks again for your help!

Best regards,
 Jochen
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Permission denied after upgrade and rebalancing

[Amar Tumballi]

Hi Jochen,

Can you try running 'gluster volume rebalance  fix-layout start'
again?

Seems that some of the layout is not valid. Also, make sure all your bricks
are up and running.

Also, are you using ACLs? can the accessing of directories/files work as
'root' user?

Regards,
Amar

On Mon, Oct 17, 2011 at 6:29 PM, Jochen Lillich  wrote:

> Hi,
>
> when our two-node distributed backup storage cluster, running on
> GlusterFS 3.0.2 (Ubuntu Lucid packages), started running out of space,
> I added a third node and tried rebalancing them. The scale-n-defrag
> script didn't work, though. I always got a "stale NFS handle".
>
> So I decided to upgrade and installed the 3.2.4 packages from
> semiosis' Launchpad PPA, followed the upgrade guide for 3.0 -> 3.2 and
> then started "gluster volume rebalance". After two days, the three
> nodes are now about equally full, so the rebalancing worked.
>
> Unfortunately, it seems that it also broke the cluster filesystem. We
> get a lot of "permission denied" errors although file system
> permissions are correct. The log contains a lot of these entries:
>
> [2011-10-17 10:11:48.979801] I
> [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
> operation failed: Permission denied
> [2011-10-17 10:11:48.979892] I [dht-common.c:478:dht_revalidate_cbk]
> 0-glfs-dht: subvolume glfs-client-2 for /bacula/mail02 returned -1
> (Permission denied)
> [2011-10-17 10:11:48.980216] I [dht-layout.c:192:dht_layout_search]
> 0-glfs-dht: no subvolume for hash (value) = 1486987642
> [2011-10-17 10:11:48.981101] I
> [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
> operation failed: Permission denied
> [2011-10-17 10:11:48.981581] I [dht-layout.c:192:dht_layout_search]
> 0-glfs-dht: no subvolume for hash (value) = 1486987642
> [2011-10-17 10:11:48.982468] I
> [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
> operation failed: Permission denied
>
> How can I repair this?
>
> Best regards,
>  Jochen
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Need help with optimizing GlusterFS for Apache

[Robert Krig]

Our webserver is configured as such:

The actual website files, php, html ,css and so on. Or on a dedicated
non-glusterfs ext4 partition.
However, the website access Videos and especially image files on a
gluster mounted directory.

The write performance for our backend gluster storage is not that
important. Since it only comes into play when someone uploads a video or
image.
However, the files are being read from constantly and randomly.
Especially all the small image files which get read everytime someone
loads our website.

It is quite a busy site, with somewhere along the line of 500 apache
accesses per second during our busy hours.

I've been taking a look at performance.cache-size and translators. But
I'm not quite sure what the ideal settings would be.

Can anyone help me out with some settings to try out, based on
experience they've had?

Thanks.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Problems with running long jobs on a replicated volume.

[Peter Linder]

Perhaps it is similar to the problem I have, see: 
http://bugs.gluster.com/show_bug.cgi?id=3712


I will try perhaps tonight to leave my find command running and see if 
that eventually breaks the mount point.


On 10/17/2011 4:11 PM, Tiago Carmona wrote:
First of all, hi guys. My name is Tiago Carmona and I'm a DevOps to be 
at Unicamp in Brazil. I started using glusterFS not a long time ago, 
but I'm loving it. I also would like to say thanks for all the help 
I've got on IRC.


I'm having a problem with running long jobs on a replicated volume. 
When I run a long job (like a chmod -R on my mount root), I got many 
"NFS stale handler" errors, and after some time my mount point is down 
with a "Transport endpoint is not connected" error, so I need to 
umount and mount it again. I think that my error is similar to the one 
at http://gluster.org/pipermail/gluster-users/2011-April/007192.html , 
from this list. Does anyone know what may be causing this?


I'm running glusterfs on two gentoo machines. Version info bellow:

glusterfs 3.2.3 built on Sep  4 2011 10:12:37
Repository revision: git://git.gluster.com/glusterfs.git 


Copyright (c) 2006-2011 Gluster Inc. 

Many thanks for all,
Tiago Carmona


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Problems with running long jobs on a replicated volume.

[Tiago Carmona]

First of all, hi guys. My name is Tiago Carmona and I'm a DevOps to be at
Unicamp in Brazil. I started using glusterFS not a long time ago, but I'm
loving it. I also would like to say thanks for all the help I've got on IRC.

I'm having a problem with running long jobs on a replicated volume. When I
run a long job (like a chmod -R on my mount root), I got many "NFS stale
handler" errors, and after some time my mount point is down with a
"Transport endpoint is not connected" error, so I need to umount and mount
it again. I think that my error is similar to the one at
http://gluster.org/pipermail/gluster-users/2011-April/007192.html , from
this list. Does anyone know what may be causing this?

I'm running glusterfs on two gentoo machines. Version info bellow:

glusterfs 3.2.3 built on Sep  4 2011 10:12:37
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. 

Many thanks for all,
Tiago Carmona
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

[Gluster-users] Permission denied after upgrade and rebalancing

[Jochen Lillich]

Hi,

when our two-node distributed backup storage cluster, running on
GlusterFS 3.0.2 (Ubuntu Lucid packages), started running out of space,
I added a third node and tried rebalancing them. The scale-n-defrag
script didn't work, though. I always got a "stale NFS handle".

So I decided to upgrade and installed the 3.2.4 packages from
semiosis' Launchpad PPA, followed the upgrade guide for 3.0 -> 3.2 and
then started "gluster volume rebalance". After two days, the three
nodes are now about equally full, so the rebalancing worked.

Unfortunately, it seems that it also broke the cluster filesystem. We
get a lot of "permission denied" errors although file system
permissions are correct. The log contains a lot of these entries:

[2011-10-17 10:11:48.979801] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied
[2011-10-17 10:11:48.979892] I [dht-common.c:478:dht_revalidate_cbk]
0-glfs-dht: subvolume glfs-client-2 for /bacula/mail02 returned -1
(Permission denied)
[2011-10-17 10:11:48.980216] I [dht-layout.c:192:dht_layout_search]
0-glfs-dht: no subvolume for hash (value) = 1486987642
[2011-10-17 10:11:48.981101] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied
[2011-10-17 10:11:48.981581] I [dht-layout.c:192:dht_layout_search]
0-glfs-dht: no subvolume for hash (value) = 1486987642
[2011-10-17 10:11:48.982468] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied

How can I repair this?

Best regards,
 Jochen
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] (3.2.4-1) many "Stopping gluster nfsd running in pid: " in log

[Krishna Srinivas]

Tomo, I have logged an enhancement bug for this for now:
http://bugs.gluster.com/show_bug.cgi?id=3725

You can add yourself to the CC list to receive updates.

Regards,
Krishna

On Fri, Oct 14, 2011 at 2:49 PM, Tomoaki Sato  wrote:
> not 'exportfs -f' but 'exportfs -r'. sorry.
>
> (2011/10/14 18:17), Tomoaki Sato wrote:
>>
>> Krishna,
>>
>> Thank you for your comments.
>> I've changed the script not to repeat gluster volume set command with same
>> arguments.
>> Do you have any plans to make gluster restart-free like 'exportfs -f' of
>> nfs.
>>
>> tomo sato
>>
>> (2011/10/14 18:03), Krishna Srinivas wrote:
>>>
>>> Hi Tomo Sato,
>>>
>>> Using gluster volume set command will restart the nfs server, hence
>>> you should change the script so that it does not restart the nfs
>>> server too often.
>>>
>>> You can consult with the person who installed the script as it is not
>>> a part of gluster installed scripts.
>>>
>>> Krishna
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users