Re: [Gluster-users] Redux: Stale NFS file handle

2013-01-04 Thread harry mangalam 
Following up on myself..

There is a RH bug report  
that sounds like what I'm seeing, but it's associated with the NFS client.  
We're using the gluster native client, so it shouldn't be related, but it 
looks like there may be some NFS client code that was used in the gluster 
native client code as well (the cause of the 'Stale NFS handle' warning in a 
non-NFS setting). 

Could this be the case?  Could the pseudo-solution for the NFS case also work 
for the gluster native client case?  ie: mount -o acdirmin=[23] ...

It doesn't seem like 'acdirmin'  is a mount option for glusterfs but is there 
a gluster equivalent?

Also, this bug  that 
complains about the the same thing as I reference below has this notation from 
Amar:
---
..this issue ['Stale NFS handle'] can happen in the case where file is changed 
on server after it was being accessed from one node (changed as in, re-
created). as long as the application doesn't see any issue, this should be ok.
---

In our case, it's causing the SGE array job to fail (or at least it appears to 
be highly related):
=
SGE error extract:

Shepherd error:
01/03/2013 20:03:03 [507:64196]: error: can't chdir to 
/gl/bio/krthornt/build_div/yak/line10_CY22B/prinses: No such file or directory
=
glusterfs log on client (native gluster client):

FS file handle. Path: /bio/krthornt/build_div/yak/line10_CY22B/prinses 
(590802f1-7fba-4103-ba30-e4d415b9db36)
[2013-01-03 20:03:03.168229] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-
gl-client-1: remote operation failed: Stale NFS file handle. Path: 
/bio/krthornt/build_div/yak/line10_CY22B/prinses (590802f1-7fba-4103-ba30-
e4d415b9db36)
[2013-01-03 20:03:03.168287] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-
gl-client-2: remote operation failed: Stale NFS file handle. Path: 
/bio/krthornt/build_div/yak/line10_CY22B/prinses (590802f1-7fba-4103-ba30-
e4d415b9db36)

(and 13 more identical lines except for the timestamp (extending out to 
'20:03:17.202688').

so while we might not be losing data /per se/, this failure is definitely 
causing our cluster to lose jobs.

If this has not been reported previously, I'd be happy to file an official bug 
report.

hjm


On Thursday, January 03, 2013 05:56:10 PM harry mangalam wrote:
> From ~ an hour's googling and reading, it looks like this (not uncommon)
> bug/warning/error has not necessarily been associated with data loss, but we
> are finding that our gluster fs is interrupting our cluster jobs with the
> 'Stale NFS handle' Warnings like this (on the client):
> 
> [2013-01-03 12:30:59.149230] W [client3_1-fops.c:2630:client3_1_lookup_cbk]
> 0- gl-client-0: remote operation failed: Stale NFS file handle. Path:
> /bio/krthornt/build_div/yak/line06_CY08A/prinses (3b0aa7b2-bf7f-4b27-
> b515-32e94b1206e3)
> 
> (and 7 more, differing by the timestamp of <<1s).
> 
> The dir mentioned existed before the job was asked to read from it and
> shortly after the SGE failed, I checked that the glusterfs (/bio) was still
> mounted and that dir was still r/w.
> 
> We are getting these errors infrequently, but fairly regularly (a couple
> times a week, usually during a big array job that heavily reads from a
> particular dir) and I haven't seen any resolutions of the fault besides the
> vocabulary being corrected.  I know it's not nec an NFS problem, but I
> haven't seen a fix from the gluster folks.
> 
> Our glusterfs on this system is set up like this (over QDR/tcpoib)
> 
> $ gluster volume info gl
> 
> Volume Name: gl
> Type: Distribute
> Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332
> Status: Started
> Number of Bricks: 8
> Transport-type: tcp,rdma
> Bricks:
> Brick1: bs2:/raid1
> Brick2: bs2:/raid2
> Brick3: bs3:/raid1
> Brick4: bs3:/raid2
> Brick5: bs4:/raid1
> Brick6: bs4:/raid2
> Brick7: bs1:/raid1
> Brick8: bs1:/raid2
> Options Reconfigured:
> auth.allow: 10.2.*.*,10.1.*.*
> performance.io-thread-count: 64
> performance.quick-read: on
> performance.io-cache: on
> nfs.disable: on
> performance.cache-size: 268435456
> performance.flush-behind: on
> performance.write-behind-window-size: 1024MB
> 
> and otherwise appears to be happy.
> 
> We were having a low-level problem with the RAID servers, where this
> LSI/3ware error was temporally close (~2m) to the gluster error:
> 
> LSI 3DM2 alert -- host: biostor4.oit.uci.edu
> Jan 03, 2013 03:32:09PM - Controller 6
> ERROR - Drive timeout detected: encl=1, slot=3
> 
> This error seemed to be related to construction around our data center and
> dust related with it.  We have had 10s of these LSI/3ware errors with no
> related gluster errors or apparent problems with the RAIDs.  No drives were
> ejected from the RAIDs and the errors did not repeat.  3ware explains:
> 
> ==
> 009h Drive timeout detected
> 
> The 3ware RAID controller has a

[Gluster-users] Redux: Stale NFS file handle

2013-01-03 Thread harry mangalam 

>From ~ an hour's googling and reading, it looks like this (not uncommon) 
bug/warning/error has not necessarily been associated with data loss, but we 
are finding that our gluster fs is interrupting our cluster jobs with the 
'Stale NFS handle' Warnings like this (on the client):

[2013-01-03 12:30:59.149230] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-
gl-client-0: remote operation failed: Stale NFS file handle. Path: 
/bio/krthornt/build_div/yak/line06_CY08A/prinses (3b0aa7b2-bf7f-4b27-
b515-32e94b1206e3)

(and 7 more, differing by the timestamp of <<1s).

The dir mentioned existed before the job was asked to read from it and shortly 
after the SGE failed, I checked that the glusterfs (/bio) was still mounted 
and that dir was still r/w.

We are getting these errors infrequently, but fairly regularly (a couple times 
a week, usually during a big array job that heavily reads from a particular 
dir) and I haven't seen any resolutions of the fault besides the vocabulary 
being corrected.  I know it's not nec an NFS problem, but I haven't seen a fix 
from the gluster folks.

Our glusterfs on this system is set up like this (over QDR/tcpoib)

$ gluster volume info gl
 
Volume Name: gl
Type: Distribute
Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332
Status: Started
Number of Bricks: 8
Transport-type: tcp,rdma
Bricks:
Brick1: bs2:/raid1
Brick2: bs2:/raid2
Brick3: bs3:/raid1
Brick4: bs3:/raid2
Brick5: bs4:/raid1
Brick6: bs4:/raid2
Brick7: bs1:/raid1
Brick8: bs1:/raid2
Options Reconfigured:
auth.allow: 10.2.*.*,10.1.*.*
performance.io-thread-count: 64
performance.quick-read: on
performance.io-cache: on
nfs.disable: on
performance.cache-size: 268435456
performance.flush-behind: on
performance.write-behind-window-size: 1024MB

and otherwise appears to be happy.  

We were having a low-level problem with the RAID servers, where this LSI/3ware 
error was temporally close (~2m) to the gluster error:

LSI 3DM2 alert -- host: biostor4.oit.uci.edu
Jan 03, 2013 03:32:09PM - Controller 6
ERROR - Drive timeout detected: encl=1, slot=3

This error seemed to be related to construction around our data center and 
dust related with it.  We have had 10s of these LSI/3ware errors with no 
related gluster errors or apparent problems with the RAIDs.  No drives were 
ejected from the RAIDs and the errors did not repeat.  3ware explains:

==
009h Drive timeout detected

The 3ware RAID controller has a sophisticated recovery mechanism to handle 
various types of failures of a disk drive. One such possible failure of a disk 
drive is a failure of a command that is pending from the 3ware RAID controller 
to complete within a reasonable amount of time. If the 3ware RAID controller 
detects this condition, it notifies the user, prior to entering the recovery 
phase, by displaying this AEN.

Possible causes of APORT time-outs include a bad or intermittent disk drive, 
power cable or interface cable.

We've checked into this and it doesn't seem to be related, but I thought I'd 
bring it up.

hjm





On Thursday, August 23, 2012 09:54:13 PM Joe Julian wrote:
> *Bug 832694* 
> -ESTALE error text should be reworded
> 
> On 08/23/2012 09:50 PM, Kaushal M wrote:
> > The "Stale NFS file handle" message is the default string given by
> > strerror() for errno ESTALE.
> > Gluster uses ESTALE as errno to indicate that the file being referred
> > to no longer exists, ie. the reference is stale.
> > 
> > - Kaushal
> > 
> > On Fri, Aug 24, 2012 at 7:03 AM, Jules Wang  > 
> > > wrote:
> > Hi, Jon ,
> > 
> > I also met the same issue, and reported a
> > 
> > bug(https://bugzilla.redhat.com/show_bug.cgi?id=851381)
> > 
> > 
> > In the bug report, I share a simple way to reproduce the bug.
> > Have fun.
> > 
> > Best Regards.
> > Jules Wang.
> > 
> > At 2012-08-23 23:02:34,"Bùi Hùng Vie^.t"  > 
> > > wrote:
> > Hi Jon,
> > I have no answer for you. Just want to share with you guys
> > that I met the same issue with this message. In my gluster
> > system, Gluster client log files have a lot of these messages.
> > I tried to ask and found nothing on the Web. Amazingly,
> > Gluster have been running for long time :)
> > 
> > On Thu, Aug 23, 2012 at 8:43 PM, Jon Tegner  > 
> > > wrote:
> > Hi, I'm a bit curious of error messages of the type
> > "remote operation failed: Stale NFS file handle". All
> > clients using the file system use Gluster Native Client,
> > so why should stale nfs file handle be reported?
> > 
> > Reg