[Nfs-ganesha-devel] Device or resource busy when runltp cleanup test-files
When I testing ganesha nfs bases on glusterfs, the runltp always warning as,
rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea14f5’:
Device or resource busy
rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory not empty
and, "rmderQsjV" also contains files at the back-end, and nfs client shows
empty.
My test environments are,
Centos 7 (kernel-3.10.0-514.10.2.el7.x86_64),
Glusterfs (glusterfs-3.8.10-1.el7.x86_64),
NFS-Ganesha (nfs-ganesha-2.3.3-1.el7.x86_64).
#cat /etc/ganesha/ganesha.conf
EXPORT
{
SecType = "sys";
Pseudo = "/gvtest";
Squash = "No_Root_Squash";
Access_Type = "RW";
Path = "/gvtest";
Export_Id = 1;
FSAL {
Name = "GLUSTER";
Hostname = "localhost";
Volume = "gvtest";
Volpath = "/";
}
}
# gluster volume info
Volume Name: gvtest
Type: Distribute
Volume ID: 65d20de1-16cd-4ae8-a860-254b3d6c56d0
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 192.168.9.111:/gluster-test/gvtest
Brick2: 192.168.9.112:/gluster-test/gvtest
Options Reconfigured:
nfs.disable: on
performance.readdir-ahead: off
transport.address-family: inet
performance.write-behind: off
performance.read-ahead: off
performance.io-cache: off
performance.quick-read: off
performance.open-behind: off
performance.stat-prefetch: off
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Device or resource busy when runltp cleanup test-files
There are some files under "rmderQsjV" (that's not a silly rename dir) really
at underlying filesystem, but the nfs client shows empty.
Are there some problems in MDCACHE or cache timeouts?
On 4/12/2017 22:48, Malahal Naineni wrote:
> Could be due to NFS client silly rename.
>
> On Apr 12, 2017 8:06 PM, "Kinglong Mee" <mailto:mijinl...@open-fs.com>> wrote:
>
> When I testing ganesha nfs bases on glusterfs, the runltp always warning
> as,
>
> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea14f5’:
> Device or resource busy
> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory not empty
>
> and, "rmderQsjV" also contains files at the back-end, and nfs client
> shows empty.
>
> My test environments are,
> Centos 7 (kernel-3.10.0-514.10.2.el7.x86_64),
> Glusterfs (glusterfs-3.8.10-1.el7.x86_64),
> NFS-Ganesha (nfs-ganesha-2.3.3-1.el7.x86_64).
>
> #cat /etc/ganesha/ganesha.conf
> EXPORT
> {
> SecType = "sys";
> Pseudo = "/gvtest";
> Squash = "No_Root_Squash";
> Access_Type = "RW";
> Path = "/gvtest";
> Export_Id = 1;
> FSAL {
> Name = "GLUSTER";
> Hostname = "localhost";
> Volume = "gvtest";
> Volpath = "/";
> }
> }
>
> # gluster volume info
>
> Volume Name: gvtest
> Type: Distribute
> Volume ID: 65d20de1-16cd-4ae8-a860-254b3d6c56d0
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.9.111:/gluster-test/gvtest
> Brick2: 192.168.9.112:/gluster-test/gvtest
> Options Reconfigured:
> nfs.disable: on
> performance.readdir-ahead: off
> transport.address-family: inet
> performance.write-behind: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.quick-read: off
> performance.open-behind: off
> performance.stat-prefetch: off
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> <mailto:Nfs-ganesha-devel@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
> <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel>
>
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>
>
>
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Device or resource busy when runltp cleanup test-files
Yes, this one is silly rename,
>>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea14f5’:
>>> Device or resource busy
But, the other one isn't client's silly rename,
>>>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory not
>>>> empty
Also, the second one often appears when testing ganesha nfs by ltp.
I try to get an tcpdump, I found, when ls under the nfs client directory,
the glusterfs client doesn't send readdir to glusterfs server, so that,
the nfs client gets an empty directory information, but it's empty underlying.
ls at some other directory,
the glusterfs client send readdir to glusterfs server, nfs client gets right
dirctory
information as the underlying.
So, I guess maybe there are some problems in MDCHANE or glusterfs client?
]# cat /var/lib/glusterd/vols/gvtest/trusted-gvtest.tcp-fuse.vol
volume gvtest-client-0
type protocol/client
option send-gids true
option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd
option username 7bc6276f-245c-477a-99d0-7ead4fcb2968
option transport.address-family inet
option transport-type tcp
option remote-subvolume /gluster-test/gvtest
option remote-host 192.168.9.111
option ping-timeout 42
end-volume
volume gvtest-client-1
type protocol/client
option send-gids true
option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd
option username 7bc6276f-245c-477a-99d0-7ead4fcb2968
option transport.address-family inet
option transport-type tcp
option remote-subvolume /gluster-test/gvtest
option remote-host 192.168.9.111
option ping-timeout 42
end-volume
volume gvtest-dht
type cluster/distribute
option lock-migration off
subvolumes gvtest-client-0 gvtest-client-1
end-volume
volume gvtest
type debug/io-stats
option count-fop-hits off
option latency-measurement off
option log-level INFO
subvolumes gvtest-dht
end-volume
thanks,
Kinglong Mee
On 4/13/2017 10:17, Malahal Naineni wrote:
> Sorry, I missed your note below. This is definitely due to an NFS
> client's silly rename. All ".nfs" are due to silly rename
> implementation in NFS client. You might want to read about it.
>
>>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea14f5’:
>>> Device or resource busy
>
> On Thu, Apr 13, 2017 at 7:37 AM, Malahal Naineni wrote:
>> What are the file names under the directory? What does "ls -la" show
>> both at the client and at the server in that directory?
>>
>> On Thu, Apr 13, 2017 at 5:13 AM, Kinglong Mee wrote:
>>> There are some files under "rmderQsjV" (that's not a silly rename dir)
>>> really
>>> at underlying filesystem, but the nfs client shows empty.
>>>
>>> Are there some problems in MDCACHE or cache timeouts?
>>>
>>> On 4/12/2017 22:48, Malahal Naineni wrote:
>>>> Could be due to NFS client silly rename.
>>>>
>>>> On Apr 12, 2017 8:06 PM, "Kinglong Mee" >>> <mailto:mijinl...@open-fs.com>> wrote:
>>>>
>>>> When I testing ganesha nfs bases on glusterfs, the runltp always
>>>> warning as,
>>>>
>>>> rm: cannot remove
>>>> ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea14f5’: Device or resource
>>>> busy
>>>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory not
>>>> empty
>>>>
>>>> and, "rmderQsjV" also contains files at the back-end, and nfs client
>>>> shows empty.
>>>>
>>>> My test environments are,
>>>> Centos 7 (kernel-3.10.0-514.10.2.el7.x86_64),
>>>> Glusterfs (glusterfs-3.8.10-1.el7.x86_64),
>>>> NFS-Ganesha (nfs-ganesha-2.3.3-1.el7.x86_64).
>>>>
>>>> #cat /etc/ganesha/ganesha.conf
>>>> EXPORT
>>>> {
>>>> SecType = "sys";
>>>> Pseudo = "/gvtest";
>>>> Squash = "No_Root_Squash";
>>>> Access_Type = "RW";
>>>> Path = "/gvtest";
>>>> Export_Id = 1;
>>>> FSAL {
>>>> Name = "GLUSTER";
>>>> Hostname = "localhost";
>>>> Volume = "gvtest";
>>>> Volpath = "/";
>>>> }
>>>> }
>>>>
>>>> # gluster volume info
>>>>
>>>> Volume Name: gvtest
>>>> Type: Distribute
>>>> Volume ID: 65d20d
Re: [Nfs-ganesha-devel] [Gluster-devel] Device or resource busy when runltp cleanup test-files
Sorry for the late reply. On 4/16/2017 22:31, Vijay Bellur wrote: > > > On Wed, Apr 12, 2017 at 10:37 PM, Kinglong Mee <mailto:mijinl...@open-fs.com>> wrote: > > Yes, this one is silly rename, > > >>> rm: cannot remove > ‘/mnt/nfs/ltp-JEYAuky2dz/.nfsaa46457a6a72f8ea14f5’: Device or resource > busy > > But, the other one isn't client's silly rename, > > >>>> rm: cannot remove ‘/mnt/nfs/ltp-JEYAuky2dz/rmderQsjV’: Directory > not empty > > Also, the second one often appears when testing ganesha nfs by ltp. > > I try to get an tcpdump, I found, when ls under the nfs client directory, > > the glusterfs client doesn't send readdir to glusterfs server, so that, > the nfs client gets an empty directory information, but it's empty > underlying. > > ls at some other directory, > the glusterfs client send readdir to glusterfs server, nfs client gets > right dirctory > information as the underlying. > > So, I guess maybe there are some problems in MDCHANE or glusterfs client? > > ]# cat /var/lib/glusterd/vols/gvtest/trusted-gvtest.tcp-fuse.vol > volume gvtest-client-0 > type protocol/client > option send-gids true > option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd > option username 7bc6276f-245c-477a-99d0-7ead4fcb2968 > option transport.address-family inet > option transport-type tcp > option remote-subvolume /gluster-test/gvtest > option remote-host 192.168.9.111 > option ping-timeout 42 > end-volume > > volume gvtest-client-1 > type protocol/client > option send-gids true > option password d80765c8-95ae-46ed-912a-0c98fdd1e7cd > option username 7bc6276f-245c-477a-99d0-7ead4fcb2968 > option transport.address-family inet > option transport-type tcp > option remote-subvolume /gluster-test/gvtest > option remote-host 192.168.9.111 > option ping-timeout 42 > end-volume > > volume gvtest-dht > type cluster/distribute > option lock-migration off > subvolumes gvtest-client-0 gvtest-client-1 > end-volume > > volume gvtest > type debug/io-stats > option count-fop-hits off > option latency-measurement off > option log-level INFO > subvolumes gvtest-dht > end-volume > > > > > Have you checked the gluster export directories after receiving ENOTEMPTY? If > you observe any files there, can you please check if they are 0-byte files > with access mode 600 and sticky bit set? > No. I have check those files at the latest test. rm: cannot remove ‘/mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj’: Directory not empty The back-end directory, # ll /gvtest/ltp-fpeUthdR0W/ total 264 drwxrwxrwx 2 root root 253952 Apr 20 15:53 fann7LHNa drwxrwxrwx 2 root root 4096 Apr 20 15:54 inop4q8Wi drwxrwxrwx 3 root root 4096 Apr 20 15:58 linu6YVST drwxrwxrwx 3 root root 4096 Apr 20 15:59 rmdVGrwdj # ll /gvtest/ltp-fpeUthdR0W/rmdVGrwdj/testdir/ total 0 The nfs client directory mounted through ganesha.nfsd, # ll /mnt/nfs/ltp-fpeUthdR0W/ total 12 drwxrwxrwx 2 root root 4096 Apr 20 15:53 fann7LHNa drwxrwxrwx 2 root root 4096 Apr 20 15:54 inop4q8Wi drwxrwxrwx 3 root root 4096 Apr 20 15:58 linu6YVST # ll /mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj/ total 4 drwxrwxrwx 2 root root 4096 Apr 20 15:59 testdir [root@C0N1 ~]# ll /mnt/nfs/ltp-fpeUthdR0W/rmdVGrwdj/testdir/ total 0 -rwxrwxrwx 1 root root 0 Apr 20 15:59 testfile 1. nfs client can't get "rmdVGrwdj" when ll the upper directory, 2. but nfs client can access it by name directly, 3. A "testfile" under nfs client's "rmdVGrwdj/testdir", but it's empty in the back-end gluster filesystem. 4. after ltp's rm finish, a duplicate rm can remove "rmdVGrwdj". So, I guess there are some problem of the ganesha.nfsd cache when deleting files. thanks, Kinglong Mee > When a file is being renamed, if the hash value of its new name happens to > fall in a range that is different than that of the old name, gluster creates > one such file. In the past we have seen instances of these files not being > cleaned up properly in some cases and that causes a ENOTEMPTY when rmdir > lands on the parent directory. Trying to see if we are hitting the same > problem here. > > Regards, > Vijay -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] About the time accuracy of dirent in mdcache
Ganesha invalidates directory dirent in mdcache_getattrs depends on mtime.tv_sec. But the time accuracy is 1s. If there are many creating after one readdir in 1s, how to make sure the dirents in mdcache is synchronous as lower filesystem? I wanna know, can we update it to depends on mtime.tv_usec? Are there some problems or some other considers? thanks, Kinglong Mee -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] About the time accuracy of dirent in mdcache
On 7/18/2017 20:54, Daniel Gryniewicz wrote: > On 07/18/2017 08:51 AM, Daniel Gryniewicz wrote: >> On 07/17/2017 08:10 PM, Kinglong Mee wrote: >>> Ganesha invalidates directory dirent in mdcache_getattrs depends on >>> mtime.tv_sec. >>> But the time accuracy is 1s. >>> >>> If there are many creating after one readdir in 1s, >>> how to make sure the dirents in mdcache is synchronous as lower filesystem? >>> >>> I wanna know, can we update it to depends on mtime.tv_usec? >>> Are there some problems or some other considers? >>> >> >> In mdcache_refresh_attrs(), it uses gsh_time_cmp() to check of the mtime has >> changed. That compares tv_sec and tv_nsec, so it should already be >> comparing to the limit of the filesystem's resolution. >> >> Daniel > > Ah, I see. This has been fixed in 2.5. You should upgrade. Yes, it's fixed. Sorry for my fault of miss checking the latest version. thanks, Kinglong Mee > > Daniel > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Any plans to support callback API model for FSALs?
Hello Daniel, I am interested in the ASYNC API for FSALs. Could you show me some information (plan, document, discussion) about it? If there is a draft of source about it, I am happy to help with the testing. thanks, Kinglong Mee On Thu, May 11, 2017 at 10:40 PM, Daniel Gryniewicz wrote: > Basically, yes. The plan is to implement an ASYNC API for FSAls, which > would allow this. It's high on my list for 2.6, so it should be done > fairly soon. > > Daniel > > On 05/11/2017 10:27 AM, Satish Chandra Kilaru wrote: >> Suppose an FSAL archives old files. When there is read request for those >> files, it can take long time to restore those files. Instead of blocking >> the worker thread for that long, it is good to allow FSAL to callback >> when data is available. >> >> Any plans to support that? >> -- >> Please Donate to www.wikipedia.org <http://www.wikipedia.org> >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> >> >> >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] The life of tcp drc
Describes in src/RPCAL/nfs_dupreq.c, * The life of tcp drc: it gets allocated when we process the first * request on the connection. It is put into rbtree (tcp_drc_recycle_t). * drc cache maintains a ref count. Every request as well as the xprt * holds a ref count. Its ref count should go to zero when the * connection's xprt gets freed (all requests should be completed on the * xprt by this time). When the ref count goes to zero, it is also put * into a recycle queue (tcp_drc_recycle_q). When a reconnection * happens, we hope to find the same drc that was used before, and the * ref count goes up again. At the same time, the drc will be removed * from the recycle queue. Only drc's with ref count zero end up in the * recycle queue. If a reconnection doesn't happen in time, the drc gets * freed by drc_free_expired() after some period of inactivety. Some questions about the life time of tcp drc, 1. The are two references of drc for xprt in nfs_dupreq_get_drc(). 629 /* xprt ref */ 630 drc->refcnt = 1; ... 638 (void)nfs_dupreq_ref_drc(drc); /* xprt ref */ ... 653 req->rq_xprt->xp_u2 = (void *)drc; I think it's a bug. The first one needs remove. Right? 2. The is no place to decrease the reference of drc for xprt. The xprt argument in nfs_dupreq_put_drc() is unused. Should it be used to decrease the ref? I think it's the right place to decrease the ref in nfs_dupreq_put_drc(). 3. My doubts is that, the life time of drc stored in req->rq_xprt->xp_u2 ? Start at #1, end at #2 (req->rq_xprt->xp_u2 = NULL) ? If that, the bad case is always lookup drc from tcp_drc_recycle_t. Otherwise, don't put the reference at #2, when to put it? the bad case is the drc ref always be 1 forever, am I right? thanks, Kinglong Mee -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] The life of tcp drc
Describes in src/RPCAL/nfs_dupreq.c, * The life of tcp drc: it gets allocated when we process the first * request on the connection. It is put into rbtree (tcp_drc_recycle_t). * drc cache maintains a ref count. Every request as well as the xprt * holds a ref count. Its ref count should go to zero when the * connection's xprt gets freed (all requests should be completed on the * xprt by this time). When the ref count goes to zero, it is also put * into a recycle queue (tcp_drc_recycle_q). When a reconnection * happens, we hope to find the same drc that was used before, and the * ref count goes up again. At the same time, the drc will be removed * from the recycle queue. Only drc's with ref count zero end up in the * recycle queue. If a reconnection doesn't happen in time, the drc gets * freed by drc_free_expired() after some period of inactivety. Some questions about the life time of tcp drc, 1. The are two references of drc for xprt in nfs_dupreq_get_drc(). 629 /* xprt ref */ 630 drc->refcnt = 1; ... 638 (void)nfs_dupreq_ref_drc(drc); /* xprt ref */ ... 653 req->rq_xprt->xp_u2 = (void *)drc; I think it's a bug. The first one needs remove. Right? 2. The is no place to decrease the reference of drc for xprt. The xprt argument in nfs_dupreq_put_drc() is unused. Should it be used to decrease the ref? I think it's the right place to decrease the ref in nfs_dupreq_put_drc(). 3. My doubts is that, the life time of drc stored in req->rq_xprt->xp_u2 ? Start at #1, end at #2 (req->rq_xprt->xp_u2 = NULL) ? If that, the bad case is always lookup drc from tcp_drc_recycle_t. Otherwise, don't put the reference at #2, when to put it? the bad case is the drc ref always be 1 forever, am I right? thanks, Kinglong Mee -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] The life of tcp drc
Hi Malahal, Thanks for your reply. #1. I'd like to post a patch to delete it, because I have some cleanups for drc. #2/#3. Sorry for my missing of nfs_rpc_free_user_data(). With it, everything is okay. thanks, Kinglong Mee On 10/13/2017 15:52, Malahal Naineni wrote: > #1. Looks like a bug! Lines 629 and 630 should be deleted > #2. See nfs_rpc_free_user_data(). It sets xp_u2 to NULL and drc ref is > decremented there. > #3. Life time of drc should start when it is allocated in > nfs_dupreq_get_drc() using alloc_tcp_drc(). > It can live beyond xprt's xp_u2 setting to NULL. It will live until we > decide to free in drc_free_expired() using free_tcp_drc(). > > Regards, Malahal. > PS: The comment "drc cache maintains a ref count." seems to imply that it > will have a refcount for keeping it in the hash table itself. I may have kept > those two lines because of that but It doesn't make sense as refcnt will > never go to zero this way. > > On Thu, Oct 12, 2017 at 3:48 PM, Kinglong Mee <mailto:kinglong...@gmail.com>> wrote: > > Describes in src/RPCAL/nfs_dupreq.c, > > * The life of tcp drc: it gets allocated when we process the first > * request on the connection. It is put into rbtree (tcp_drc_recycle_t). > * drc cache maintains a ref count. Every request as well as the xprt > * holds a ref count. Its ref count should go to zero when the > * connection's xprt gets freed (all requests should be completed on the > * xprt by this time). When the ref count goes to zero, it is also put > * into a recycle queue (tcp_drc_recycle_q). When a reconnection > * happens, we hope to find the same drc that was used before, and the > * ref count goes up again. At the same time, the drc will be removed > * from the recycle queue. Only drc's with ref count zero end up in the > * recycle queue. If a reconnection doesn't happen in time, the drc gets > * freed by drc_free_expired() after some period of inactivety. > > Some questions about the life time of tcp drc, > 1. The are two references of drc for xprt in nfs_dupreq_get_drc(). > 629 /* xprt ref */ > 630 drc->refcnt = 1; > ... > 638 (void)nfs_dupreq_ref_drc(drc); /* xprt > ref */ > ... > 653 req->rq_xprt->xp_u2 = (void *)drc; > > I think it's a bug. The first one needs remove. Right? > > 2. The is no place to decrease the reference of drc for xprt. > The xprt argument in nfs_dupreq_put_drc() is unused. > Should it be used to decrease the ref? > I think it's the right place to decrease the ref in > nfs_dupreq_put_drc(). > > 3. My doubts is that, the life time of drc stored in req->rq_xprt->xp_u2 ? > Start at #1, end at #2 (req->rq_xprt->xp_u2 = NULL) ? > If that, the bad case is always lookup drc from tcp_drc_recycle_t. > > Otherwise, don't put the reference at #2, when to put it? > the bad case is the drc ref always be 1 forever, am I right? > > thanks, > Kinglong Mee > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > <mailto:Nfs-ganesha-devel@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > <https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel> > > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] ganesha crash when stopping with gluster handles exist
With the latest codes (nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64),
set ganesha.conf includes,
CACHEINODE {
Dir_Max = 50;
Dir_Chunk = 0;
}
that forces mdcache_readdir enter mdcache_dirent_populate, but mdc_add_cache
return ERR_FSAL_OVERFLOW when a directory contains larger than 50 files
(I tests with 100 files).
After return ERR_FSAL_OVERFLOW, a handle is left in gluster fsal,
but no entry added to mdcache.
Right now, restarting nfs-ganesha will crash as,
#0 0x7f75bb9ea6e7 in __inode_unref () from /lib64/libglusterfs.so.0
#1 0x7f75bb9eaf41 in inode_unref () from /lib64/libglusterfs.so.0
#2 0x7f75bbccdad6 in glfs_h_close () from /lib64/libgfapi.so.0
#3 0x7f75bc0eb93f in handle_release ()
from /usr/lib64/ganesha/libfsalgluster.so
#4 0x7f75c12c6faf in destroy_fsals ()
#5 0x7f75c12e158f in admin_thread ()
#6 0x7f75bf849dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x7f75bef1b73d in clone () from /lib64/libc.so.6
For gluster, those sources (eg, handles, inode) belong to a glfs,
the glfs is freed at remove_all_exports() before destory_fsal(),
so that, glfs_h_close() use after freed memory.
I have no idea how to fix the problem,
1. Adds the gluster handle (meets the ERR_FSAL_OVERFLOW) to mdcache entry ?
2. Binds gluster handles to a glfs export, and release before freeing glfs?
3. Only move the shutdown_handles() before remove_all_exports()?
valgrind shows many messages as,
==13796== Invalid read of size 8
==13796==at 0xA5E39C2: ??? (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA5E5D58: inode_forget (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA3A0ACD: glfs_h_close (in /usr/lib64/libgfapi.so.0.0.0)
==13796==by 0x9F6B93E: ??? (in /usr/lib64/ganesha/libfsalgluster.so.4.2.0)
==13796==by 0x147FAE: destroy_fsals (in /usr/bin/ganesha.nfsd)
==13796==by 0x16258E: admin_thread (in /usr/bin/ganesha.nfsd)
==13796==by 0x6441DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==13796==by 0x6DAA73C: clone (in /usr/lib64/libc-2.17.so)
==13796== Address 0x1d3b6d58 is 104 bytes inside a block of size 256 free'd
==13796==at 0x4C28CDD: free (vg_replace_malloc.c:530)
==13796==by 0xA5FD9BB: free_obj_list (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA5FDDA7: mem_pools_fini (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA38A2D6: glfs_fini (in /usr/lib64/libgfapi.so.0.0.0)
==13796==by 0x9F6687A: glusterfs_free_fs (in
/usr/lib64/ganesha/libfsalgluster.so.4.2.0)
==13796==by 0x9F66C19: ??? (in /usr/lib64/ganesha/libfsalgluster.so.4.2.0)
==13796==by 0x223E1E: ??? (in /usr/bin/ganesha.nfsd)
==13796==by 0x200FFA: free_export_resources (in /usr/bin/ganesha.nfsd)
==13796==by 0x212288: free_export (in /usr/bin/ganesha.nfsd)
==13796==by 0x215329: remove_all_exports (in /usr/bin/ganesha.nfsd)
==13796==by 0x1624B8: admin_thread (in /usr/bin/ganesha.nfsd)
==13796==by 0x6441DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==13796== Block was alloc'd at
==13796==at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==13796==by 0xA5FE3C4: mem_get (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA5FE4D2: mem_get0 (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA5E3BD3: ??? (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0xA5E510A: inode_new (in /usr/lib64/libglusterfs.so.0.0.1)
==13796==by 0x178B5779: ???
==13796==by 0x178EA503: ???
==13796==by 0x178C3E46: ???
==13796==by 0xA8B8E7F: rpc_clnt_handle_reply (in
/usr/lib64/libgfrpc.so.0.0.1)
==13796==by 0xA8B9202: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.1)
==13796==by 0xA8B4F82: rpc_transport_notify (in
/usr/lib64/libgfrpc.so.0.0.1)
==13796==by 0x1741D595: ??? (in
/usr/lib64/glusterfs/3.13.1/rpc-transport/socket.so)
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] Ganesha caches bad directory entries when other glusterfs client(ganesha) creating files
I use ganesha on glusterfs as, 1. Glusterfs contains two bricks (brick1,brick2), 2. Two ganeshas bases on glusterfs, and two nfs clients through the two ganeshas, 3. One nfs client(ganeshaA, clientA) copy small files in; the other nfs client(ganeshaB, clientB) do "find /mnt/nfs/ | wc -l" by cycle. After some times later, ctrl+c the copying and the finding, do "find /mnt/nfs/ | wc -l" on clientA and clientB. When the system time deviation between brick1 and brick2 is larger than 0.1 seconds (eg, the system time of brick1 is 0.1s lesser than brick2), the files count showed by clientB is less than clientA. Ganesha caches the directory entries at memory depends on directory's mtime, those cached entries are valid until the directory's mtime change. But if ganesha caches bad entries, the client will never shows the right entries until the directory's mtime change. It's the problem I have meet. All following analysis are based on the system time deviation between brick1 is 0.1s less than brick2; 1. DHT readdirs from brick1 to brick2 in proper order; 2 A directory's mtime at brick2 is newer than brick1; 3. GaneshaB's dht readdirs from brick1, gets old entries from the directory; 4. GaneshaA's dht creates one file under the directory, and update the mtime of directory on brick1; ** for the 0.1s lesser than brick2, the new mtime at brick1 maybe less than brick2. 5. GaneshaB's dht readdirs from brick2, gets latest entries from the same name directory; 6. GaneshaB gets the brick2's mtime as the directory's mtime; 7. At bad case, ganeshaB caches those entries (without the file created at #4). Is it a known issue? Maybe we should update the directory's mtime on all nodes at #4 (on brick1 and brick2)? thanks, Kinglong Mee -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] nfsv3 client writing file gets Invalid argument on glusterfs with quota on
When using nfsv3 on glusterfs-3.13.1-1.el7.x86_64 and nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64, I gets strange "Invalid argument" when writing file. 1. With quota disabled; nfs client mount nfs-ganesha share, and do 'll' in the testing directory. 2. Enable quota; # getfattr -d -m . -e hex /root/rpmbuild/gvtest/nfs-ganesha/testfile92 getfattr: Removing leading '/' from absolute path names # file: root/rpmbuild/gvtest/nfs-ganesha/testfile92 trusted.gfid=0xe2edaac0eca8420ebbbcba7e56bbd240 trusted.gfid2path.b3250af8fa558e66=0x3966313434352d653530332d343831352d396635312d3236633565366332633137642f7465737466696c653932 trusted.glusterfs.quota.9f1445ff-e503-4815-9f51-26c5e6c2c17d.contri.3=0x0201 Notice: testfile92 without trusted.pgfid xattr. 3. restart glusterfs volume by "gluster volume stop/start gvtest" 4. echo somedata > testfile92 5. ll testfile92 -rw-r--r-- 1 root root0 Mar 6 21:43 testfile92 There are some error in brick log message, [2018-03-07 02:30:19.222045] W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-gvtest-quota: parent is NULL [Invalid argument] [2018-03-07 02:30:32.328977] W [marker-quota.c:33:mq_loc_copy] 0-marker: src loc is not valid [2018-03-07 02:30:32.329001] E [marker-quota.c:1488:mq_initiate_quota_task] 0-gvtest-marker: loc copy failed The message "W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-gvtest-quota: parent is NULL [Invalid argument]" repeated 136 times between [2018-03-07 02:30:19.222045] and [2018-03-07 02:30:32.329708] [2018-03-07 02:30:32.329725] E [MSGID: 115067] [server-rpc-fops.c:1407:server_writev_cbk] 0-gvtest-server: 211: WRITEV 5 (e2edaac0-eca8-420e-bbbc-ba7e56bbd240), client: CENTOS7-MINI-2758-2018/03/07-02:26:14:328922-gvtest-client-1-0-2, error-xlator: gvtest-quota [Invalid argument] [2018-03-07 02:43:18.435729] W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-gvtest-quota: parent is NULL [Invalid argument] [2018-03-07 02:43:18.435781] E [MSGID: 115067] [server-rpc-fops.c:1407:server_writev_cbk] 0-gvtest-server: 387: WRITEV 5 (e2edaac0-eca8-420e-bbbc-ba7e56bbd240), client: CENTOS7-MINI-2758-2018/03/07-02:26:14:328922-gvtest-client-1-0-2, error-xlator: gvtest-quota [Invalid argument] Ps, the corresponding tcpdump is attached. nfsv3-client-write-Invalid argument.cap Description: Binary data -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfsv3 client writing file gets Invalid argument on glusterfs with quota on
On 2018/3/7 10:59, Kinglong Mee wrote: > When using nfsv3 on glusterfs-3.13.1-1.el7.x86_64 and > nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64, > I gets strange "Invalid argument" when writing file. > > 1. With quota disabled; > nfs client mount nfs-ganesha share, and do 'll' in the testing directory. > > 2. Enable quota; > # getfattr -d -m . -e hex /root/rpmbuild/gvtest/nfs-ganesha/testfile92 > getfattr: Removing leading '/' from absolute path names > # file: root/rpmbuild/gvtest/nfs-ganesha/testfile92 > trusted.gfid=0xe2edaac0eca8420ebbbcba7e56bbd240 > trusted.gfid2path.b3250af8fa558e66=0x3966313434352d653530332d343831352d396635312d3236633565366332633137642f7465737466696c653932 > trusted.glusterfs.quota.9f1445ff-e503-4815-9f51-26c5e6c2c17d.contri.3=0x0201 > > Notice: testfile92 without trusted.pgfid xattr. The trusted.pgfid will be created by the next name lookup; nameless lookup don't create it. > 3. restart glusterfs volume by "gluster volume stop/start gvtest" Restarting glusterfsd here cleanup all inode cache from memory; after starting, inode of testfile92's parent is NULL. > 4. echo somedata > testfile92 Because, nfs-ganesha and nfs client has cache for testfile92, before write fops, no name lookup happens that trusted.pgfid is not created for testfile92. Quota_writev call quota_build_ancestry building the ancestry in quota_check_limit, but testfile92 doesn't contain trusted.pgfid, so that write fops failed with Invalid argument. I have no idea of fixing this problem, any comments are welcome. thanks, Kinglong Mee > 5. ll testfile92 > -rw-r--r-- 1 root root0 Mar 6 21:43 testfile92 > > There are some error in brick log message, > [2018-03-07 02:30:19.222045] W [MSGID: 120003] > [quota.c:821:quota_build_ancestry_cbk] 0-gvtest-quota: parent is NULL > [Invalid argument] > [2018-03-07 02:30:32.328977] W [marker-quota.c:33:mq_loc_copy] 0-marker: src > loc is not valid > [2018-03-07 02:30:32.329001] E [marker-quota.c:1488:mq_initiate_quota_task] > 0-gvtest-marker: loc copy failed > The message "W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] > 0-gvtest-quota: parent is NULL [Invalid argument]" repeated 136 times between > [2018-03-07 02:30:19.222045] and [2018-03-07 02:30:32.329708] > [2018-03-07 02:30:32.329725] E [MSGID: 115067] > [server-rpc-fops.c:1407:server_writev_cbk] 0-gvtest-server: 211: WRITEV 5 > (e2edaac0-eca8-420e-bbbc-ba7e56bbd240), client: > CENTOS7-MINI-2758-2018/03/07-02:26:14:328922-gvtest-client-1-0-2, > error-xlator: gvtest-quota [Invalid argument] > [2018-03-07 02:43:18.435729] W [MSGID: 120003] > [quota.c:821:quota_build_ancestry_cbk] 0-gvtest-quota: parent is NULL > [Invalid argument] > [2018-03-07 02:43:18.435781] E [MSGID: 115067] > [server-rpc-fops.c:1407:server_writev_cbk] 0-gvtest-server: 387: WRITEV 5 > (e2edaac0-eca8-420e-bbbc-ba7e56bbd240), client: > CENTOS7-MINI-2758-2018/03/07-02:26:14:328922-gvtest-client-1-0-2, > error-xlator: gvtest-quota [Invalid argument] > > Ps, the corresponding tcpdump is attached. > > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfsv3 client writing file gets Invalid argument on glusterfs with quota on
On 2018/3/7 21:10, Daniel Gryniewicz wrote: > On 03/06/2018 10:10 PM, Kinglong Mee wrote: >> On 2018/3/7 10:59, Kinglong Mee wrote: >>> When using nfsv3 on glusterfs-3.13.1-1.el7.x86_64 and >>> nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64, >>> I gets strange "Invalid argument" when writing file. >>> >>> 1. With quota disabled; >>> nfs client mount nfs-ganesha share, and do 'll' in the testing directory. >>> >>> 2. Enable quota; >>> # getfattr -d -m . -e hex /root/rpmbuild/gvtest/nfs-ganesha/testfile92 >>> getfattr: Removing leading '/' from absolute path names >>> # file: root/rpmbuild/gvtest/nfs-ganesha/testfile92 >>> trusted.gfid=0xe2edaac0eca8420ebbbcba7e56bbd240 >>> trusted.gfid2path.b3250af8fa558e66=0x3966313434352d653530332d343831352d396635312d3236633565366332633137642f7465737466696c653932 >>> trusted.glusterfs.quota.9f1445ff-e503-4815-9f51-26c5e6c2c17d.contri.3=0x0201 >>> >>> Notice: testfile92 without trusted.pgfid xattr. >> >> The trusted.pgfid will be created by the next name lookup; nameless lookup >> don't create it. >> >>> 3. restart glusterfs volume by "gluster volume stop/start gvtest" >> >> Restarting glusterfsd here cleanup all inode cache from memory; >> after starting, inode of testfile92's parent is NULL. >> >>> 4. echo somedata > testfile92 >> >> Because, nfs-ganesha and nfs client has cache for testfile92, >> before write fops, no name lookup happens that trusted.pgfid is not created >> for testfile92. >> >> Quota_writev call quota_build_ancestry building the ancestry in >> quota_check_limit, >> but testfile92 doesn't contain trusted.pgfid, so that write fops failed with >> Invalid argument. >> >> I have no idea of fixing this problem, any comments are welcome. >> > > I think, ideally, Gluster would send an invalidate upcall under the > circumstances, causing Ganesha do drop it's cached entry. It doesn't work. I try to restarting nfs-ganesha, and echo data to testfile92, the problem also exists. After nfs-ganesha restart, 1. A GETATTR is send from nfs-client for the testfile92, ganesha translates it to nameless lookup. 2. A ACCESS gets attributes from nfs-ganesha's cache (cached by #1). 3. A SETATTR sets the testfile92's size to 0, ganesha translates it to setattr fop. 4. A WRITE also get Invalid argument error. If ganesha drops its cache, nfs client may write file by filehandle; ganesha lookup it by nameless lookup from glusterfs, so that, trusted.pgfid isn't created too. I think, a name lookup is needed for testfile92 after quota enable. thanks, Kinglong Mee -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] nfsv3 client writing file gets Invalid argument on glusterfs with quota on
On Thu, Mar 8, 2018 at 10:17 PM Daniel Gryniewicz wrote: > On 03/07/2018 10:21 PM, Kinglong Mee wrote: > > On 2018/3/7 21:10, Daniel Gryniewicz wrote: > >> On 03/06/2018 10:10 PM, Kinglong Mee wrote: > >>> On 2018/3/7 10:59, Kinglong Mee wrote: > >>>> When using nfsv3 on glusterfs-3.13.1-1.el7.x86_64 and > nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64, > >>>> I gets strange "Invalid argument" when writing file. > >>>> > >>>> 1. With quota disabled; > >>>> nfs client mount nfs-ganesha share, and do 'll' in the testing > directory. > >>>> > >>>> 2. Enable quota; > >>>> # getfattr -d -m . -e hex /root/rpmbuild/gvtest/nfs-ganesha/testfile92 > >>>> getfattr: Removing leading '/' from absolute path names > >>>> # file: root/rpmbuild/gvtest/nfs-ganesha/testfile92 > >>>> trusted.gfid=0xe2edaac0eca8420ebbbcba7e56bbd240 > >>>> > trusted.gfid2path.b3250af8fa558e66=0x3966313434352d653530332d343831352d396635312d3236633565366332633137642f7465737466696c653932 > >>>> > trusted.glusterfs.quota.9f1445ff-e503-4815-9f51-26c5e6c2c17d.contri.3=0x0201 > >>>> > >>>> Notice: testfile92 without trusted.pgfid xattr. > >>> > >>> The trusted.pgfid will be created by the next name lookup; nameless > lookup don't create it. > >>> > >>>> 3. restart glusterfs volume by "gluster volume stop/start gvtest" > >>> > >>> Restarting glusterfsd here cleanup all inode cache from memory; > >>> after starting, inode of testfile92's parent is NULL. > >>> > >>>> 4. echo somedata > testfile92 > >>> > >>> Because, nfs-ganesha and nfs client has cache for testfile92, > >>> before write fops, no name lookup happens that trusted.pgfid is not > created for testfile92. > >>> > >>> Quota_writev call quota_build_ancestry building the ancestry in > quota_check_limit, > >>> but testfile92 doesn't contain trusted.pgfid, so that write fops > failed with Invalid argument. > >>> > >>> I have no idea of fixing this problem, any comments are welcome. > >>> > >> > >> I think, ideally, Gluster would send an invalidate upcall under the > circumstances, causing Ganesha do drop it's cached entry. > > > > It doesn't work. > > I try to restarting nfs-ganesha, and echo data to testfile92, the > problem also exists. > > > > After nfs-ganesha restart, > > 1. A GETATTR is send from nfs-client for the testfile92, ganesha > translates it to nameless lookup. > > 2. A ACCESS gets attributes from nfs-ganesha's cache (cached by #1). > > 3. A SETATTR sets the testfile92's size to 0, ganesha translates it to > setattr fop. > > 4. A WRITE also get Invalid argument error. > > > > If ganesha drops its cache, nfs client may write file by filehandle; > > ganesha lookup it by nameless lookup from glusterfs, > > so that, trusted.pgfid isn't created too. > > > > I think, a name lookup is needed for testfile92 after quota enable. > > > > thanks, > > Kinglong Mee > > > > If a name lookup is needed, then gluster cannot provide NFS semantics in > these circumstances. NFS *requires* that access by handle continue to > work once the client has a handle. In fact, there's no way to get a > name for this, since the name doesn't exist anywhere in the handle (by > design, since a name is an attribute of a dirent, not of a handle/inode). > > So, this likely is a bug in Gluster, and needs to be fixed there. Would > it be possible to enable quota globally at the start as a workaround? > Quota is a function that can be enabled at any time. If enable quota immediately after creating filesystem (contains only one directory "/"), the trusted.pgfid extend-attribute is created automatically when creating file, this problem is not exit. thanks, Kinglong Mee > > Daniel > > > -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
