Re: [Nfs-ganesha-devel] chunk/dirent LRU memory management first pass pushed
Hi Frank, Entries were supposed to gradually move to L2 independent of any other process--the only connection was intended to be that the L1 -> L2 transition might induce other compaction, hints, etc as a side effect. Matt - "Frank Filz" wrote: > Take a look: > > https://review.gerrithub.io/356041 > > It's not complete, and not tested, but it's at least something to look > at > and discuss. > > I do have one question about the lru run thread. For mdcache entries, > it's > running seems dependent on open file descriptors, not number of > mdcache > entries in use, and thus it looks like only when there's too many open > file > descriptors will it put pressure to demote mdache entries from the L1 > to the > L2 queue. I'm not sure that's a good behavior, especially as we get > away > from having so many open file descriptors tracked by mdcache (open fd > associated with state aren't counted against the high water mark). > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] revert/repair ntirpc patch b4254b92
Hi Bill,
Here's a thought: why not spend more time doing useful work, and less time
trying to find someone to blame for changed behavior your own work introduced.
Matt
- "William Allen Simpson" wrote:
> Last week, Daniel G had to revert one of my patches, because the CEA
> tests segfaulted during shutdown. Neither of us could reproduce.
>
> Yesterday, I updated my Fedora debuginfos. Suddenly, I could
> reproduce!
>
> After _many_ *MANY* hours, I've discovered that it wasn't in my patch.
> :(
>
> ntirpc git log b4254b92 claims that it is a patch based upon
> valgrind.
> The long log message is almost entirely about Ganesha, not ntirpc.
> It
> was approved by Matt Benjamin.
>
> In fact, the commit is almost entirely useless for ntirpc.
>
> Here's the change in its entirely. It's shorter than the log
> message:
>
> git diff b4254b92^ b4254b92
> diff --git a/src/clnt_vc.c b/src/clnt_vc.c
> index 174e7e2..57f2b23 100644
> --- a/src/clnt_vc.c
> +++ b/src/clnt_vc.c
> @@ -312,7 +312,7 @@ clnt_vc_ncreate2(int fd,/* open file
> descriptor */
> if (ct->ct_addr.len)
> mem_free(ct->ct_addr.buf, ct->ct_addr.len);
>
> - if (xd->refcnt == 1) {
> + if (xd->refcnt == 0) {
> XDR_DESTROY(&xd->shared.xdrs_in);
> XDR_DESTROY(&xd->shared.xdrs_out);
> free_x_vc_data(xd);
> diff --git a/src/svc_vc.c b/src/svc_vc.c
> index cabd325..e5071c1 100644
> --- a/src/svc_vc.c
> +++ b/src/svc_vc.c
> @@ -658,10 +658,13 @@ svc_rdvs_destroy(SVCXPRT *xprt, u_int flags,
> const char *tag, const int line)
> mem_free(rdvs, sizeof(struct cf_rendezvous));
> mem_free(xprt, sizeof(SVCXPRT));
>
> - refcnt = rpc_dplx_unref(rec,
> - RPC_DPLX_FLAG_LOCKED |
> RPC_DPLX_FLAG_UNLOCK);
> - if (!refcnt)
> - mem_free(xd, sizeof(struct x_vc_data));
> + (void)rpc_dplx_unref(rec, RPC_DPLX_FLAG_LOCKED |
> RPC_DPLX_FLAG_UNLOCK);
> +
> + if (xd->refcnt == 0) {
> + XDR_DESTROY(&xd->shared.xdrs_in);
> + XDR_DESTROY(&xd->shared.xdrs_out);
> + free_x_vc_data(xd);
> + }
> }
>
> static void
> ===
>
> It's crashing on the first XDR_DESTROY().
>
> That will never work, because as the code states in
> svc_vc_ncreate2():
>
> /* duplex streams are not used by the rendevous transport */
> memset(&xd->shared.xdrs_in, 0, sizeof xd->shared.xdrs_in);
> memset(&xd->shared.xdrs_out, 0, sizeof xd->shared.xdrs_out);
>
> ===
>
> That code was copy and pasted from clnt_vc.c in the same patch?!?!
>
> Why did my patch discover the problem? Because svc_rdvs_destroy()
> was
> not being called on shutdown My patch fixes that bug.
>
> *** So valgrind couldn't possibly have found this as an error. ***
>
> I'll fix it in my future patch. But we need to be a lot more
> careful,
> and should not have irrelevant commit log messages that obscure the
> actual underlying changes in the patch.
>
> Time for bed
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Deepsea support for NFS-Ganesha
Very nice, thanks folks. Matt - "Supriti Singh" wrote: > Hello all, > > As I said in the concall this week, at SUSE, we are working on > salt-based deployment system for ceph, called Deepsea. > https://github.com/SUSE/DeepSea > > Recently we added support to automatically deploy Ganesha server in > the ceph cluster. Users can select nodes to > act as ganesha server. Deepsea takes care of installing nfs-ganesha, > copying the correct config file and starting the > service. We support only CephFS and RGW FSALs. > > For CephFS, we don't use admin keyring. Hence, it works only for > Ganesha v2.5-dev7 or higher. > > Deepsea at the moment can be used only for SUSE distributions. > > Thanks, > Supriti > > > > -- > Supriti Singh > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, > HRB 21284 (AG Nürnberg) > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] ganesha-2.4.1 + RGW FSAL + Ceph Jewel
Hi Karol, - "Karol Mroz" wrote: > Hi Matt, > > Thanks for the input. Please see inline. > > On Sun, Nov 27, 2016 at 10:58:34PM -0500, Matt W. Benjamin wrote: > [...] > > > 1. `echo` or `touch` a new file > > > > > > File creation via `echo` or `touch` (createmode=1, offset=0, > length=N) > > > has > > > so far been successful in my test environment. > > > > > > > > > 2. Append to an existing file using `echo` > > > > > > On an append via `echo`, the behaviour is not consistent. > > > > > > i) offset=0, length=N > > > - A full overwrite via `echo` has shown to be successful. > > > > > > ii) offset=M, length=(N - M) > > > - The librgw:rgw_write() code path ultimately rejects this > write > > > and > > > (-5) bubbles back up to ganesha. This makes sense as RGW > > > objects need > > > to be completely overwritten. > > > > Right, that's the current expected behavior. > > > > > > > > In the case of (2ii), I've seen inconsistency between the S3 > object > > > and the > > > NFS representation. The file will have an updated size as if the > > > append > > > succeeded, however S3 shows that the object has not been modified. > > > > Trying to > > > then read the file from NFS would generate I/O errors. At this > point, > > > a restart > > > of ganesha was needed. > > > > Seems like a bug, sorry. > > > > > > > > Also, it isn't clear to me why nfs4_write() interprets one `echo` > > > invocation as > > > an overwrite, and another identical `echo` as an append? Given > the > > > rgw_write() > > > offset limitations, it seems file modification/appends are not > yet > > > well supported? > > > > For the moment, what you should be doing is mounting with -osync > (Linux) to get consistent results. Can you give that a try? > > Mounting with '-o sync' appears to remove (2ii). A failed, simple > write (echo) on the NFS side > does not trigger IO errors when reading the file. However, the write > failures still persist when > trying to write into the file at an offset, rather than overwriting > the file fully (log snippet > below). I may not be following you completely. Are you saying that there is a way to attempt -whole-file overwrite- which fails? Because that's the only supported write mode, currently. The mechanisms by which nfsv4 and nfs3 detect the end of the overwrite cycle differ (nfs3 lacks open), but both should work. I.e, you should be able to write sequentially from offset 0, but not otherwise. Matt > > What I see is that the first write (new file or append) via an echo > succeeds (offset=0, length=N). > The next write(s) will fail (offset=M, length=(N - M)). _However_, > re-reading the file (cat) and then > issuing an echo will succeed (offset=0, length=N). So re-reading the > file between writes appears to help. > > I'm wondering if it's a client side problem (passing inconsistent > offset and length values to nfs4_write())? > The fact that cat'ing the file seems to reset the offset and length to > an overwrite is interesting... > > Log of failed write: -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] ntirpc now has master and next branches
Hi Bill, Ok, thanks for the heads up, I at least do support this change. Matt - "William Allen Simpson" wrote: > Today, there was a small mixup on pulls. Malahal's was added to > duplex-13, while mine were added to duplex-14. > > After consultation with Matt, I've changed the repository to match > the main nfs-ganesha style: master and next. > > master is the stable protected branch (plus Malahal's recent patch). > > next is the active and default branch. Had to force push to > install Malahal's before mine, so that his local copy should have the > same commit hashes. But from here, hopefully will be OK. > > All the old duplex-nn branches are gone, although personal branches > with that prefix remain. > > Happy Holidays. > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] ganesha-2.4.1 + RGW FSAL + Ceph Jewel
Hi Karol, - "Karol Mroz" wrote: > Hi Ganesha developers, > > I've been playing with the RGW FSAL (ganesha-2.4.1) on a fairly > recent > Jewel-based (10.2.3-493-gb4c314a) Ceph cluster and wanted to share > some > observations. > > In a read-only scenario, the FSAL appears to function well! Great > work! > > However, I am experiencing some problems when attempting writes. I am > curious > if anyone else is testing writes with the RGW FSAL and if the below > are already > known issues? If not, I'd be happy to help debug further. > > Thanks in advance! > > > 0. Bucket/Object creation via S3 > > No issues on the S3 side, however my NFS mount will not see the new > buckets > or objects until ganesha is restarted. For some scenarios, this is probably expected, without some ganesha-side limits on caching, presently. > > > 1. `echo` or `touch` a new file > > File creation via `echo` or `touch` (createmode=1, offset=0, length=N) > has > so far been successful in my test environment. > > > 2. Append to an existing file using `echo` > > On an append via `echo`, the behaviour is not consistent. > > i) offset=0, length=N > - A full overwrite via `echo` has shown to be successful. > > ii) offset=M, length=(N - M) > - The librgw:rgw_write() code path ultimately rejects this write > and > (-5) bubbles back up to ganesha. This makes sense as RGW > objects need > to be completely overwritten. Right, that's the current expected behavior. > > In the case of (2ii), I've seen inconsistency between the S3 object > and the > NFS representation. The file will have an updated size as if the > append > succeeded, however S3 shows that the object has not been modified. > Trying to > then read the file from NFS would generate I/O errors. At this point, > a restart > of ganesha was needed. Seems like a bug, sorry. > > Also, it isn't clear to me why nfs4_write() interprets one `echo` > invocation as > an overwrite, and another identical `echo` as an append? Given the > rgw_write() > offset limitations, it seems file modification/appends are not yet > well supported? For the moment, what you should be doing is mounting with -osync (Linux) to get consistent results. Can you give that a try? Regards, Matt -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Build with Heimdal Kerberos?
agree. I'm not certain nfs-ganesha has built against heimdal, and I think the credential cache bits may also conflict. Matt - "Daniel Gryniewicz" wrote: > On 10/31/2016 12:19 PM, Wyllys Ingersoll wrote: > > Has anyone worked on making nfs-ganesha build and run with Heimdal > > Kerberos libraries instead of MIT? There are a few hard-coded MIT > > dependencies in the build configuration that currently break if the > > build server has Heimdal instead of MIT packages. > > > > I don't think anyone is working on this at the moment, but we'd be > happy > to accept patches if you'd like to get it working. It'd be nice, once > > it's working, to get the build into one of the CI bots so it doesn't > break again, but I'm not sure how hard that would be, given the > environments they work in. > > Daniel > > > -- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Pros and cons of setting high/low on Dispatch_Max_Reqs_Xprt?
Hi, - "Akira Hayakawa" wrote: > Dispatch_Max_Reqs_Xprt is settable attribute that affects > stalling the ganesha server when the outstanding requests in the > queue > supersedes the set value. The default value is 512 and can take 1 to > 2048. > Ganesha unstalls the server when the queuing requests decreases to > half of > the value. > > I learned about the attribute from the code but still I couldn't > fully > understand it. > > We are developing our own FSAL and what we are seeing now is stalling > occurs occasionally when the requests is too much. And the server > always wakes up > in one second. And when the server stalls clients seems to stop > sending requests > somehow (does Ganesha back-pressures when it stalls?) Yes, the xprt that is stalled will not be serviced for new requests until it unstalls. > > We are considering setting high value on the attribute but we aren't > sure what > the pros and cons of setting high value on it. Intuitively, setting > high value > may lead to stalling much less often but may lead to OOM on the other > hand. The main issue is fairness. A high value allows aggressive clients to consume an unfair fraction of available bandwidth/server resources. > > I think Ganesha doesn't expect itself stalls too often because the > code says it > unlikely. That's why we are thinking of setting high value on it. > > Do you think setting high value is a good idea for us? And what else > do you have > in mind to solve our problem? You can experiment with it. YMMV, basically. Matt > > Akira > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [ntirpc] refcount issue ?
Hi Swen,
Yes, please do.
Matt
- "Swen Schillig" wrote:
> On Mo, 2016-10-10 at 11:46 -0400, Matt Benjamin wrote:
> > Hi Swen,
> >
> > Are you going to re-push your recent closed PR as 2 new PRs as
> > planned?
> >
> > Dan has made some time this week to review and test, so if you have
> > time, it will be easier to get to merge.
> >
> > Cheers,
> >
> > Matt
> Hi Matt
>
> It is re-pushed , I just didn't create a new PR yet.
> I was thinking we should test and review before creating a new PR.
>
> But if you prefer to have a PR right away, I can do that.
>
> Cheers Swen
> >
> > - Original Message -
> > >
> > > From: "Matt Benjamin"
> > > To: "Swen Schillig"
> > > Cc: "Daniel Gryniewicz" , "nfs-ganesha-devel"
> > > -ganesha-de...@lists.sourceforge.net>
> > > Sent: Tuesday, September 6, 2016 4:16:39 PM
> > > Subject: Re: [ntirpc] refcount issue ?
> > >
> > > Hi,
> > >
> > > inline
> > >
> > > - Original Message -
> > > >
> > > > From: "Swen Schillig"
> > > > To: "Matt Benjamin" , "Daniel Gryniewicz"
> > > >
> > > > Cc: "nfs-ganesha-devel"
>
> > > > Sent: Tuesday, September 6, 2016 4:10:32 PM
> > > > Subject: [ntirpc] refcount issue ?
> > > >
> > > > Matt, Dan.
> > > >
> > > > Could you please have a look at the following code areas and
> > > > verify
> > > > what I think is a refcount issue.
> > > >
> > > > clnt_vc_ncreate2()
> > > > {
> > > > ...
> > > > if ((oflags & RPC_DPLX_LKP_OFLAG_ALLOC) || (!rec->hdl.xd)) {
> > > > xd = rec->hdl.xd = alloc_x_vc_data();
> > > > ...
> > > > } else {
> > > > xd = rec->hdl.xd;
> > > > ++(xd->refcnt); <=== this is not right. we're not
> > > > taking an addtl'
> > > > ref
> > >
> > > Aren't we? We now share a ref to previously allocated
> rec->hdl.xd.
> > >
> > > >
> > > > here.
> > > > }
> > > > ...
> > > > clnt->cl_p1 = xd; <=== but here we should increment
> > > > the
> > > > refocunt.
> > > > }
> > > >
> > > > another code section with the same handling.
> > > >
> > > > makefd_xprt()
> > > > {
> > > > ...
> > > > if ((oflags & RPC_DPLX_LKP_OFLAG_ALLOC) || (!rec->hdl.xd)) {
> > > > newxd = true;
> > > > xd = rec->hdl.xd = alloc_x_vc_data();
> > > > ...
> > > > } else {
> > > > xd = (struct x_vc_data *)rec->hdl.xd;
> > > > /* dont return destroyed xprts */
> > > > if (!(xd->flags & X_VC_DATA_FLAG_SVC_DESTROYED)) {
> > > > if (rec->hdl.xprt) {
> > > > xprt = rec->hdl.xprt;
> > > > /* inc xprt refcnt */
> > > > SVC_REF(xprt, SVC_REF_FLAG_NONE);
> > > > } else
> > > > ++(xd->refcnt); < not right, no
> > > > addtl' ref to xd taken.
> > >
> > > so this looks more likely to be incorrect, need to review
> > >
> > > >
> > > > }
> > > > /* return extra ref */
> > > > rpc_dplx_unref(rec,
> > > > RPC_DPLX_FLAG_LOCKED |
> > > > RPC_DPLX_FLAG_UNLOCK);
> > > > *allocated = FALSE;
> > > >
> > > > /* return ref'd xprt */
> > > > goto done_xprt;
> > > > }
> > > > ...
> > > > xprt->xp_p1 = xd; < but here we should increment the
> > > > refcount
> > > >
> > > > ...
> > > > }
> > > >
> > > > Both areas handle the refcount'ing wrong, but it might balance
> > > > out
> > > > sometimes.
> > > >
> > > >
> > > > What do you think ?
> > > >
> > > > Cheers Swen
> > > >
> > > >
> > >
> > > --
> > > Matt Benjamin
> > > Red Hat, Inc.
> > > 315 West Huron Street, Suite 140A
> > > Ann Arbor, Michigan 48103
> > >
> > > http://www.redhat.com/en/technologies/storage
> > >
> > > tel. 734-707-0660
> > > fax. 734-769-8938
> > > cel. 734-216-5309
> > >
> >
>
>
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lis
Re: [Nfs-ganesha-devel] [PATCH] Fix refcount leak in svc_rpc_gss_data under RPCSEC_GSS_DESTROY
Hi Malahal, Is this intended to be a PR? Matt - "Malahal Naineni" wrote: > A umount/mount loop for a krb5 mount was sending RPCSEC_GSS_DESTROY > calls at times. We end up a huge number of contexts that are expired > but > not destroyed, eventually failing mounts after some time. > > Release reference we got on GSS data object after deleting it from > hash > table. > > Signed-off-by: Malahal Naineni > --- > src/authgss_hash.c | 2 +- > src/svc_auth_gss.c | 7 ++- > 2 files changed, 7 insertions(+), 2 deletions(-) > > diff --git a/src/authgss_hash.c b/src/authgss_hash.c > index d88a378..21ecaf9 100644 > --- a/src/authgss_hash.c > +++ b/src/authgss_hash.c > @@ -187,7 +187,7 @@ authgss_ctx_hash_set(struct svc_rpc_gss_data *gd) > gss_ctx = (gss_union_ctx_id_desc *) (gd->ctx); > gd->hk.k = gss_ctx_hash(gss_ctx); > > - ++(gd->refcnt); /* locked */ > + (void)atomic_inc_uint32_t(&gd->refcnt); > t = rbtx_partition_of_scalar(&authgss_hash_st.xt, gd->hk.k); > mutex_lock(&t->mtx); > rslt = > diff --git a/src/svc_auth_gss.c b/src/svc_auth_gss.c > index 3322ab5..cc7b1f4 100644 > --- a/src/svc_auth_gss.c > +++ b/src/svc_auth_gss.c > @@ -612,7 +612,7 @@ _svcauth_gss(struct svc_req *req, struct rpc_msg > *msg, > > *no_dispatch = true; > > - (void)authgss_ctx_hash_del(gd); /* unrefs, can destroy gd */ > + (void)authgss_ctx_hash_del(gd); > > /* avoid lock order reversal gd->lock, xprt->xp_lock */ > mutex_unlock(&gd->lock); > @@ -622,6 +622,11 @@ _svcauth_gss(struct svc_req *req, struct rpc_msg > *msg, > svc_sendreply(req->rq_xprt, req, (xdrproc_t) xdr_void, > (caddr_t) NULL); > > + /* We acquired a reference on gd with authgss_ctx_hash_get > + * call. Time to release the reference as we don't need > + * gd anymore. > + */ > + unref_svc_rpc_gss_data(gd, SVC_RPC_GSS_FLAG_NONE); > req->rq_auth = &svc_auth_none; > > break; > -- > 1.8.3.1 > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH] Fix refcount leak in svc_rpc_gss_data under RPCSEC_GSS_DESTROY
Oh, it is. Matt - "Matt W. Benjamin" wrote: > Hi Malahal, > > Is this intended to be a PR? > > Matt > > - "Malahal Naineni" wrote: > > > A umount/mount loop for a krb5 mount was sending RPCSEC_GSS_DESTROY > > calls at times. We end up a huge number of contexts that are > expired > > but > > not destroyed, eventually failing mounts after some time. > > > > Release reference we got on GSS data object after deleting it from > > hash > > table. > > > > Signed-off-by: Malahal Naineni > > --- > > src/authgss_hash.c | 2 +- > > src/svc_auth_gss.c | 7 ++- > > 2 files changed, 7 insertions(+), 2 deletions(-) > > > > diff --git a/src/authgss_hash.c b/src/authgss_hash.c > > index d88a378..21ecaf9 100644 > > --- a/src/authgss_hash.c > > +++ b/src/authgss_hash.c > > @@ -187,7 +187,7 @@ authgss_ctx_hash_set(struct svc_rpc_gss_data > *gd) > > gss_ctx = (gss_union_ctx_id_desc *) (gd->ctx); > > gd->hk.k = gss_ctx_hash(gss_ctx); > > > > - ++(gd->refcnt); /* locked */ > > + (void)atomic_inc_uint32_t(&gd->refcnt); > > t = rbtx_partition_of_scalar(&authgss_hash_st.xt, gd->hk.k); > > mutex_lock(&t->mtx); > > rslt = > > diff --git a/src/svc_auth_gss.c b/src/svc_auth_gss.c > > index 3322ab5..cc7b1f4 100644 > > --- a/src/svc_auth_gss.c > > +++ b/src/svc_auth_gss.c > > @@ -612,7 +612,7 @@ _svcauth_gss(struct svc_req *req, struct > rpc_msg > > *msg, > > > > *no_dispatch = true; > > > > - (void)authgss_ctx_hash_del(gd); /* unrefs, can destroy gd */ > > + (void)authgss_ctx_hash_del(gd); > > > > /* avoid lock order reversal gd->lock, xprt->xp_lock */ > > mutex_unlock(&gd->lock); > > @@ -622,6 +622,11 @@ _svcauth_gss(struct svc_req *req, struct > rpc_msg > > *msg, > > svc_sendreply(req->rq_xprt, req, (xdrproc_t) xdr_void, > > (caddr_t) NULL); > > > > + /* We acquired a reference on gd with authgss_ctx_hash_get > > +* call. Time to release the reference as we don't need > > +* gd anymore. > > +*/ > > + unref_svc_rpc_gss_data(gd, SVC_RPC_GSS_FLAG_NONE); > > req->rq_auth = &svc_auth_none; > > > > break; > > -- > > 1.8.3.1 > > > > > > > -- > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > -- > Matt Benjamin > CohortFS, LLC. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://cohortfs.com > > tel. 734-761-4689 > fax. 734-769-8938 > cel. 734-216-5309 > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Help!!! nfs client can't see any buckets when client mount on nfs-ganesha rgw
Hi,
I don't notice anything offhand. If mount succeeded, the most likely thing is
that the Access Key you've used can't see them. Can you create new
buckets/objects?
Matt
- "yiming xie" wrote:
> env :centos7, nfs-ganesha 2.3, jewel
> nfs server ：192.168.77.61
> 1.cmake && make&& make install
> 2.vi /etc/ganesha/ganesha.conf
>
> EXPORT
> {
>
> Export_ID=1;
>
>
> Path = "/";
>
>
> Pseudo = "/";
>
>
> Access_Type = RW;
>
>
> NFS_Protocols = 4;
>
>
> Transport_Protocols = TCP;
>
>
> FSAL {
> Name = RGW;
> User_Id = "testuid";
> Access_Key_Id ="N6WENRWBZJWZ9ARS1UDD";
> Secret_Access_Key = "testsecret";
> }
> }
>
>
> RGW {
> ceph_conf = "/etc/ceph/ceph.conf";
> }
>
>
> 3. cp nfs-ganesha/src/scripts/systemd/* /usr/lib/systemd/system/
>
>
> 4. systemctl start nfs-ganesha.service
> systemctl status nfs-ganesha
> nfs-ganesha.service - NFS-Ganesha file server
> Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled;
> vendor preset: disabled)
> Active: active (running) since Wed 2016-09-28 14:02:02 CST; 4s ago
>
>
>
>
> 5. client host:
> s3cmd ls
> 2016-09-22 10:29 s3://foo1209_bucket
> 2016-09-28 02:31 s3://nike_bucket
> 2016-08-10 14:07 s3://test_bucket
>
>
> sudo mount -t nfs 192.168.77.61:/ /home/cep/xx
> ls /home/cep/xx
> xx is empty. Can not see any buckets name.
>
>
> Which steps may be wrong? Wait your reply，thanks。
>
>
> --
>
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Support_ex Oops...
submit a change; I dont know if rgw needs share reservations, can we refuse them? Matt - "Frank Filz" wrote: > Crud, while working on the documentation for support_ex, just came > across a > method other folks implementing support_ex missed... > > There is the possibility that two open2() calls to open by name will > race, > resulting in two fsal_obj_handle's being created. When the MDCACHE > attempts > to cache the 2nd entry, it will detect a collision, and then it needs > to fix > things up so the 2nd object handle can be released. > > The FSAL is expected to implement the merge() method to accomplish > merging > the share reservations. > > FSAL_GLUSTER, FSAL_GPFS, and FSAL_RGW do not implement this method... > > Which means the default that does nothing will be called. This will > result > in the 2nd share reservation being dropped... > > This is not good... > > Frank > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ready for Matt's c++ patches and then tag V2.4.0!
Is RDMA feature complete for NFS3? That would be interesting. Matt - "Frank Filz" wrote: > Matt, Could you rebase and submit your c++ compile patches? > > Assuming nothing crashes and burns, I will merge those and tag V2.4.0 > by > tomorrow afternoon. > > On the concall, we discussed keeping things a bit in flux and not > formally > opening V2.5 until after Bakeathon. > > Bill has some RDMA patches he wants tested at Bakeathon. We could > include > those (though he could also just bring a branch running those to > BAT...) in > next. I suspect there will also be some performance fixes and other > bug > fixes over the next few weeks. > > We could tag V2.4.1 right after Bakeathon with anything up until that > point. > > Thanks all, > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ready for Matt's c++ patches and then tag V2.4.0!
I rebased yesterday--is there more to do? Matt - "Frank Filz" wrote: > Matt, Could you rebase and submit your c++ compile patches? > > Assuming nothing crashes and burns, I will merge those and tag V2.4.0 > by > tomorrow afternoon. > > On the concall, we discussed keeping things a bit in flux and not > formally > opening V2.5 until after Bakeathon. > > Bill has some RDMA patches he wants tested at Bakeathon. We could > include > those (though he could also just bring a branch running those to > BAT...) in > next. I suspect there will also be some performance fixes and other > bug > fixes over the next few weeks. > > We could tag V2.4.1 right after Bakeathon with anything up until that > point. > > Thanks all, > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] pre-merge for V2.4-rc6
Hi Frank, I couldn't push your branch to gerrit due to it being ahead and having some illegal email wrt my account. However, it's pushed to the linuxbox2 nfs-ganesha repo, as branch rebase-c++. thanks! Matt - "Frank Filz" wrote: > I have posted a pre-merge for V2.4-rc6, it contains the patches that > have so > far been code review +2, verif +1 as found here: > > https://review.gerrithub.io/#/q/status:open+label:Code-Review%253D2+project: > ffilz/nfs-ganesha+branch:next > > I have posted to my ffilz githib next branch: > > https://github.com/ffilz/nfs-ganesha/commits/next > > The primary intent is for Matt to do a rebase of his C++ changes, but > if > anyone else wants to throw mud at it and see what sticks, we have a > day or > two for additional patches before we have to close down 2.4.0. > > NOTE: This branch is subject to rebasing or other git history > mangling > things, that's why I'm NOT pushing it to nfs-ganesha/next... > > Thanks > > Frank > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] No merge this week - and entering rc phase
Sounds good, thanks Frank! Matt - "Frank Filz" wrote: > Sorry, with a short week for vacation yesterday and today, I ran out > of time > to get a merge out. Will shoot for a merge Monday. > > And that merge will be rc1. This primarily is a statement that we are > seeking to close in on a code base we can tag as 2.4.0. For the most > part, > this means no more feature type work, just bug fixes, stabilization, > and > addressing Coverity and valgrind issues. Individual FSALs have some > additional flexibility. > > If there's some significant work effort hovering out there that you > need > into 2.4, please let us know asap so we can accommodate. > > Thanks > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] FSAL locks implementation
inline - "Frank Filz" wrote: > From: kanishk rastogi [mailto:kanishk...@gmail.com] > Sent: Tuesday, July 26, 2016 10:22 PM > To: Frank Filz > Cc: Soumya Koduri ; > nfs-ganesha-devel@lists.sourceforge.net > Subject: Re: [Nfs-ganesha-devel] FSAL locks implementation > > > > > If the only access to the files is via Ganesha, and the filesystem is > not clustered, then no, there would be no benefit to pushing the locks > into the FSAL. that would be rather "special" but, yea > > Matt -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] multi-fd
VFS? Matt - "Marc Eshel" wrote: > Do we have an FSAL that implemented multi-fd ? > > > > From: "Frank Filz" > To: Marc Eshel/Almaden/IBM@IBMUS > Cc: > Date: 07/25/2016 04:10 PM > Subject:RE: multi-fd > > > > > Why do we have reopen2 as one of the multi-fd support, I thought > that > one > > of the resones for multi fd is so we don't have to reopen files when > we > get > > differnet/conflicting open options. > > Reopen2 is for open upgrade/downgrade. > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > > > > > -- > What NetFlow Analyzer can do for you? Monitors network bandwidth and > traffic > patterns at an interface-level. Reveals which users, apps, and > protocols are > consuming the most bandwidth. Provides multi-vendor support for > NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity > planning > reports.http://sdm.link/zohodev2dev > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports.http://sdm.link/zohodev2dev ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Announce Push of V2.4-dev-26
:) - "Marc Eshel" wrote: > We are making progress, now we return the file layout attribute >197.618970099 9141 TRACE_GANESHA: [work-65] > nfs4_FSALattr_To_Fattr > :NFS4 :F_DBG :Encoded attr 62, name = FATTR4_FS_LAYOUT_TYPES > > but we fail on the first layout get >210.125317087 9147 TRACE_GANESHA: [work-71] nfs4_Compound :NFS4 > :DEBUG :Request 2: opcode 50 is OP_LAYOUTGET >210.137484089 9147 TRACE_GANESHA: [work-71] nfs4_Compound :NFS4 > :M_DBG :NFS4: MID DEBUG: Check export perms export = 00f0 req = > 0040 >210.149620502 9147 TRACE_GANESHA: [work-71] state_add :RW LOCK > :CRIT > :Error 35, write locking 0x7f5e040012b0 (&obj->state_hdl->state_lock) > at > /nas/ganesha/new-ganesha/src/SAL/nfs4_state.c:299 > > Marc. > > > > From: "Frank Filz" > To: > Date: 07/22/2016 02:16 PM > Subject:[Nfs-ganesha-devel] Announce Push of V2.4-dev-26 > > > > Branch next > > Tag:V2.4-dev-26 > > Release Highlights > > * A variety of small fixes > > * RGW: Add 3 new config options > > Signed-off-by: Frank S. Filz > > Contents: > > 5ba03b2 Frank S. Filz V2.4-dev-26 > 77c71ae Marc Eshel GPFS_FSAL: Use a shorter file handle. > a910381 Swen Schillig [valgrind] memory leak in mdcache_exp_release() > 1c45f6a Matt Benjamin rgw: add 3 new config options > 93631a9 Malahal Naineni Chomp tailing slash from pseudopath > 036703e Kaleb S KEITHLEY misc fsals: 32-bit fmt strings, gcc-6.1 > possible > uninit'd use > a336200 Soumya Koduri FSAL_GLUSTER/Upcall: Change poll interval to > 10us > 0923be1 Soumya Koduri FSAL_GLUSTER: Coverity fixes > e9f4d55 ajay nair Fixes error in sample configuration file of FSAL > ZFS > f7c37f7 ajay Removed warnings in make by solving conflicts from > cong_yacc.y > 7d67e10 Daniel Gryniewicz Clang fix - don't double initialize > 9ec03b6 Daniel Gryniewicz Coverity fixes > d3eff0f Daniel Gryniewicz MDCACHE - stack all the pNFS export ops > f75b336 Frank S. Filz NFS4: Actually set fsid and fileid in the > returned > attributes > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > What NetFlow Analyzer can do for you? Monitors network bandwidth and > traffic > patterns at an interface-level. Reveals which users, apps, and > protocols > are > consuming the most bandwidth. Provides multi-vendor support for > NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity > > planning > reports.http://sdm.link/zohodev2dev > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > > > > > -- > What NetFlow Analyzer can do for you? Monitors network bandwidth and > traffic > patterns at an interface-level. Reveals which users, apps, and > protocols are > consuming the most bandwidth. Provides multi-vendor support for > NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity > planning > reports.http://sdm.link/zohodev2dev > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports.http://sdm.link/zohodev2dev ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Remove attributes from fsal_obj_handle, copy them out on create, and the Change attribute
++ 1. did you really mean you think it should always be based on ctim? 2. we should probably allow FSAL's to override Ganesha's default mtime behavior, e.g., for AFS or DCE every vnode has a monotonically increasing version number (that's called out in RFC5661 or an errata by David Noveck, I can't recall which) Matt - "Frank Filz" wrote: > I have pushed my latest version of patches that includes removing the > attributes from the fsal_obj_handle, and also copy them out on creates > and > lookups. > > I am having issues with pynfs SATT15 test case. Something is causing > the > change attribute to roll back, such that it doesn't appear to change. > > Ganesha is faking things up to make this test pass, but I wonder, we > currently base change attribute on the greater of mtim or ctim. My > suspicion > is that it should only be based on ctim, but even doing that is not > sufficient to make SATT15 pass without the code inside the #if 0 in > mdcache_refresh_attrs that prevents rolling back change. > > I think we need to have a discussion about how to best maintain a > change > attribute. > > Thanks > > Frank > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in > San > Francisco, CA to explore cutting-edge tech and listen to tech > luminaries > present their vision of the future. This family event has something > for > everyone, including kids. Get more information and register today. > http://sdm.link/attshape > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San Francisco, CA to explore cutting-edge tech and listen to tech luminaries present their vision of the future. This family event has something for everyone, including kids. Get more information and register today. http://sdm.link/attshape ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Challenges with FSAL_MDCACHE
Hi Frank, - "Frank Filz" wrote: > As I am digging more into what actually happens with FSAL_MDCACHE, I > am > finding some structural issues that are probably best resolved by > hastening > the migration to the support_ex (multiple-fd) API. > > The biggest issue that I see is the disappearance of the content_lock > means > that the assumptions that open "file descriptors" are protected during > the > operation of an I/O operation. In fact, even before an I/O operation > commences, there are atomicity problems because the code: > > 1. checks to see if the file is open in an appropriate mode > 2. if not, re-open the file in an appropriate mode > 3. perform the I/O > > The code used to: > > 1. take a read lock on the content_lock > 2. check to see if the file is open in an appropriate mode > 2a. if not, drop the read lock, take a write lock on the content_lock > 2b. check again if the file is open in the appropriate mode > 2c. if not, re-open the file in the appropriate mode > 3. perform the I/O > 4. drop the content_lock These are very good points, thanks for sorting through all this. > > One option is that we put the content lock back in, it would have to > go into > the fsal_obj_handle. Right, this works, but it's bad^tm, because Dan's lock streamlining is important for current and future performance, correct semantics for different FSALs--and most important for the user-space FSALs we're most suited for. > > The other option is to move to the support_ex API which expects the > FSAL to > protect open file resources appropriately. > > The problem with this option is that it requires significant coding > effort > across all FSALs (though it would also allow each FSAL to determine > what is > actually necessary for open file management - some might not actually > use > any resource, at least for anonymous I/O (I/O not associated with > state). Exactly. I think this is the right option, most aligned with our goals, and the intent of MDCACHE--but it will entail work. > > Another issue is the management of open_fd_count, which is used to > pressure > the LRU thread to run more frequently to close open files. This count > is > not well managed now, and additionally is not actually used with the > support_ex API. I'd like to propose we work out a mechanism where the > FSAL > is able to communicate to the MDCACHED stacked above it that it has > too many > open files and would like the LRU thread to work on addressing that. This makes sense. Can existing upcalls cover some of this? Matt > > Frank > > -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Mobile security can be enabling, not merely restricting. Employees who bring their own devices (BYOD) to work are irked by the imposition of MDM restrictions. Mobile Device Manager Plus allows you to control only the apps on BYO-devices by containerizing them, leaving personal data untouched! https://ad.doubleclick.net/ddm/clk/304595813;131938128;j ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Ganesha tcp tuning question.
Hi Malahal, IIRC, we're -mostly- relying on environmental settings. Improvements welcome. Matt - "Malahal Naineni" wrote: > We seem to have some network (tcp/ip) related issues, one of the guys > asked "Does Ganesha use tcp autotuning? Or fixed buffer?". Does > anyone > know answer to this question? > > Thank you in advance. > > Regards, Malahal. > > > -- > Mobile security can be enabling, not merely restricting. Employees > who > bring their own devices (BYOD) to work are irked by the imposition of > MDM > restrictions. Mobile Device Manager Plus allows you to control only > the > apps on BYO-devices by containerizing them, leaving personal data > untouched! > https://ad.doubleclick.net/ddm/clk/304595813;131938128;j > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Mobile security can be enabling, not merely restricting. Employees who bring their own devices (BYOD) to work are irked by the imposition of MDM restrictions. Mobile Device Manager Plus allows you to control only the apps on BYO-devices by containerizing them, leaving personal data untouched! https://ad.doubleclick.net/ddm/clk/304595813;131938128;j ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Topic for discussion - Out of Memory Handling
I think this is probably the best approach for now, ++. Matt - "Frank Filz" wrote: > From: Swen Schillig [mailto:s...@vnet.ibm.com] > > Regardless of what's decided on how to react to out of mem > conditions, we > > must check and detect them, fast and reliable, always. > > It is not acceptable to silently accept such a condition and risk to > crash or > > modify other memory areas. > > Yes, we should check them. My idea is in gsh_alloc and friends, to > test for NULL return from malloc and friends, and if they fail, to > then log a message and abort(). Then ANY code that calls gsh_alloc and > friends need not (and should not) check for NULL return (checking for > NULL return would thus just be wasted code that will never be > executed). There are other library calls that also allocate memory, > and we DO need to make sure we check in all of those places. > > > Besides, the argument that an early abort would be advantageous for > > memory leak conditions is hopefully a joke. > > Memory leaks must be "closed" regardless of whether we abort or try > to > > recover. > > I think the major goal must be system stability and reliability. > > It is not a sign of a reliable system if it's restarted for every > little unusual > > situation. > > It's not a joke... The reality is any complex system probably has a > memory leak. We must fix them. But loading up the code with attempts > to keep running in the face of ENOMEM caused by a memory leak will > just HIDE the memleak by being mostly stable (or causing a more > obscure error as the system tries to limp along). > > Frank > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Announce Push of V2.3-rc5
Hi, - "William Allen Simpson" wrote: > Updated to today's ganesha V2.3-rc5. > > Frank, could you please change nfs-ganesha/ntirpc to default > branch duplex-12? Matt forgot, and I've reminded him twice. Ok, fine, I've done this. I don't think this was a blocker. > > Dan, you forgot to push your dang/ntirpc external to both > linuxbox2/ntirpc Not assured by us to be kept at parity with the upstream repo, it's a 3rd-party repo, wrt nfs-ganesha. and nfs-ganesha/ntirpc duplex-12 I merged Dan's PR Thursday AM eastern. What exactly is not merged? -- RDMA > cmake and compile fail. Um... Will revisit next week, I guess. Matt > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Missing files from directory with large entries
Hi Malahal,
I'll send something in the next week or so.
Matt
- "Malahal Naineni" wrote:
> This is an old email but any progress on this?
>
> Regards, Malahal.
>
> Frank Filz [ffilz...@mindspring.com] wrote:
> >Hmm, that looks like a constant that needs to be replaced with a
> config
> >value...
> >
> >
> >
> >That does appear to be an absolute hard limit on directory size
> (and if
> >there are multiuple READDIR operations in progress, it would seem
> like you
> >need even more).
> >
> >
> >
> >One option would be to make it some percentage of the number of
> cache
> >inodes allowed if we didn't want to introduce a new config
> option.
> >
> >
> >
> >Frank
> >
> >
> >
> >From: Krishna Harathi [mailto:khara...@exablox.com]
> >Sent: Monday, July 27, 2015 5:53 PM
> >To: nfs-ganesha-devel
> >Subject: [Nfs-ganesha-devel] Missing files from directory with
> large
> >entries
> >
> >
> >
> >In a test-case, we are finding that a few files(5) are missing
> sometimes
> >from a directory with > 100K entries, while listing with "ls".
> >
> >
> >
> >Debug logging is showing multiple entries with the following.
> >
> >
> >
> >nfs-ganesha-5025[work-15] cache_inode_avl_insert_impl :INODE
> :DEBUG
> >:inserted new dirent on entry=0x6780d300 cookie=1925138672
> collisions 1
> >
> >
> >
> >I saw that in cache_inode_avl_insert_impl(), if I double this
> value of the
> >check below, the problem
> >
> >goes away apparently.
> >
> >
> >
> > if ((!node) && (avltree_size(c) > 65535)) {
> > /* ie, recycle the smallest deleted entry */
> > node = avltree_first(c);
> > }
> >
> >
> >
> >
> >
> >Any insight into this problem is appreciated, we are using
> Ganesha 2.1.0.
> >I also looked at recent changes/fixes in this area in the current
> code I
> >don't think the recent code addresses this issue, but I may be
> wrong.
> >
> >Regards.
> >
> >Krishna Harathi
>
> >
> --
>
> > ___
> > Nfs-ganesha-devel mailing list
> > Nfs-ganesha-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>
>
> --
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Is the Export ID visible to NFS clients?
Hi Dirk, Yes, Ganesha creates an expanded handle from your FSAL private handle and its own steering data. See the logic in support/nfs_filehandle_mgmt_c. Matt - "Dirk Jagdmann" wrote: > > The exportid is not meaningful to the client, however, since it is > part of > > the handle, and the clients expect handles to be unchanging for a > specific > > file, changing the exportid for a given client's VM file would > definitely > > cause problems. > > But when I write my own FSAL, I do construct my own wire handle. If I > don't > include the Export ID in my own wire handle (my file has a unique ID, > unique > even across different file systems), will be client still see the > Export ID? > Does the server software add something to the wire handle, what the > FSAL doesn't > put in there? > > -- > ---> Dirk Jagdmann > > http://cubic.org/~doj > -> http://llg.cubic.org > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH v2] ntirpc: implement setting CPU affinity of sender threads
Hi, I think we're not staying completely on-point. I agree with Wei's suggestion that different transports likely need different affinity strategies, and presumably you do too. There is probably no big issue with affinity syscalls--we should not do them inline with work, but different pools (e.g.) could resolve that. The RPC subsystem does has at least one group of threads (sender) that TCP consumers care about, so, it's legit to look at affinity for it, I think. Matt - "William Allen Simpson" wrote: > On 9/28/15 4:48 AM, Wei Fang wrote: > > On 2015/9/25 2:06, William Allen Simpson wrote: > >> Do any of these make a system call? Then they are not useful. > >> This thread pool has been designed to have only one system call > >> per invocation -- or better yet continue processing the next > >> task without a system call. > > > > Yes, there has a system call in shed_setaffinity. But I don't > understand > > why they can't make a system call. > > This is a userland program, the whole point is to run with limited > system calls. Otherwise, we'd just use the Linux nfsd > > > > Do you mean that it may lower > > performance of threads? I think it depends on how much work the > thread > > is going to do. > > > There seems to be some confusion here. Every system call limits > the performance. It has nothing what-so-ever to do with how much > work the thread is going to do. > > If the thread has a little to do, you'll have serious latency > issues and cap the ops-per-second. This is already the case, > and we're trying to fix it. > > If the thread has a lot to do, the system calls may be amortized > over the longer work load. But the latency remains. > > > > I do have tested it on Linux, and obtained some performance > benefits. > > > What's the increase in ops-per-second? What environment? > > > > May those users use the same work pool simultaneously? I thought > every > > user has it's own pool, and every pool can set it's own CPU > affinity > > separately. > > > All users share this same work pool. So each connection will > grab a worker. And each request over the connection will > grab another worker, although that worker should continue > using the same thread for many subsequent requests. > > What happened is you tested without RDMA (the primary user of > this pool). Moreover, likely many requests were over one > connection. So there was only one main user. But that's not a > typical workload. > > > >> Your current code has them oddly counting themselves. No idea > >> how this would work. > > > > I count the threads created in the pool and use it to split the > threads to > > different CPUs. There's problem I mentioned on the TODO list in my > patch, > > and should do more work on this later. > > > Looks like you're depending on serendipity for affinity. > > As I mentioned in the next message, perhaps combined with > Malahal's lanes gives us some affinity per connection. > > But my biggest objection is the additional system overhead. > Pin the CPU assignment algorithmically at thread creation. > Minimize system calls. -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH] ntirpc: remove redundant assignment of ->warnx
Hi, The svc_init() call is intended to precede any call to tirpc_control, and we do want this call to default-initialize. Regards, Matt - fangw...@huawei.com wrote: > From: Wei Fang > > ->warnx has been assigned defaultly. If we assigned here when > SVC_INIT_WARNX unsetted, ->warnx assigned by tirpc_control would be > overwrited to default, which we don't want to happen. > > Signed-off-by: Wei Fang > --- > src/svc.c |2 -- > 1 files changed, 0 insertions(+), 2 deletions(-) > > diff --git a/src/svc.c b/src/svc.c > index 1ff1c84..c91386c 100644 > --- a/src/svc.c > +++ b/src/svc.c > @@ -147,8 +147,6 @@ svc_init(svc_init_params *params) > > if (params->flags & SVC_INIT_WARNX) > __ntirpc_pkg_params.warnx = params->warnx; > - else > - __ntirpc_pkg_params.warnx = warnx; > > /* svc_vc */ > __svc_params->xprt_u.vc.nconns = 0; > -- > 1.7.1 -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] rc2 staged
Hi, If we permit conditional linkage against a static libntirpc, I would strongly prefer that it be explicit (config option that defaults to OFF). I don't think static linkage actually adds much to ease of use, since everybody should understand platform rules for shlibs. Matt - "Daniel Gryniewicz" wrote: > Correct, however the non-VFS FSALs all need external libraries (such > as libcephfs for Ceph) that would not be in lib/ganesha. > > Also, since FSALs are dlopen()'d by full path, it's not necessary to > have lib/ganesha in your LD_LIBRARY_PATH. > > Is it worth making a static version of libntirpc that Ganesha can > optionally link against? It would never be deployed that way by a > distro, but it might ease Ganesha development. > > My hope for my personal work is to install libntirpc separately in my > system, and not have to build it during Ganesha development at all, > unless I'm actively working on it. > > > Daniel > > On Fri, Sep 11, 2015 at 9:00 AM, Malahal Naineni > wrote: > > FSALs are typically installed under ganesha directory, so if he is > using > > a non-standard location, he probably is already using ".../ganesha" > > path, now he needs to add "..." directory as well, correct? > > > > Regards, Malahal. > > > > Daniel Gryniewicz [d...@redhat.com] wrote: > >> It's installed in /lib[64]. If you're not installing into > a > >> standard location, you will need to add /lib[64] to your > >> LD_LIBRARY_PATH, but it probably should have been there before. > >> Certainly that would have been necessary if any non-VFS FSALs were > in > >> use. > >> > >> ntirpc used to be statically linked, it's dynamically linked now > >> (since it's a shared library) so library path matters. > >> > >> Daniel > >> > >> On Thu, Sep 10, 2015 at 11:41 PM, William Allen Simpson > >> wrote: > >> > On 9/10/15 9:52 PM, William Allen Simpson wrote: > >> >> Probably something I did wrong. It's the first time I've > >> >> downloaded from gerrithub rather than posted -- but I'm going > to > >> >> have to re-build my entire repository and try again. Tomorrow. > >> >> > >> > Bah. Stayed up tonight and rebuilt my repository and tried > again. > >> > > >> > Dan's commit message has a grammar error: it's -> its. > >> > > >> > Tested on top of my own https://review.gerrithub.io/#/c/246391/ > >> > (which is also ready to go as discussed on the call today). > >> > > >> > Build and link OK with USE_NFS_RDMA = ON. > >> > > >> > But couldn't test, the shared library must be somewhere else > now: > >> > > >> > :/home/bill/rdma/install/lib:/home/bill/rdma/install/lib64 > >> > root@simpson:~/rdma/install# ./bin/ganesha.nfsd > >> > ./bin/ganesha.nfsd: error while loading shared libraries: > libntirpc.so.1.3: cannot open shared object file: No such file or > directory > >> > root@simpson:~/rdma/install# ls bin > >> > ganesha.nfsd > >> > root@simpson:~/rdma/install# ls lib > >> > ls: cannot access lib: No such file or directory > >> > root@simpson:~/rdma/install# ls lib64 > >> > ganesha libntirpc.so libntirpc.so.1.3 libntirpc.so.1.3.0 > >> > root@simpson:~/rdma/install# ls lib64/ganesha/ > >> > libfsalgpfs.solibfsalpanfs.solibfsalvfs.so > >> > libfsalgpfs.so.4 libfsalpanfs.so.4 libfsalvfs.so.4 > >> > libfsalgpfs.so.4.2.0 libfsalpanfs.so.4.2.0 libfsalvfs.so.4.2.0 > >> > libfsalnull.solibfsalproxy.solibntirpc.so > >> > libfsalnull.so.4 libfsalproxy.so.4 > >> > libfsalnull.so.4.2.0 libfsalproxy.so.4.2.0 > >> > > >> > > >> > > -- > >> > ___ > >> > Nfs-ganesha-devel mailing list > >> > Nfs-ganesha-devel@lists.sourceforge.net > >> > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > >> > >> > -- > >> ___ > >> Nfs-ganesha-devel mailing list > >> Nfs-ganesha-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > >> > > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Lost wakeup with nfs_rpc_dequeue_req() and nfs_rpc_enqueue_req()
I'm not sure I believe that there is hang case, but if there is, the simple fix for it is to bound the offending wait. I have a branch that uses a lock-free bounded mpmc queue based work we did last year in the Ceph codebase. I have something working but not efficiently, and I haven't have time to explore the issues/tuning. Any dramatic change here should probably not be for 2.3. Matt - "Malahal Naineni" wrote: > William Allen Simpson [william.allen.simp...@gmail.com] wrote: > > On 9/8/15 9:03 PM, Malahal Naineni wrote: > > > The nfs_rpc_enqueue_req is called by the producer and > > > nfs_rpc_dequeue_req is called by the consumer. Here is the high > level > > > view of those functions. > > > > > > In particular, if a consumer finds no request in the queue, and > then > > > attempts to go to sleep, but before he adds himself to the wait > list > > > (before step 2b), if producer adds another request at that > instant, the > > > producer would find the wait list empty and couldn't wake up any > > > consumer. > > > > > I've not paid much attention to this, as my goal is to eliminate > it, > > but isn't there a lock involved? That would make all the > difference. > > Locks are there but they fail to ensure the race! How are you going > to > eliminate it? Wondering if having a lane for each worker thread is a > good idea here. This would make it SPSC (single producer single > consumer) which should be easy for a lockless code. > > Regards, Malahal. > > > -- > Monitor Your Dynamic Infrastructure at Any Scale With Datadog! > Get real-time metrics from all of your servers, apps and tools > in one place. > SourceForge users - Click here to start your Free Trial of Datadog > now! > http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Lost wakeup with nfs_rpc_dequeue_req() and nfs_rpc_enqueue_req()
>From memory.
The code isn't trying to eliminate that race, no. If/when it arises, and all
threads are idle, then the timeout will ensure progress.
In particular, if there is no work for -any- threads, it is desirable for at
least
most to go idle. As work ramps up, threads will be woken.
The next optimization, I would intuit, might be to have a leader that is
resistant
to idle.
Matt
- "Malahal Naineni" wrote:
> The nfs_rpc_enqueue_req is called by the producer and
> nfs_rpc_dequeue_req is called by the consumer. Here is the high level
> view of those functions.
>
> In particular, if a consumer finds no request in the queue, and then
> attempts to go to sleep, but before he adds himself to the wait list
> (before step 2b), if producer adds another request at that instant,
> the
> producer would find the wait list empty and couldn't wake up any
> consumer.
>
> Later the consumer would add himself to the wait list and goes to
> sleep
> (in a loop actually due to timedwait call. This will continue if
> there
> was no other request coming in. Am I missing something?
>
> nfs_rpc_dequeue_req()
> {
> 1. request = nfs_rpc_consume_req()
> 2. if (request == NULL) { /* no work to do */
> a) set Wqe_LFlag_WaitSync flag
> b) add itself to the wait list
> c) while not Wqe_LFlag_SyncDone {
> pthread_cond_timedwait(5 seconds)
> }
> goto step 1; /* woke up with SyncDone */
> }
> }
>
> nfs_rpc_enqueue_req()
> {
> 1. add request to the list
> 2. if (wait list is not empty) {
> a) pick up first waiting thread.
> b) set Wqe_LFlag_SyncDone
> c) if Wqe_LFlag_WaitSync() pthread_cond_signal().
> }
> }
>
>
> --
> Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
> Get real-time metrics from all of your servers, apps and tools
> in one place.
> SourceForge users - Click here to start your Free Trial of Datadog
> now!
> http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Fridge worker pool to ntirpc?
Hi, inline - "Malahal Naineni" wrote: > > perf record shows that too much time is spent in malloc/free > functions. > Reported functions are alloc_nfs_request, alloc_nfs_res, and few > objects > in src/xdr_ioq.c file. alloc_nfs_res seems thread specific, so could > be > allocated one per thread. If we can make other pools lockless > (instead > of malloc), that would be great! We have work-in-progress changes on former-cohortfs Ceph branches that uses, MCAS lock-free dictionaries and their lock-free allocator caches. The allocators can be used standalone, but we were interested in exploring both potentially for use in Ganesha. Our Ceph-based inode cache simulator which used these could sustain over 2M create/s, IIRC. Marcus did some work on the pool allocator to upgrade its epoch-based reclamation logic, to NEBR (discussed here http://csng.cs.toronto.edu/publication_files//0159/jpdc07.pdf TLS would be probably work for some things, as you say. Other things we've used in the last year or so: 1. pool interface exposed in Mellanox Accelio library--initially was special purpose, but has been generalized (partly in our Accelio Ceph work); It has special capabilities (registration) for use with RDMA. I don't know if this is mature as MCAS, but would be work a look. 2. for some key allocations in our Ceph OSD, we used a lock-free bounded queue of structures--this appeared to very work well, for large objects. For small allocations, jemalloc was much faster. Regards, Matt > > Regards, Malahal. > > -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Monitor Your Dynamic Infrastructure at Any Scale With Datadog! Get real-time metrics from all of your servers, apps and tools in one place. SourceForge users - Click here to start your Free Trial of Datadog now! http://pubads.g.doubleclick.net/gampad/clk?id=241902991&iu=/4140 ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] testing shutdown on rdma-was
Thanks, guys!
Matt
- "William Allen Simpson" wrote:
> On 8/3/15 5:13 AM, Dominique Martinet wrote:
> > Hi,
> >
> > William Allen Simpson wrote on Mon, Aug 03, 2015 at 04:45:33AM
> -0400:
> >> I've pushed some ntirpc changes to shutdown that might fix
> >> the problems reported by Dominique. Ignore the RDMA, I used
> >> an existing branch for convenience.
> >>
> >> To test, you'd fetch the linuxbox2 versions of nfs_genesha
> >> and ntirpc -- as in git remote add followed by git fetch --all
> >> for both parts respectively.
> >
> > Just to make sure, I picked:
> > linuxbox2/nfs-ganesha branch rdma-was e.g. 9a00e909f0bf ("Integrate
> > Mooshika into RPC RDMA")
> >
> > with its appropriate submodule (matched the submodule id and the
> branch
> > rdma-was):
> > ebcc64fd425 ("Integrate Mooshika into RPC RDMA")
> >
> That's them! They won't be pushed, the actual fixes precede them.
>
>
> >
> > Shutdown worked well, I'll run it again just to make sure but
> > considering it fixed now, thank you.
> >
> Hallelujah! Always a bit difficult to figure out without a
> reproducible error.
>
>
> --
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] event channel rbtree over-caching
Bill,
Please don't make this change. We can discuss tuning changes at your leisure.
Thanks,
Matt
- "William Allen Simpson" wrote:
> As I've been fixing the problem with ntirpc *_cleanup(), have
> discovered that it all passes through ntirpc svc_rqst.[ch].
>
> Trying to grok it, it's all rbtrees.
>
> But noticed that there's a fixed cache size of 8192 -- it
> logs a message telling me it should be a small prime.
>
> Then noticed there are 7 partitions, each of which have
> this cache size of 8192.
>
> Then noticed the svc_rqst_new_evchan() that sets it up is
> called in only 2 places:
>
> nfs_rpc_dispatcher_thread.c, nfs_Init_svc()
>
> for (ix = 0; ix < N_EVENT_CHAN; ++ix) {
> rpc_evchan[ix].chan_id = 0;
> code = svc_rqst_new_evchan(&rpc_evchan[ix].chan_id,
> NULL /* u_data */,
> SVC_RQST_FLAG_NONE);
>
> svc_rqst.c, xprt_register()
>
> /* Create a legacy/global event channel */
> if (!(__svc_params->ev_u.evchan.id)) {
> code =
> svc_rqst_new_evchan(&(__svc_params->ev_u.evchan.id),
> NULL /* u_data */ ,
> SVC_RQST_FLAG_CHAN_AFFINITY);
>
> ===
>
> Conclusion: 8 event channels don't need 7*8192 cached slots,
> nor do they need rbtrees for "fast" lookup.
>
> Moreover, these never seem to be looked up, as SVCXPRT (xp_ev)
> points to the channel.
>
> Rather, the list is checked for old transports to cleanup.
> __svc_clean_idle2()
>
> Transports are sorted by memory address.
>
> I'm converting to linked lists, which will be very easy to
> clean up. Speak now, or forever hold your peace.
>
>
> --
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] event channel re-arming, [un]hooking
Addressing this (and the sequence we'll address it in) is already in the work plan, right? Matt - "William Allen Simpson" wrote: > Every time there's an event, the current code re-arms the fd > from the epoll. > > That means 3 system calls for each incoming event. Hopefully, > I'm not doing it that way for RDMA events > > What's the reason for re-arming? Also, EPOLLONESHOT? > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] inode cache
Hi Marc, Probably. I was writing to malahal in irc that we have code changes that will reduce lock contention for xprt->xp_lock a LOT, and more changes coming that will address latency in dispatch and reduce locking in SAL. The first of those changes will be coming in hopefully still this week. One thing I think could be out of whack is the lru lane selector, I cand send a hotfix if we have a skewed object-lane distribution in LRU. Alternatively, there is tuning for #partitions and the size of a per-partition hash table in both the cache_inode "hash" and HashTable (used a lot of other places) which could apply, if that's the bottleneck. Do you have a trivial reproducer to experiment with? Matt - "Marc Eshel" wrote: > Hi Matt, > I see bad perfromance when stating milloins of files, the inode cache > is set to 1.5 million. Are there any configuration changes that I can > make to the inode cache, even code changes of some hard coded values > that will help with performance of big nuber of files? > Thanks , Marc. -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Multiple-fd work
i.e., I like that it captures the idea (in multi-fd already) to allocate all at once, and this is how I would have intuited we move to opaque user data. I think we can deal with whatever workaround solves locking currently required for that object (e.g., Adam's suggestion to call up into SAL layer in some cases), esp. since iirc an eventual step in Dan's MDCACHE work involved lock reorganization, and a lot things will likely shake out from that? Matt ----- "Matt W. Benjamin" wrote: > This feels to be on the right track. > > Matt > > - "Daniel Gryniewicz" wrote: > > > On 07/28/2015 03:51 PM, Frank Filz wrote: > > > > > > > > I was avoiding the free standing blob (at least for NFS, 9p it > was > > not so easy), so for NFS state_t objects, the fsal_fd is allocated > > along with the state_t. > > > > > > The FSAL has to be able to indicate how much space it needs. > > > > > > Yes, we COULD pass the state_t (still needs to allocate some > space > > for FSAL somehow) instead of the fsal_fd. The question I have is > what > > in there is of value to the FSAL. If there really is something of > > value, then sure, let's pass the state_t. Without something of > value > > though, I'm wary of exposing state_t stuff to the FSAL (for one > thing > > it's protected by the state_lock, which the FSAL normally doesn't > have > > access to). > > > > > > However, there is another hitch... When doing an open/create, we > > don't have a cache entry or state_t yet (which currently means > fsal_fd > > is actually something that needs to be able to be copied). > > > > > > > In my MDCACHE work, the state_t is completely allocated by the FSAL, > > > since it has to (potentially) store it as well, if there's no > MDCACHE > > > > layer for that FSAL. So, it can just layer state_t, like it layers > > > fsal_obj_handle. The advantage of this is that it looks and behaves > > > just like object handles, exports, and other things that the FSAL > > already layers. Under this paradigm, open/create would return the > > state_t as an out param anyway. > > > > If this is interesting, the code to call into the FSAL to get/put > the > > > > state is fairly separable, and can go in soon. > > > > Dan > > > > > > > -- > > ___ > > Nfs-ganesha-devel mailing list > > Nfs-ganesha-devel@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > -- > Matt Benjamin > CohortFS, LLC. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://cohortfs.com > > tel. 734-761-4689 > fax. 734-769-8938 > cel. 734-216-5309 > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Multiple-fd work
This feels to be on the right track. Matt - "Daniel Gryniewicz" wrote: > On 07/28/2015 03:51 PM, Frank Filz wrote: > > > > > I was avoiding the free standing blob (at least for NFS, 9p it was > not so easy), so for NFS state_t objects, the fsal_fd is allocated > along with the state_t. > > > > The FSAL has to be able to indicate how much space it needs. > > > > Yes, we COULD pass the state_t (still needs to allocate some space > for FSAL somehow) instead of the fsal_fd. The question I have is what > in there is of value to the FSAL. If there really is something of > value, then sure, let's pass the state_t. Without something of value > though, I'm wary of exposing state_t stuff to the FSAL (for one thing > it's protected by the state_lock, which the FSAL normally doesn't have > access to). > > > > However, there is another hitch... When doing an open/create, we > don't have a cache entry or state_t yet (which currently means fsal_fd > is actually something that needs to be able to be copied). > > > > In my MDCACHE work, the state_t is completely allocated by the FSAL, > since it has to (potentially) store it as well, if there's no MDCACHE > > layer for that FSAL. So, it can just layer state_t, like it layers > fsal_obj_handle. The advantage of this is that it looks and behaves > just like object handles, exports, and other things that the FSAL > already layers. Under this paradigm, open/create would return the > state_t as an out param anyway. > > If this is interesting, the code to call into the FSAL to get/put the > > state is fairly separable, and can go in soon. > > Dan > > > -- > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Multiple-fd work
Hi, - "Frank Filz" wrote: > > The problem is that "fd"s are an internal detail that we do not want > leaking > > out into the general server. > > > > There are several reasons for this adduced in my and Adam's comments > in > > gerrit. > > > > I may be missing something, but what is the generic (non-VFS or few > other > > FSAL-specific) concept that is being represented here? Is it an > OPEN > > complex? > > From what I can tell, this is useful to more than just a few FSALs. > Now maybe "file descriptor" is not the best choice (since FSAL_PROXY > is going to use the "thing" to stash information about stateids from > the upstream server (which will be different though perhaps related to > Ganesha's stateids)). I think well-defining the abstraction is key. Adam didn't intuit that the issue was just one of naming, although I agree, I the name "fd" may be distracting. But I am more concerned about interfaces. I share Dan's (and perhaps Adam's) intuition that we'd like to unify around a smaller number of objects, and in making what the general server is doing more transparent to FSAL, when that can make for a cleaner interface (we're not trying to hide state from FSAL, but we might be interested in avoiding the need to allocate more free-standing blobs. So I liked Dan's suggestion to pass states to FSAL calls. Could we talk more about that idea? > > It's context to associate with share reservations (NFS 4 OPEN, NLM > SHARE) and lock state. Different FSALs will have different context, > but perhaps not all FSALs will need the context. > > But having a "few" FSALs need an API also isn't a good reason to turn > down that API. > > I am working to make this "thing" as generic as makes sense. If > there's some FSAL that has needs that I haven't considered, then > please share those needs rather than shooting down ideas. I'm very happy to share needs, but I care about implementation decisions, too. Matt > > Frank > > > > > Hi Frank, list. > > > > > > > > Frank Filz wrote on Mon, Jul 27, 2015 at 09:33:54AM -0700: > > > > > FSAL_LUSTRE actually also uses fcntl to get POSIX locks, so it > has > > > the > > > > > same issues as FSAL_VFS. > > > > > > > > LUSTRE would actually like one fd per open or equivalent if we > can > > > reap them > > > > efficiently, for the same reason as GPFS -- sharing a fd kills > > > readahead > > > > behaviour and lustre needs the client to do readahead and has > us > > > disable > > > > readahead disk-side, so ganesha performs quite worse with > multiple > > > clients. > > > > > > > > Well, I guess that's close enough to what we're doing anyway. > > > > > > Fds that are associated with NFS 4 OPEN or NFS 3 SHARE will be > closed > > > appropriately. If the FSAL didn't need to keep them open, we > could > > > enhance the cache inode LRU to be able to consider pinned entries > for > > > advisory file close. The FSAL could also try to do something to > keep > > > track and I dunno, LRU garbage collection of idle file > descriptors > > > might actually belong in the FSAL anyway instead of cache inode. > > > > > > > > Now I don't know about other FSALs, though I wouldn't be > surprised > > > if > > > > > at least some other FSALs might benefit from some mechanism > to > > > > > associate SOMETHING with each lock owner per file. > > > > > > > > > > So I came up with the idea of the FSAL providing the size of > the > > > "thing" > > > > > (which I have currently named fsal_fd, but we can change the > name > > > if > > > > > it would make folks feel more comfortable) with each Ganesha > > > stateid. > > > > > And then to bring NFS v3 locks and share reservations into > the > > > > > picture, I've made them able to use stateids (well, state_t > > > > > structures), which also cleaned up part of the SAL lock > interface > > > > > (since NFS v3 can hide it's "state" value in the state_t and > stop > > > overloading > > > > state_t *state... > > > > > > > > 9P has a state_t per fid (which will translate into per open), > so > > > that should > > > > work well for us as well. > > > > > > Yep, that's why I added state_t to 9P. There's actually a > complication > > > when an FSAL actually needs both share_fds and lock_fds. I will > be > > > looking into that some more. > > > > > > > > Now each FSAL gets to define exactly what an fsal_fd actually > is. > > > At > > > > > the moment I have the open mode in the generic structure, but > > > maybe it > > > > > can move into the FSAL private fsal_fd. > > > > > > > > > > Not all FSALs will use both open and lock fds, and we could > > > provide > > > > > separate sizes for them so space would only be allocated when > > > > > necessary (and if zero, a NULL pointer is passed). > > > > > > > > sounds good. > > > > > > > > > For most operations, the object_handle, and both a share_fd > and > > > > > lock_fd are passed. This allows the FSAL to decide exactly > which > > > ones > > > > > it needs (if it needs a generic "thing" for anonymous I/O
Re: [Nfs-ganesha-devel] Multiple-fd work
The problem is that "fd"s are an internal detail that we do not want leaking out into the general server. There are several reasons for this adduced in my and Adam's comments in gerrit. I may be missing something, but what is the generic (non-VFS or few other FSAL-specific) concept that is being represented here? Is it an OPEN complex? Matt - "Frank Filz" wrote: > > Hi Frank, list. > > > > Frank Filz wrote on Mon, Jul 27, 2015 at 09:33:54AM -0700: > > > FSAL_LUSTRE actually also uses fcntl to get POSIX locks, so it has > the > > > same issues as FSAL_VFS. > > > > LUSTRE would actually like one fd per open or equivalent if we can > reap them > > efficiently, for the same reason as GPFS -- sharing a fd kills > readahead > > behaviour and lustre needs the client to do readahead and has us > disable > > readahead disk-side, so ganesha performs quite worse with multiple > clients. > > > > Well, I guess that's close enough to what we're doing anyway. > > Fds that are associated with NFS 4 OPEN or NFS 3 SHARE will be closed > appropriately. If the FSAL didn't need to keep them open, we could > enhance the cache inode LRU to be able to consider pinned entries for > advisory file close. The FSAL could also try to do something to keep > track and I dunno, LRU garbage collection of idle file descriptors > might actually belong in the FSAL anyway instead of cache inode. > > > > Now I don't know about other FSALs, though I wouldn't be surprised > if > > > at least some other FSALs might benefit from some mechanism to > > > associate SOMETHING with each lock owner per file. > > > > > > So I came up with the idea of the FSAL providing the size of the > "thing" > > > (which I have currently named fsal_fd, but we can change the name > if > > > it would make folks feel more comfortable) with each Ganesha > stateid. > > > And then to bring NFS v3 locks and share reservations into the > > > picture, I've made them able to use stateids (well, state_t > > > structures), which also cleaned up part of the SAL lock interface > > > (since NFS v3 can hide it's "state" value in the state_t and stop > overloading > > state_t *state... > > > > 9P has a state_t per fid (which will translate into per open), so > that should > > work well for us as well. > > Yep, that's why I added state_t to 9P. There's actually a complication > when an FSAL actually needs both share_fds and lock_fds. I will be > looking into that some more. > > > > Now each FSAL gets to define exactly what an fsal_fd actually is. > At > > > the moment I have the open mode in the generic structure, but > maybe it > > > can move into the FSAL private fsal_fd. > > > > > > Not all FSALs will use both open and lock fds, and we could > provide > > > separate sizes for them so space would only be allocated when > > > necessary (and if zero, a NULL pointer is passed). > > > > sounds good. > > > > > For most operations, the object_handle, and both a share_fd and > > > lock_fd are passed. This allows the FSAL to decide exactly which > ones > > > it needs (if it needs a generic "thing" for anonymous I/O it can > stash > > > a generic fsal_fd in it's object_handle). > > > > They're passed with what we have and it's left to the FSAL function > to open if > > it needs something? > > One of the problem we have at the moment in VFS is fstat() racing on > the > > "generic thing", which can be closed/open in other threads. > > Do we leave it up to FSALs to protect their generic bits then ? > (since we can't > > know which FSAL call will use which bit, doing locking ourselves > will have to > > be too much for most) > > Note the FSAL_VFS fstat race has been resolved (by protecting the fd > inside the FSAL - I plan to continue this migration, allow the FSAL to > protect it's file descriptors or whatever they are). And yes, it's up > to the FSAL when it actually needs to open something. > > Note that lock_fds can get left open longer than necessary, > particularly in NFS v4.0 since there isn't a way for the client to > indicate it's done with the lock stateid (for NFS v3 the CURRENT code > will result in a close of the lock_fd anytime the lock owner drops to > zero locks on a file - I think we may want to proactively allow for > keeping the state_t around a bit longer and thus allow the FSAL the > option of also keeping the fd (or whatever) open longer). > > > > The benefit of this interface is that the FSAL can leverage cache > > > inode AVL tree and SAL hash tables without making upcalls (that > would > > > be fraught with locking issues) or duplicating hash tables in > order to > > > store information entirely separately. It also may open > possibilities > > > to better hinting between the layers so garbage collection can be > > improved. > > > > yay. (just need to make sure we "close" anything that has been left > open on > > GC, so need a common level flag to say it's used maybe?) > > The fsal_fd that are attached to state_t have a pretty defined > lifetime.
Re: [Nfs-ganesha-devel] Client specification question
Hi Krishna, I don't have and haven't seen reports about issues. Please report any problems you find. Regards, Matt - "Krishna Harathi" wrote: > Is there any read/write performance hit by using multiple CLIENT > sections in configuration to filter client access? > > > Also I assume IPv6 addresses can be given, although the configuration > examples does not mention IPv6? > > > We use Ganesha 2.1.0, will be moving to 2.2.0 soon. > > > Thanks. > > > Regards. > > > > > Krishna Harathi > -- > > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] FW: New Defects reported by Coverity Scan for nfs-ganesha
Yup, will do. Was going to do it over the weekend, but got tied up.
Matt
- "Frank Filz" wrote:
> Looks like we can clear all of these. I think they should mostly all
> be marked as intentional, our use of flags to control when lock/unlock
> is done is hard for an automated tool to follow (it's not even always
> that clear to me... I wish in some cases we would use functions like
> foo_locked that assumes the lock is held, and inline foo that takes
> the lock and calls foo_locked for those callers that don't already
> hold the lock...
>
> Matt, would you mind going into Coverity and clearing all of these?
>
> Thanks
>
> Frank
>
> > -Original Message-
> > From: William Allen Simpson
> [mailto:william.allen.simp...@gmail.com]
> > Sent: Sunday, July 12, 2015 5:06 AM
> > To: Frank Filz; 'NFS Ganesha Developers'
> > Subject: Re: FW: New Defects reported by Coverity Scan for
> nfs-ganesha
> >
> > On 7/10/15 10:45 AM, Frank Filz wrote:
> > > Looks like a bunch of new Coverity defects in the online Coverity.
> Primarily
> > ntirpc related.
> > >
> > Thanks.
> >
> > Heh, not exactly ;)
> >
> >
> > > See the online report:
> > >
> > > https://scan.coverity.com/projects/2187/view_defects
> > >
> > > -Original Message-
> > > From: scan-ad...@coverity.com [mailto:scan-ad...@coverity.com]
> > > Sent: Friday, July 10, 2015 12:56 AM
> > > To: ffilz...@mindspring.com
> > > Subject: New Defects reported by Coverity Scan for nfs-ganesha
> > >
> > >
> > > Hi,
> > >
> > > Please find the latest report on new defect(s) introduced to
> nfs-ganesha
> > found with Coverity Scan.
> > >
> > > 10 new defect(s) introduced to nfs-ganesha found with Coverity
> Scan.
> > > 12 defect(s), reported by Coverity Scan earlier, were marked fixed
> in the
> > recent build analyzed by Coverity Scan.
> > >
> > > New defect(s) Reported-by: Coverity Scan Showing 10 of 10
> defect(s)
> > >
> > >
> > > ** CID 124499: Resource leaks (RESOURCE_LEAK)
> > > /nfs-ganesha/src/libntirpc/src/xdr_ioq.c: 291 in xdr_ioq_create()
> > >
> > >
> > >
> > __
> >
> > > __
> > > *** CID 124499: Resource leaks (RESOURCE_LEAK)
> > > /nfs-ganesha/src/libntirpc/src/xdr_ioq.c: 291 in xdr_ioq_create()
> > > 285 struct xdr_ioq_uv *uv =
> > xdr_ioq_uv_create(min_bsize, uio_flags);
> > > 286 xioq->ioq_uv.uvqh.qcount = 1;
> > > 287 TAILQ_INSERT_HEAD(&xioq->ioq_uv.uvqh.qh, &uv-
> > >uvq, q);
> > > 288 xdr_ioq_reset(xioq, 0);
> > > 289 }
> > > 290
> > CID 124499: Resource leaks (RESOURCE_LEAK)
> > Variable "xioq" going out of scope leaks the storage it
> points to.
> > > 291 return (xioq->xdrs);
> > > 292 }
> >
> > False positive. xioq is a container wrapped around the XDR. The XDR
> is an
> > array of one. This C shortcut is the same as &xioq->xdrs[0].
> Somebody needs
> > to teach Coverity about array pointers (something beginners often
> find
> > confusing).
> >
> > This technique has been used everywhere since xdr_ioq was
> introduced
> > some years ago.
>
> I wonder if it would do ok even if the return WAS &xiod->xdrs[0]...
>
> > > 293
> > > 294 /*
> > > 295 * Advance read/insert or fill position.
> > > 296 *
> > >
> > > ** CID 124498: Concurrent data access violations (MISSING_LOCK)
> > > /nfs-ganesha/src/libntirpc/src/svc_vc.c: 1210 in
> svc_vc_override_ops()
> > >
> > Heavy sigh. False positive.
> >
> > The locks merely prevent 2 _inits from running at the same time,
> apparently
> > by dbus -- but the fields are always set to the same values.
> >
> > Moreover, address access is usually atomic (on 64-bit processors),
> and this
> > could be made explicit with atomic operators.
> >
> > So the pointer is idempotent for access, and correct here.
> >
> > Moreover, Coverity only looked at this because of a name change.
> > The operations are exactly the same as they always have been.
>
> Mark this one as intentional.
>
> > __
> >
> > > __
> > > *** CID 124498: Concurrent data access violations
> (MISSING_LOCK)
> > > /nfs-ganesha/src/libntirpc/src/svc_vc.c: 1210 in
> svc_vc_override_ops()
> > > 1204 }
> > > 1205
> > > 1206 static void
> > > 1207 svc_vc_override_ops(SVCXPRT *xprt, SVCXPRT *newxprt)
> > > 1208 {
> > > 1209 if (xprt->xp_ops->xp_getreq)
> > CID 124498: Concurrent data access violations
> (MISSING_LOCK)
> > Accessing "newxprt->xp_ops->xp_getreq" without holding
> lock
> > "ops_lock". Elsewhere, "xp_ops.xp_getreq" is accessed with
> "ops_lock" held
> > 7 out of 8 times (3 of these accesses strongly imply that it is
> necessary).
> > > 1210 newxprt->xp_ops->xp_getreq = xprt->xp_ops-
> > >xp_getreq;
> > > 1211 if (xprt->xp_ops->xp_dispatch)
> > > 1212
Re: [Nfs-ganesha-devel] custom FSAL to access storage from another user space process
Hi Dirk, Well, the proxy fsal proxies to another NFS server over TCP or UDP, so you might find that helpful. The Ceph fsal is a fairly minimal example of a fsal that delegates to a library with a posix-like api. The problem of how to do IPC between your fsal and a remote using a custom protocol is not one Ganesha solves per se. You could do a lot of things, obviously. Matt - "Dirk Jagdmann" wrote: > Hello developers, > > I'm currently evaluating how to best integrate NFS-Ganesha with our > storage > product. My company is developing a user space Linux application to > handle > storage and we'd like to provide a NFS service to access files and > directories. > > As a proof of concept we've currently used the Linux FUSE feature to > expose our > storage as a regular Linux VFS handled filesystem, and NFS-Ganesha can > use this > just fine. However using FUSE to pass data between 2 user space > processes is > probably not the best solution and also probably not the fastest > solution. I'm > therefore currently investigating how feasible it is to develop a > custom FSAL, > which will communicate directly with another user space process. I'm > thinking > about writing a simple "network protocol" which will forward the FSAL > calls to > our storage user space process. It could use a stream or datagram > style > "connection", which could be regular TCP/UDP, maybe unix domain > sockets, maybe > Linux netlink sockets, maybe even SysV shared memory (if that's even > necessary > for performance). > > I did read all the NFS-Ganesha Wiki pages and read the fsal_api.h > header once, > and have a couple of questions now: > > - does the NFS-Ganesha server use multiple processes or multiple > threads? > > - if it does use multiple process/threads, how are multiple connected > NFS > clients multiplexed to the processes/threads? > > - if multiple threads are used, how are the FSAL objects allocated and > used with > regards to the threads? Is locking required "inside" the FSAL > implementation? > > - did anybody write a FSAL which forwards the calls made from > NFS-Ganesha to the > FSAL objects to another process? I'm thinking about writing a FSAL > with a > generic protocol which different user space processes can connect to > and provide > the storage/files/directories. > > - looking at the current FSAL implementations, which is the simplest > to get me > started, but implements all the necessary features? I'm thinking about > studying > the implementation and using that as a template for my own FSAL. > > > -- > ---> Dirk Jagdmann > > http://cubic.org/~doj > -> http://llg.cubic.org > > -- > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] This week
Thanks, Frank - "Frank Filz" wrote: > There will be no concall this week. > > As of this time, I don't plan another dev-release this week. I just > pushed > Bill's RPC changes (with libntirpc update) as dev-10. Please make sure > you > rebase all patch submissions and make sure you update your sub-module. > If by > Wednesday morning there is a reasonable set of patches, I may decide > to do a > dev-11 on Wednesday, otherwise, dev-11 will be next week. > > Thanks > > Frank > > > > -- > Don't Limit Your Business. Reach for the Cloud. > GigeNET's Cloud Solutions provide you with the tools and support that > you need to offload your IT needs and focus on growing your business. > Configured For All Businesses. Start Your Cloud Today. > https://www.gigenetcloud.com/ > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Don't Limit Your Business. Reach for the Cloud. GigeNET's Cloud Solutions provide you with the tools and support that you need to offload your IT needs and focus on growing your business. Configured For All Businesses. Start Your Cloud Today. https://www.gigenetcloud.com/ ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] clnt_vc_geterr() accessing freed memory
Hi Malahal, My read on this is that the clnt_err() callout probably has to be deprecated, and clnt_call() updated to take a pointer to struct rpc_err to return the error detail. The behavior in clnt_vc_call() and friends was ok for duplex-10, but it conflicts with idempotent and backchannel calls. We'd want to do this and the matching change to the callers on Ganesha 2.0 and ntirpc duplex-11. Thoughts on that? Matt - "Malahal Naineni" wrote: > Matt, clnt_vc_call() frees the memory that is later accessed by > clnt_vc_geterr. It all happens in the same *thread*. You see two > stack > traces because I enabled valgrind to keep origins. > > Regrads, Malahal. > > Matt W. Benjamin [m...@cohortfs.com] wrote: > > Hi, > > > > So this is all happening in clnt_vc_call()? I'll have a look at > it. > > > > Matt > > > > - "Malahal Naineni" wrote: > > > > > Hi All, > > > > > > valgrind reported that clnt_vc_geterr() is accessing freed > memory. > > > Code review shows that nlm_send_async() calls clnt_call() and > then > > > calls > > > clnt_sperror() on some errors. This is clearly a bug to access > rpc > > > context > > > memory that was freed in a prior call. I am not familiar with RPC > > > code, > > > it should be an easy fix for someone familiar with RPC code. Can > > > someone > > > please fix or give pointers to fix it. > > > > > > Here is the entire valgrind report (ran against V2.2-stable, but > line > > > numbers should NOT change that much). > > > > > > ==7012== Thread 35: > > > ==7012== Invalid read of size 8 > > > ==7012==at 0x535704: clnt_vc_geterr (clnt_vc.c:582) > > > ==7012==by 0x533FFB: clnt_sperror (clnt_perror.c:83) > > > ==7012==by 0x48FF07: nlm_send_async (nlm_async.c:283) > > > ==7012==by 0x491132: nlm4_send_grant_msg (nlm_util.c:205) > > > ==7012==by 0x49B203: state_async_func_caller > (state_async.c:81) > > > ==7012==by 0x508E5A: fridgethr_start_routine > (fridgethr.c:562) > > > ==7012==by 0x61EEDF2: start_thread (in > > > /usr/lib64/libpthread-2.17.so) > > > ==7012==by 0x67011AC: clone (in /usr/lib64/libc-2.17.so) > > > ==7012== Address 0x87f5a88 is 136 bytes inside a block of size > 184 > > > free'd > > > ==7012==at 0x4C29577: free (in > > > /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) > > > ==7012==by 0x53AAB9: free_rpc_call_ctx (rpc_ctx.c:251) > > > ==7012==by 0x5356B3: clnt_vc_call (clnt_vc.c:567) > > > ==7012==by 0x48FE71: nlm_send_async (nlm_async.c:267) > > > ==7012==by 0x491132: nlm4_send_grant_msg (nlm_util.c:205) > > > ==7012==by 0x49B203: state_async_func_caller > (state_async.c:81) > > > ==7012==by 0x508E5A: fridgethr_start_routine > (fridgethr.c:562) > > > ==7012==by 0x61EEDF2: start_thread (in > > > /usr/lib64/libpthread-2.17.so) > > > ==7012==by 0x67011AC: clone (in /usr/lib64/libc-2.17.so) > > > > > > Regards, Malahal. > > > PS: I can make the context pointer NULL when we free, and use > > > xdrs->x_lib[1] > > > in the check instead of xdrs->x_lib[0] in clnt_vc_geterr(). Wonder > if > > > that is a right method. > > > > -- > > Matt Benjamin > > CohortFS, LLC. > > 315 West Huron Street, Suite 140A > > Ann Arbor, Michigan 48103 > > > > http://cohortfs.com > > > > tel. 734-761-4689 > > fax. 734-769-8938 > > cel. 734-216-5309 > > -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [PATCH] pool->n_threads should be decremented with mutex
Summarizing from call. Presumably we'd normally fix this without taking larger reorganization changes from duplex-12, but IIRC, there's a larger list of things, and we agreed to bring a list to the next call for discussion. Matt - "William Allen Simpson" wrote: > On 6/24/15 1:24 PM, Malahal Naineni wrote: > >>/* cleanup thread context */ > >>destroy_wait_entry(&thrd->ctx.we); > >> - --(thrd->pool->n_threads); > >>mutex_lock(&pool->we.mtx); > >> + --(thrd->pool->n_threads); > >>cond_signal(&pool->we.cv); > >>mutex_unlock(&pool->we.mtx); > >>mem_free(thrd, 0); > >> -- > My solution in duplex-12 was: > > /* cleanup thread context */ > - destroy_wait_entry(&thrd->ctx.we); > - --(thrd->pool->n_threads); > - mutex_lock(&pool->we.mtx); > - cond_signal(&pool->we.cv); > - mutex_unlock(&pool->we.mtx); > - mem_free(thrd, 0); > + atomic_dec_uint32_t(&pool->n_threads); > + cond_destroy(&wpt->pqcond); > + mem_free(wpt, sizeof(*wpt)); -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] clnt_vc_geterr() accessing freed memory
Hi, So this is all happening in clnt_vc_call()? I'll have a look at it. Matt - "Malahal Naineni" wrote: > Hi All, > > valgrind reported that clnt_vc_geterr() is accessing freed memory. > Code review shows that nlm_send_async() calls clnt_call() and then > calls > clnt_sperror() on some errors. This is clearly a bug to access rpc > context > memory that was freed in a prior call. I am not familiar with RPC > code, > it should be an easy fix for someone familiar with RPC code. Can > someone > please fix or give pointers to fix it. > > Here is the entire valgrind report (ran against V2.2-stable, but line > numbers should NOT change that much). > > ==7012== Thread 35: > ==7012== Invalid read of size 8 > ==7012==at 0x535704: clnt_vc_geterr (clnt_vc.c:582) > ==7012==by 0x533FFB: clnt_sperror (clnt_perror.c:83) > ==7012==by 0x48FF07: nlm_send_async (nlm_async.c:283) > ==7012==by 0x491132: nlm4_send_grant_msg (nlm_util.c:205) > ==7012==by 0x49B203: state_async_func_caller (state_async.c:81) > ==7012==by 0x508E5A: fridgethr_start_routine (fridgethr.c:562) > ==7012==by 0x61EEDF2: start_thread (in > /usr/lib64/libpthread-2.17.so) > ==7012==by 0x67011AC: clone (in /usr/lib64/libc-2.17.so) > ==7012== Address 0x87f5a88 is 136 bytes inside a block of size 184 > free'd > ==7012==at 0x4C29577: free (in > /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) > ==7012==by 0x53AAB9: free_rpc_call_ctx (rpc_ctx.c:251) > ==7012==by 0x5356B3: clnt_vc_call (clnt_vc.c:567) > ==7012==by 0x48FE71: nlm_send_async (nlm_async.c:267) > ==7012==by 0x491132: nlm4_send_grant_msg (nlm_util.c:205) > ==7012==by 0x49B203: state_async_func_caller (state_async.c:81) > ==7012==by 0x508E5A: fridgethr_start_routine (fridgethr.c:562) > ==7012==by 0x61EEDF2: start_thread (in > /usr/lib64/libpthread-2.17.so) > ==7012==by 0x67011AC: clone (in /usr/lib64/libc-2.17.so) > > Regards, Malahal. > PS: I can make the context pointer NULL when we free, and use > xdrs->x_lib[1] > in the check instead of xdrs->x_lib[0] in clnt_vc_geterr(). Wonder if > that is a right method. -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical & virtual servers, alerts via email & sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] fridge threads and worker latency
Hi Bill,
This isn't the current work plan/design as I understand it, so I'd appreciate it
if you walk through your blocker with us here.
Also "I" is me, as I could have told you.
Thank you.
Matt
- "William Allen Simpson" wrote:
> I've been struggling with the fridge threads. For NFS/RDMA,
> once we are running a work thread, we really don't want to
> hand off to another work thread during processing in
> thr_decode_rpc_request() -- adds considerable latency.
>
> Panasas also reports latency and stalls in VFS::PanFS.
>
> Currently, the fridge starts processing some work, then stops
> and re-queues a work request nfs_rpc_enqueue_req(nfsreq) at
> the end of thr_decode_rpc_request().
>
> By the comments in nfs_rpc_dispatcher_thread.c:
>
> * Next, the preferred dispatch thread should be, I speculate, one
> * which has (most) recently handled a request for this xprt.
>
> ("I" isn't identified.)
>
> So there is extra complication in nfs_rpc_getreq_ng()
> * calling fridgethr_submit(req_fridge, thr_decode_rpc_requests,
> xprt),
> ** which in turn runs thr_decode_rpc_requests()
> *** to loop on any multiple requests per xprt,
> handing each to a separate worker.
>
> All to attempt "locality of reference" for a worker.
>
> This is particularly bad for RDMA, as serializing multiple
> requests this way means the most buffers have to be held
> outstanding at a time!
>
> Perhaps a simpler and more efficient design would borrow from
> Internet Routing: Weighted Fair Queuing.
>
> We could more easily insert jobs into a weighted array of queues,
> and then the thread keeps going without any handoff until done
> (or another wait for event).
>
> If after completing a req, then another req for the same xprt is
> found, the next req should be moved to the end of the weighted
> queue, so that other xprts aren't treated unfairly.
>
> Those are the two basic elements of Weighted Fair Queuing (WFQ).
>
> --
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] system max fd and Cache Inode parameter
Hi, Sorry for the confusion--I had forgotten or never noticed that Ganesha updates these itself. Clearly, the number of fds is partially independent of the number of cache entries--for FSALs which in fact have OS fds. Among other things, the lru thread is working to close entries when they drop to L2. Matt - "Malahal Naineni" wrote: > It is probably distro specific, but 4096 is the default NOFILE limit > on > RHEL6/RHEL7 systems. It can be set to as high as 1024*1024. See > 14447420966b9dfcdeb9975ac134a36613ade3c4 for further details on how > it > is set to 1M by default for ganesha daemon. This is available in > 2.2-stable. > > Regards, Malahal. > > Krishna Harathi [khara...@exablox.com] wrote: > >Matt, > >Yes we are using VFS FSAL. I will re-run the workload after > increasing the > >limits. > >Thanks. > >Regards. > >Krishna Harathi > >On Sat, May 16, 2015 at 8:36 AM, Matt W. Benjamin > <[1]m...@cohortfs.com> > >wrote: > > > > Hi Krishna, > > > > If you're using a VFS-like FSAL, I wouldn't think you'd want to > be > > hard limited to 4K open files. > > > > It could be useful to have more flexibility in (choice of?) > LRU > > reclaim strategy, but to start with, I'd raise this limit (on > > Linux, /proc/sys/fs/nr_open, etc). > > > > Matt > > - "Krishna Harathi" <[2]khara...@exablox.com> wrote: > > > > > To be clear, this comment cache_inode_lru.c did not help > much. > > > > > > > > > /* Set high and low watermark for cache entries. This seems > a > > > bit fishy, so come back and revisit this. */ > > > lru_state.entries_hiwat = cache_param.entries_hwmark; > > > lru_state.entries_used = 0 ; > > > > > > > > > > > > > > > Regards. > > > Krishna Harathi > > > > > > On Fri, May 15, 2015 at 7:44 PM, Krishna Harathi < > > > [3]khara...@exablox.com > wrote: > > > > > > > > > > > > > > > I am trying to determine the relation between the system > imposed limit > > > of max FDs (4096 now) and > > > the Cache Inode HW Mark (10 default). > > > > > > > > > In a test work load, I am getting errno 24 (too many open > files) error > > > returned, > > > and here is our current settings. > > > > > > > > > > > > 15/05/2015 18:48:23 : epoch 5556a1e7 : OneBlox-0333 : > > > nfs-ganesha-12814[main] cache_inode_lru_pkginit :INODE LRU > :INFO > > > :Attempting to increase soft limit from 1024 to hard limit of > 4096 > > > 15/05/2015 18:48:23 : epoch 5556a1e7 : OneBlox-0333 : > > > nfs-ganesha-12814[main] cache_inode_lru_pkginit :INODE LRU > :INFO > > > :Setting the system-imposed limit on FDs to 4096 > > > > > > > > > > > > > > > I looked at the related code in Cache_inode but could not > determine > > > the relationship between > > > the system limit and cache_inode or how many file FDs Ganesha > will try > > > to open. > > > > > > > > > Any help is (1) What is the system limit to be set, given the > default > > > cache inode HWmark? > > > (2) For the given system limit, what is the cache inode > setting. We > > > are using Ganesha 2.1. > > > > > > > > > > > > > > > > > > Regards. > > > Krishna Harathi > > > > > > > > > -- > > > One dashboard for servers and applications across > > > Physical-Virtual-Cloud > > > Widest out-of-the-box monitoring support with 50+ > applications > > > Performance metrics, stats and reports that give you > Actionable > > > Insights > > > Deep dive visibility with transaction tracing using APM > Insight. > > > [4]http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > > > ___ > > > Nfs-ganesha-devel mailing list > &g
Re: [Nfs-ganesha-devel] system max fd and Cache Inode parameter
Hi Krishna, If you're using a VFS-like FSAL, I wouldn't think you'd want to be hard limited to 4K open files. It could be useful to have more flexibility in (choice of?) LRU reclaim strategy, but to start with, I'd raise this limit (on Linux, /proc/sys/fs/nr_open, etc). Matt - "Krishna Harathi" wrote: > To be clear, this comment cache_inode_lru.c did not help much. > > > /* Set high and low watermark for cache entries. This seems a > bit fishy, so come back and revisit this. */ > lru_state.entries_hiwat = cache_param.entries_hwmark; > lru_state.entries_used = 0 ; > > > > > Regards. > Krishna Harathi > > On Fri, May 15, 2015 at 7:44 PM, Krishna Harathi < > khara...@exablox.com > wrote: > > > > > I am trying to determine the relation between the system imposed limit > of max FDs (4096 now) and > the Cache Inode HW Mark (10 default). > > > In a test work load, I am getting errno 24 (too many open files) error > returned, > and here is our current settings. > > > > 15/05/2015 18:48:23 : epoch 5556a1e7 : OneBlox-0333 : > nfs-ganesha-12814[main] cache_inode_lru_pkginit :INODE LRU :INFO > :Attempting to increase soft limit from 1024 to hard limit of 4096 > 15/05/2015 18:48:23 : epoch 5556a1e7 : OneBlox-0333 : > nfs-ganesha-12814[main] cache_inode_lru_pkginit :INODE LRU :INFO > :Setting the system-imposed limit on FDs to 4096 > > > > > I looked at the related code in Cache_inode but could not determine > the relationship between > the system limit and cache_inode or how many file FDs Ganesha will try > to open. > > > Any help is (1) What is the system limit to be set, given the default > cache inode HWmark? > (2) For the given system limit, what is the cache inode setting. We > are using Ganesha 2.1. > > > > > > Regards. > Krishna Harathi > > -- > One dashboard for servers and applications across > Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable > Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] DISCUSSION: V2.3 workflow and how we proceed
Hi, There clearly are people who think reviewing something large in small chunks is categorically easier, but, I don't completely understand the reasoning. 1) It seems to me that whether the small chunks are easier to actually understand, depends on whether the chunks can really be understood in isolation. Sometimes, that's not really the case. It's well known from Linux kernel development that not infrequently, splitting a logical change actually makes review and intake more difficult, producing extra workload on submitters who are asked to make chunks build separately, bisectable, etc. 2) When a set of chunks could have been understood in isolation, then a competent reviewer can probably look at sets of related, changed files (navigating to them in the review tool), with the added benefit that the reviewer can jump to any other part of the logical change to compare context. Matt - "Frank Filz" wrote: > > From: Malahal Naineni [mailto:mala...@us.ibm.com] > > DENIEL Philippe [philippe.den...@cea.fr] wrote: > > > issue. If I do a big patch or 10 small ones, all of my changed > files > > > will have to be reviewed, which does have no impact on the > workload. > > > In fact, one big patch is a cool situation : it is easy to rebase, > and > > > it depends only on stuff already pushed and landed. If I publish > 5 > > > patches, what if patch 1, 2 and 3 merge fine, but 4 produce a > conflict > > > ? How could I rebase 4 without touch 1,2 and 3 ? This leads to a > > > dependencies maze, and this is precisely the situation we fell > into. > > > > There is no doubt that a "well" split patchset is easier to review. > I did > rebase > > mega patches from others (that happens when you pick up someone > else's > > work) in the past and it is a PITA. Even if it is my code, I find it > lot > easier to > > rebase small patches than one big patch. > > > From: Dominique Martinet [mailto:dominique.marti...@cea.fr] > > > - we should provide big and squashed patches, one per > feature. > > > For example, CEA will soon push a rework of FSAL_HPSS, this will > be a > > > single commit. > > > > We do that for HPSS because the FSAL is old, has been pruned out a > while > > ago, and currently doesn't checkpatch/individual patches won't > checkpatch. > > That's purely selfish and we don't actually expect anyone to review > that > code > > anyway, since no-one in the active community will use it - it's > pretty far > from > > normal workflow submitting... > > > > > > I believe patches should be kept reasonably small for review, > really, but > > "reasonably small" is subjective and flexible. > > what we really need is for more people to review and that's not > going to > > improve if we push big hunks :) > > I absolutely agree. Maybe we don't need to have the patch granularity > as > fine as Linux kernel, but I like to see smaller patches. They also > make for > more bisectable code since we can see "ok, this part of the feature > caused > the problem" rather than "hmm, there's a problem somewhere in this > new huge > feature." > > Frank > > > > -- > One dashboard for servers and applications across > Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable > Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] DISCUSSION: V2.3 workflow and how we proceed
Hi, - "GerritForge Support" wrote: > >> From what I can tell, Frank finds problematic > > > > 1) a lack in gerrit of a "changeset" concept, plus, apparently > > It actually exists and is called “topic”. It will be very soon > possible as well to merge a whole topic atomically with one click. I found "topic" mentioned in some online discussion, but there seemed to be mixed reviews from some developers. I didn't come away with a clear sense of the strengths and weaknesses of the feature, but there appear to be tradeoffs. > > It is actually in beta and will be released very soon: it is called > “NoteDB”. It is basically the archive of all review history (including > ratings and comments) as Git objects. This sounds interesting, but what I think Frank and Dominique were specifically looking for is for the affirmative reviews in gerrit to be transformed into "Acked-By" statements in the primary commit msg in git on merge. Is that behavior part of NoteDB, or, alternately, something that we can straightforwardly accomplish with gerrithub? Thanks, Matt > Kind Regards. > --- > GerritForge Support > http://www.gerritforge.com > > supp...@gerritforge.com > Fax: +44 (0) 203 318-3620 > UK: +44-(0) 208 144-9448 > US: +1-(650) 741-1436 > AU: +61-(0)280 911448 > Skype: gerrit.support -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] DISCUSSION: V2.3 workflow and how we proceed
Hi, As I mentioned in an IRC discussion, my recollection from when OpenAFS implemented gerrit (also their own installation, but not to my knowledge customized), it was basically mandated that developers submit "one patch per change." >From what I can tell, Frank finds problematic 1) a lack in gerrit of a "changeset" concept, plus, apparently 2) some missing automation to propagate review information into git's history/commit msg I think both are potentially legitimate considerations, with a claim to be related to best practices (so potentially on a par with the motivations to update the review process). In large corporate development environments I've worked in, the aspiration to get the 20% lacking in the off-the-shelf tool would be solved by some internal development. In other cases, someone said, "no discussion, we're using VSS." I don't have a very strong opinion, but I do somewhat feel that however much gerrit helps with some review beset practices, just doing what gerrithub appears to support oob probably brings arbitrary processes, not just best practices. At the same time, I think the objection to "one patch per change" (which appears to be, "this well-formed, logical change which touches many files is too big for me t review") may be overblown. If we could do one patch per-change, objection #1 might be overcome. To overcome objection #2 seems to require some minimal tooling--I'm unclear whether it is compatible with gerrithub; If for some reason not, is there an organization volunteering to host and maintain/co-maintain aprivate gerrit installation? Matt - "DENIEL Philippe" wrote: > Hi Frank, > > reply is in the message body. I cut some pieces of your original > message > to avoid a too long message where information would be diluted. > > On 04/24/15 23:59, Frank Filz wrote: > > 1. Abandon gerrithub, revert to using github branches for review and > merge. > > This has a few problems. > The issue with github-based reviews are known. Avoiding solving a > current issue by stepping back to formerly known problems may seem > comfortable, but this is no evolution, it's clearly a regression. > > > 2a. One solution here is use an e-mail review system (like the > kernel > > process). > This is prehistory... Kernel community works like this because they > have > this habit since the (Linux) world was new and nothing else existed. > Can > someone seriously say it would be user-friendly ? OK, mail are > archived, > but not indexed. Looking for a review will be like lookingfora needle > > ina haystack. Once hundreds of mail will have been sent and received > and > re-re-re-re-replied finding useful information will become impossible. > > Please don't bring us back to Stone Age. > > > 3. Change our workflow to work with gerrithub better (stop using > the > > incremental patch process). One loss here would be the ability to > bisect and > > hone in on a small set of changes that caused a bug. > It seems like a much better idea. People in my team are developing in > > the Lustre code. The Lustre community works with a private gerrit > since > the beginning. They have their best practices and their workflow. > In particular, they have "Patch windows" : when the window is opened, > > people can submit patchsets. As it closed, people review it, fix > stuff, > rebase code, and the branch is released. No new patchset comes at this > > time. Then the window opens again and a new cycle starts. One > important > point : the "master repo" is the git inside gerrit and no other. This > > means that contributors would only fetch gerrithub to rebase their > work, > github will then become a simple mirror. > Clearly, the "merge/fix/rebase" process is longer than a week. We > could > work this way by simply abandon the one-week cycle we are accustomed > to. > It's just a matter of using new, more adapted, rules and best > practises. > > > 3a. The most extreme option would be to abandon incremental patches. > If you > > have a body of work for submission in a given week, you submit one > patch for > > that work. > Again, I believe that the one-week cycle is the real issue and it's > such > a constraint for release management. You should open/close the > submission window at your will. It would ease your work a lot, > wouldn't > it ? Remember that gerrit was designed to help the release manager, > it's > not designed to be that painful. We may just use the tool the wrong > way. > > > 3c. A process implied by a post Matt found: Perform code review > with > > incremental patches. Once submission is ready, squash the submission > into a > > single patch and get final signoffs on that. Then the single patch > is > > merged. > People can rebase their patchset, even when submitted to gerrit and I > > think they should keep the same changeid. Remember that changeid is > nothing but a mark on a commit to allow gerrit to keep patchset > history. > It's not a commit id. > > > > > If we
Re: [Nfs-ganesha-devel] PLEASE READ - Merge nightmare - may not be a dev-1 this week
It seems to me that openafs gerrit (dating back to 2008 or earlier) had an ability to keep sets of patches pushed together, as a kind of changeset. I'll ask around. Matt - "Frank Filz" wrote: > Ok, using gerrithub is really feeling like it's not going to work for > me... > > Merging a patch set with multiple patches is impossible to manage. > There's > no assistance in keeping track of the order of merging patches. > > At this moment, my gerrithub next branch is partially merged. This > MUST NOT > BE RELIED ON - PLEASE DO NOT REBASE - HISTORY WILL BE CHANGING > > I'm not sure how to proceed, there probably won't be a dev-1 this > week. > > I'm ready to go back to github and skip gerrithub. > > Either that or we abandon the incremental patch idea and each feature > is ONE > patch. Heaven help us to try and review those... > > Frank > > > > -- > One dashboard for servers and applications across > Physical-Virtual-Cloud > Widest out-of-the-box monitoring support with 50+ applications > Performance metrics, stats and reports that give you Actionable > Insights > Deep dive visibility with transaction tracing using APM Insight. > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Matt Benjamin CohortFS, LLC. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://cohortfs.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309 -- One dashboard for servers and applications across Physical-Virtual-Cloud Widest out-of-the-box monitoring support with 50+ applications Performance metrics, stats and reports that give you Actionable Insights Deep dive visibility with transaction tracing using APM Insight. http://ad.doubleclick.net/ddm/clk/290420510;117567292;y ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] [Nfs-ganesha-support] Troubles with configuration
Hi,
The Ceph FSAL uses the libcephfs (and librados) to consume the cluster.
It's not re-exporting a mounted Ceph filesystem.
Matt
- "Timofey Titovets" wrote:
> showmount -e 172.20.252.12
> Export list for 172.20.252.12:
> / (everyone)
>
> 2015-04-24 10:33 GMT+03:00 DENIEL Philippe :
> > Hi,
> >
> > what does " showmount -e 172.20.252.12 " say ? This way, you'll see
> if
> > Ganesha was capable of building the export entries.
> > My knowledge of CEPH is weak, so I forward your question to
> > nfs-gamesha-devel. In particular, I do not know if:
> > - CEPH support open_by_handle_at() syscall
> > - FSAL_CEPH can work on a "regular" CEPH distribution. I
> remember
> > that Matt and Adam made some changes to CEPH to ease pNFS
> > implementation, and I do not know if those commits went upstream.
> They
> > will tell us about this.
> >
> > Regards
> >
> > Philippe
> >
> > On 04/24/15 09:18, Timofey Titovets wrote:
> >> Good time of day, I've try to setup nfs-ganesha server on Ubuntu
> 15.04
> >> Builded by last source from master tree.
> >>
> >> 1. Can't be mounted with nfsv3
> >> Config:
> >> EXPORT
> >> {
> >> # Export Id (mandatory, each EXPORT must have a unique
> Export_Id)
> >> Export_Id = 77;
> >> # Exported path (mandatory)
> >> Path = /storage;
> >> # Pseudo Path (required for NFS v4)
> >> Pseudo = /cephfs;
> >> Access_Type = RW;
> >> NFS_Protocols = 3;
> >> FSAL {
> >> Name = VFS;
> >> }
> >> }
> >> Or
> >> EXPORT
> >> {
> >> Export_ID = 1;
> >> Path = "/";
> >> Pseudo = "/cephfs";
> >> Access_Type = RW;
> >> NFS_Protocols = 3;
> >> Transport_Protocols = TCP;
> >> FSAL {
> >> Name = CEPH;
> >> }
> >> }
> >>
> >> On client I've get:
> >> # mount -o nfsvers=3 172.20.252.12:/ /mnt -v
> >> mount.nfs: timeout set for Fri Apr 24 09:47:50 2015
> >> mount.nfs: trying text-based options
> 'nfsvers=3,addr=172.20.252.12'
> >> mount.nfs: prog 13, trying vers=3, prot=6
> >> mount.nfs: trying 172.20.252.12 prog 13 vers 3 prot TCP port
> 2049
> >> mount.nfs: prog 15, trying vers=3, prot=17
> >> mount.nfs: trying 172.20.252.12 prog 15 vers 3 prot UDP port
> 40321
> >> mount.nfs: mount(2): Permission denied
> >> mount.nfs: access denied by server while mounting 172.20.252.12:/
> >>
> >> What's did I wrong? %)
> >>
> >
> >
> >
> --
> > One dashboard for servers and applications across
> Physical-Virtual-Cloud
> > Widest out-of-the-box monitoring support with 50+ applications
> > Performance metrics, stats and reports that give you Actionable
> Insights
> > Deep dive visibility with transaction tracing using APM Insight.
> > http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> > ___
> > Nfs-ganesha-support mailing list
> > nfs-ganesha-supp...@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-support
>
>
>
> --
> Have a nice day,
> Timofey.
>
> --
> One dashboard for servers and applications across
> Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable
> Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> ___
> Nfs-ganesha-devel mailing list
> Nfs-ganesha-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
--
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103
http://cohortfs.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
--
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
___
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
