[Nfs-ganesha-devel] read-only open causing file to be truncated with NFSv4/ganesha.

izeof(buf)); printf("write returned %ld\n", ret); rc = stat(fn, &stbuf); printf("size: %ld\n", stbuf.st_size); int fd1; fd1 = open(fn, O_RDONLY); printf("open(%s) = %d\n", fn, fd1); rc = stat(fn, &

[Nfs-ganesha-devel] Connection to V4 port gets closed soon after it is established.

This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org.Hi Bill, I'm seeing a case where the client establishes a connection and sends V4 NULL request. But before ganesha processes it, the socket is closed by TIRPC. Here is the TIRPC logs I captured. The FD

[Nfs-ganesha-devel] Couple of stat issues.

Hello, These are couple of statistics related issues I noticed: 1. NFS4 compounds are incremented multiple times: For individual OPs, ganesha takes this path: server_stats_nfsv4_op_done() -> record_nfsv4_op() -> record_op(&sp->compounds..) Once the compound is complete, we take another path serv

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

, the latency increased from 5ms (with 17 lanes) to 22 ms (with 1023 lanes) and the test failed to achieve required IOPs. Thanks, Pradeep On Tue, Apr 3, 2018 at 7:58 AM, Pradeep wrote: > Hi Daniel, > > Sure I will try that. > > One thing I tried is to not allocate ne

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

a better approach. I will try your suggestion next. With 1023 lanes, it is unlikely that all lanes will have an active entry. Thanks, Pradeep On 4/3/18, Daniel Gryniewicz wrote: > So, the way this is supposed to work is that getting a ref when the ref > is 1 is always an LRU_REQ_INITIAL

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

s called, the entry is supposed to be out of LRU. Thanks, Pradeep On Mon, Aug 14, 2017 at 5:38 PM, Pradeep wrote: > > > On Fri, Aug 11, 2017 at 8:52 AM, Daniel Gryniewicz > wrote: > >> Right, this is reaping. I was thinking it was the lane thread. Reaping >> only

Re: [Nfs-ganesha-devel] Multiprotocol support in ganesha

/oplock_linux.c Thanks, Pradeep On Wed, Mar 7, 2018 at 7:18 AM, Frank Filz wrote: > Unfortunately out in the real world, people want to mix POSIX and > Microsoft semantics… > > > > So we do the best we can. > > > > I wonder how much of the multi-protocol use falls into two

Re: [Nfs-ganesha-devel] Multiprotocol support in ganesha

of Samba). I > personally wouldn't reject such support if it was offered, but as far as I > know, no one is even thinking about working on it. > > Daniel > > > On 03/06/2018 12:20 PM, Pradeep wrote: > >> Hello, >> >> Is there plans to implement m

[Nfs-ganesha-devel] Multiprotocol support in ganesha

Hello, Is there plans to implement multiprotocol (NFS and CIFS accessing same export/share) in ganesha? I believe current FD cache will need changes to support that. Thanks, Pradeep -- Check out the vibrant tech

Re: [Nfs-ganesha-devel] READDIR doesn't return all entries.

Hi Frank/Sachin, I will get back to you with tcpdump/logs if I can recreate it. It was around 2000 directories and a "find" from the export root was issued before the "ls" from client. Thanks, Pradeep On Tue, Feb 13, 2018 at 9:38 AM, Frank Filz wrote: > What FSAL? Is

[Nfs-ganesha-devel] READDIR doesn't return all entries.

/985564cbd147b6acc5dd6de61a3ca8fbc6062eda (reverted the change and verified that all entries are returned without this change) Still looking into why it broke READDIR for me. Any insights on debugging this would be helpful. Thanks, Pradeep -- Check

[Nfs-ganesha-devel] Using root privileges when using kerberos exports with Ganesha.

UID/GID 0 on Ganesha side. Have anyone successfully used privileged access to NFS exports with Kerberos/AD with Ganesha server? If yes, could share how you were able to achieve that? Thanks, Pradeep -- Check out the

Re: [Nfs-ganesha-devel] Crash in graceful shutdown.

If so, I can track down the leak. > > Daniel > > > On 02/01/2018 09:51 AM, Pradeep wrote: > >> Hello, >> >> In graceful shutdown of ganesha (do_shutdown()), the way object >> handles are released is first by calling unexport() and then >> destroy_fsal

[Nfs-ganesha-devel] Crash in graceful shutdown.

osed to free all handles in MDCACHE? This is with 2.6-rc2 in case you want to look at code. Thanks, Pradeep -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashd

Re: [Nfs-ganesha-devel] Change in ffilz/nfs-ganesha[next]: Move nfs_Init_svc() after create_pseudofs().

On Wed, Jan 31, 2018 at 2:23 PM, William Allen Simpson < william.allen.simp...@gmail.com> wrote: > On 1/31/18 3:11 PM, GerritHub wrote: > >> Frank Filz *posted comments* on this change. >> >> View Change >> >> It seems to me that the dupreq2_pkginit() is alread

Re: [Nfs-ganesha-devel] Correct initialization sequence

(gdb) print drc_st $1 = (struct drc_st *) 0x0 (gdb) print nfs_init.init_complete $2 = false On Tue, Jan 30, 2018 at 1:39 PM, Matt Benjamin wrote: > reordering, I hope > > Matt > > On Tue, Jan 30, 2018 at 1:40 PM, Pradeep wrote: > > Hello, > > > > It is poss

[Nfs-ganesha-devel] Correct initialization sequence

() so that rest of ganesha is ready to process requests as soon as we start listing on the NFS port? Another way is to return NFS4ERR_DELAY until 'nfs_init.init_complete' is true. Any thoughts? Thanks, Pradeep --

Re: [Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?

Hi Bill, Are you planning to pull this into the next ganesha RC? Thanks, Pradeep On Sun, Jan 28, 2018 at 7:13 AM, William Allen Simpson < william.allen.simp...@gmail.com> wrote: > On 1/27/18 4:07 PM, Pradeep wrote: > >> ​Here is what I see in the log (the '2' i

Re: [Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?

h recv failed): nfs-ganesha-199008[svc_948] rpc :TIRPC :WARN :svc_vc_recv: 0x7f91c0861400 fd 21 recv errno 11 (try again) 2 176​ The fix looks good. Thanks Bill. Pradeep -- Check out the vibrant tech community on one of the

Re: [Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?

> the socket is ready again, it will trigger a new event on the thread > waiting > > on the epoll. > > > > Bill, please correct me if I'm wrong. > > > > Daniel > > > > > > On 01/25/2018 09:13 PM, Matt Benjamin wrote: > >> > >&g

Re: [Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?

If recv() returns EAGAIN, then svc_vc_recv() returns without rearming the epoll_fd. How does it get back to svc_vc_recv() again? On Wed, Jan 24, 2018 at 9:26 PM, Pradeep wrote: > Hello, > > I seem to be hitting a corner case where ganesha (2.6-rc2) does not > respond to a RENEW requ

[Nfs-ganesha-devel] 'missed' recv with 2.6-rc2?

qst_epoll_event() resets SVC_XPRT_FLAG_ADDED. Is it possible for another thread to svc_rqst_rearm_events()? In that case if svc_rqst_epoll_event() could reset the flag set by svc_rqst_rearm_events and complete the current receive before the other thread could call epoll_ctl(), right?

Re: [Nfs-ganesha-devel] NULL pointer deref in clnt_ncreate_timed()

Sure. I will try it. I think it can be reproduced by removing 'localhost' from /etc/hosts for the IPv6 address (::1). On Tue, Jan 23, 2018 at 11:01 AM, Daniel Gryniewicz wrote: > Hi, Pradeep. > > Can you try with this patch on ntirpc? > > https://github.com/nf

[Nfs-ganesha-devel] NULL pointer deref in clnt_ncreate_timed()

Hello, I'm running into a crash in libntirpc with rc2: #2 #3 0x7f9004de31f4 in clnt_ncreate_timed (hostname=0x57592e "localhost", prog=100024, vers=1, netclass=0x57592a "tcp", tp=0x0) at /usr/src/debug/nfs-ganesha-2.6-rc2/libntirpc/src/clnt_generic.c:197 #4 0x0049a21c in clnt_

Re: [Nfs-ganesha-devel] Segfault in dupreq cache.

Sure - I will give it a try. Thanks Daniel. On 1/15/18, Daniel Gryniewicz wrote: > Hi, Pradeep. > > There's been 6 sets of code changes to dupreq since dev-22, including > refcount bugfixes. Can you try with 2.6-rc2? > > Daniel > > On 01/13/2018 08:50 PM, Prade

[Nfs-ganesha-devel] Segfault in dupreq cache.

Hello, I'm seeing a segfault in nfs_dupreq_finish() with 2.6.dev.22. This is when using NFSv4.0 clients. The dupreq_entry seems to be malformed (drc is still NULL) #0 0x7f761902023b in raise () from /lib64/libpthread.so.0 #1 0x00442a67 in crash_handler (signo=11, info=0x7f75ed745730

Re: [Nfs-ganesha-devel] Race between fsal_find_fd() and 'open downgrade'

gt; Frank > >> -Original Message- >> From: Pradeep [mailto:pradeep.tho...@gmail.com] >> Sent: Wednesday, January 10, 2018 8:06 PM >> To: nfs-ganesha-devel >> Subject: [Nfs-ganesha-devel] Race between fsal_find_fd() and 'open >> downgrade' >&g

[Nfs-ganesha-devel] Race between fsal_find_fd() and 'open downgrade'

Hello, I'm seeing a write failure because another thread in ganesha doing 'open downgrade' closed the FD the find_fd() returned. Any suggestion on how to fix the race? vfs_reopen2() {... status = vfs_open_my_fd(myself, openflags, posix_flags, my_fd); if (!FSAL_IS_ERROR(status)) {

[Nfs-ganesha-devel] Is there a way to update params in FSAL block with SIGHUP?

Hello, It is possible to update export specific parameters added in the FSAL block using SIGHUP only(instead of first doing RemoveExport and then doing SIGHUP)? RemoveExport has the downside that it will blow away the cache and fail inflight I/Os. For example, FSAL VFS has parameter fsid_type. It

Re: [Nfs-ganesha-devel] XID missing in error path for RPC AUTH failure.

Thanks for the patch Bill. dev.22 works for me. On Fri, Dec 15, 2017 at 9:15 AM, William Allen Simpson < william.allen.simp...@gmail.com> wrote: > On 12/14/17 1:13 PM, William Allen Simpson wrote: > >> This is May 2015 code, based upon 2012 code. Obviously, we haven't >> been testing error respo

Re: [Nfs-ganesha-devel] XID missing in error path for RPC AUTH failure.

possible that some other function was used to encode XID that got removed recently? Also I see that XID is missing in other error cases as well. Perhaps it is better to move this up to handle all 'MSG_DENIED' cases. > Matt > > On Wed, Dec 13, 2017 at 9:38 AM, Pradeep wrote

[Nfs-ganesha-devel] XID missing in error path for RPC AUTH failure.

Hello, When using krb5 exports, I noticed that TIRPC does not send XID in response - see xdr_reply_encode() for MSG_DENIED case. Looks like Linux clients can't decode the message and go in to an infinite loop retrying the same NFS operation. I tried adding XID back (like it is done for normal case

[Nfs-ganesha-devel] enqueued_reqs/dequeued_reqs

moved to nfs_rpc_process_request()? Thanks, Pradeep -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_

[Nfs-ganesha-devel] Ganesha stuck in release_openstate().

o an infinite loop (the errcnt is never incremented). Also since we hold 'clientid->cid_mutex' before calling release_openstate(), all new mounts are hung as well. Any thoughts on how ganesha ends up having 'open state' in nfs4_owner->so_st

[Nfs-ganesha-devel] crash in jemalloc leading to a deadlock.

I'm using Ganesha 2.6 dev.12 with jemalloc-3.6.0 and hitting a case where jemalloc seem to be holding a lock and crashing. In Ganesha's gsh_backtrace(), we try to allocate memory and that hangs (ended up in deadlock). Have you seen this before? Perhaps it is a good idea not to allocate memory in ba

Re: [Nfs-ganesha-devel] [PATCH v2] nfs: Fix ugly referral attributes

Hi Frank, I should be able to do that. Thanks Chuck for the fix. On Mon, Nov 6, 2017 at 9:34 AM, Frank Filz wrote: > Pradeep, > > Could you verify this patch with nfs-ganesha? Looking at the code, it looks > like we will supply the attributes requested. > > Thanks > >

Re: [Nfs-ganesha-devel] NFSv4 referrals not working with ganesha.

On Wed, Nov 1, 2017 at 8:49 AM, Chuck Lever wrote: > >> On Nov 1, 2017, at 10:53 AM, Pradeep wrote: >> >> Adding linux-nfs (did not work last couple of times because of email format). >> >> Is this supposed to work with Linux NFS clients (see the problem >&g

Re: [Nfs-ganesha-devel] NFSv4 referrals not working with ganesha.

t; >> It may also be that you need a fully qualified path (starting with a /). >> >> >> >> It looks like Ganesha is doing the right thing though. >> >> >> >> Frank >> >> >> >> *From:* Pradeep [mailto:pradeep.tho...@gmai

Re: [Nfs-ganesha-devel] NFSv4 referrals not working with ganesha.

ou try any other clients? This may be a client issue (I did see some >>> suspicious code in the client). >>> >>> >>> >>> It may also be that you need a fully qualified path (starting with a /). >>> >>> >>> >>> It look

Re: [Nfs-ganesha-devel] NFSv4 referrals not working with ganesha.

gt; > > > It looks like Ganesha is doing the right thing though. > > > > Frank > > > > *From:* Pradeep [mailto:pradeep.tho...@gmail.com] > *Sent:* Monday, October 30, 2017 2:21 PM > *To:* Frank Filz > *Cc:* nfs-ganesha-devel ; > ssaurabh.w...

Re: [Nfs-ganesha-devel] NFSv4 referrals not working with ganesha.

ily works, but I’m > unable to test it. > > > > If you have code for FSAL_VFS to add the fs locations attribute, go ahead > and post it and I could poke at it. > > > > Also, tcpdump traces might help understand what is going wrong. > > > > Frank > > >

[Nfs-ganesha-devel] NFSv4 referrals not working with ganesha.

Hi all, We are testing NFSv4 referral for Linux CentOS 7 with nfs-ganesha and are running into some serious issues. Although, we were able to set up NFSv4 referral using the native Ganesha FSAL, we could not get it fully functional for all Linux client system calls. Basically, the NFSv4 spec sugg

[Nfs-ganesha-devel] Crash in mdcache_alloc_handle() during unexport

fore LRU_DQ_SAFE() in mdcache_lru_cleanup_push() should fix it too. Thanks, Pradeep -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_

[Nfs-ganesha-devel] Refreshing an export

block I/Os for the 'grace' period which is not ideal. Thanks, Pradeep -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org!

Re: [Nfs-ganesha-devel] Crash in TIRPC with Ganesha 2.6-dev.5

icz wrote: > >> On 08/30/2017 10:06 PM, Pradeep wrote: >> >>> Hi all, >>> >>> I'm hitting a crash in TIRPC with Ganesha 2.6-dev.5. It appears to me >>> that there is a race between a incoming RPC message on a new xprt (for >>> which ac

[Nfs-ganesha-devel] Crash in TIRPC with Ganesha 2.6-dev.5

Hi all, I'm hitting a crash in TIRPC with Ganesha 2.6-dev.5. It appears to me that there is a race between a incoming RPC message on a new xprt (for which accept() was done on the FD) and TIRPC setting the process_cb on the new xprt. We set the xprt->xp_dispatch.process_cb() from the rendezvous f

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

() to invoke lru_run_lane() if number of cached entries is above entries_hiwat. 2. Removed the limit on per_lane_work. There were some comments on limiting promotions (from L2 to L1 or within L1). Any suggestions on specific things to try out? Thanks, Pradeep > > Frank, this, in combination wit

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

Hi Daniel, I'm testing with 2.5.1. I haven't changed those parameters. Those parameters only affect once you are in lru_run_lane(), right? Since the FDs are lower than low-watermark, it never calls lru_run_lane(). Thanks, Pradeep On Fri, Aug 11, 2017 at 5:43 AM, Daniel Gryniew

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

is greater than low water mark. In my case, the open_fd_count is not high; so lru_run() doesn't call lru_run_lane() to demote those entries to L2. What is the best approach to handle this scenario? Thanks, Pradeep On Mon, Aug 7, 2017 at 6:08 AM, Daniel Gryniewicz wrote: > It n

Re: [Nfs-ganesha-devel] Dir_Chunk and Detached_Mult got values reversed?

mdcache_parameter, entries_hwmark), CONF_ITEM_UI32("Chunks_HWMark", 1, UINT32_MAX, 10, On Tue, Aug 8, 2017 at 5:58 AM, Daniel Gryniewicz wrote: > On 08/07/2017 11:24 PM, Pradeep wrote: > >> It appears that the commit 440a048887c99e17e2a582e8fb242d5c0a042a79 >>

Re: [Nfs-ganesha-devel] Change in ffilz/nfs-ganesha[next]: Ensure partition lock and qlane lock ordering

.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:691 #5 0x0051eac6 in lru_try_reap_entry () at /usr/src/debug/nfs-ganesha-2.5.1-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:718 Thanks, Pradeep On 8/7/17, GerritHub wrote: > From Daniel Gryniewicz : > > Daniel Gry

[Nfs-ganesha-devel] Dir_Chunk and Detached_Mult got values reversed?

It appears that the commit 440a048887c99e17e2a582e8fb242d5c0a042a79 has reversed two mdcache config parameters and as a result the default value of dir.avl_chunk became 1 from 128. Is this intentional? Ideally the "Dir_Chunk" should map to dir.avl_chunk and "Detached_Mult" should map to dir.avl_det

Re: [Nfs-ganesha-devel] mdcache growing beyond limits.

My mistake. As you both correctly pointed out, refcnt needs to be 1 for reclaim. It is initialized with 2. So some must be doing an unref()/put() to make it 1. On 8/4/17, Daniel Gryniewicz wrote: > On 08/04/2017 01:14 PM, Pradeep wrote: >> Hello, >> >> I'm hitting

[Nfs-ganesha-devel] mdcache growing beyond limits.

to claim it. Any ideas? Thanks, Pradeep -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___

Re: [Nfs-ganesha-devel] assert() in mdcache_lru_clean()

? On 8/4/17, Daniel Gryniewicz wrote: > Yep. We save the old context, run our loop, and then restore the old > context. > > On 08/04/2017 10:45 AM, Pradeep wrote: >> Thanks Daniel. I see it being initialized. But then it is overwritten >> from saved_ctx, right? >

Re: [Nfs-ganesha-devel] assert() in mdcache_lru_clean()

s-ganesha/blob/next/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c#L1127 > > On 08/04/2017 10:36 AM, Pradeep wrote: >> Hi Daniel, >> >> I could not find where op_ctx gets populated in lru_run_lane(). I'm using >> 2.5.1. >> >> https://github.com/nf

Re: [Nfs-ganesha-devel] assert() in mdcache_lru_clean()

p op_ctx, so it should be set > correctly even in the LRU thread case. > > Daniel > > On 08/04/2017 09:54 AM, Pradeep wrote: >> It looks like the assert() below and the comment in >> mdcache_lru_clean() may not be valid in all cases. For example, if >> cache is

[Nfs-ganesha-devel] assert() in mdcache_lru_clean()

It looks like the assert() below and the comment in mdcache_lru_clean() may not be valid in all cases. For example, if cache is getting cleaned in the context of the LRU background thread, the op_ctx will be NULL and the code may get into the 'else' part (lru_run() -> lru_run_lane() -> _mdcache_lru

Re: [Nfs-ganesha-devel] deadlock in lru_reap_impl()

goto next_lane; > > > > It could: > > > > QUNLOCK(qlane); > > mdcache_put(entry); > > continue; > > > > Fix posted here: > > > > https://review.gerrithub.io/371764 > > > > Frank > > > > > > *From:* Pradee

[Nfs-ganesha-devel] deadlock in lru_reap_impl()

I'm hitting another deadlock in mdcache with 2.5.1 base. In this case two threads are in different places in lru_reap_impl() Thread 1: 636 QLOCK(qlane); 637 lru = glist_first_entry(&lq->q, mdcache_lru_t, q); 638 if (!lru) 639

[Nfs-ganesha-devel] deadlock between mdcache_rename() and mdcache_getattrs()

Hello, I seem to be hitting a deadlock in Ganesha mdcache. I have debugged it and here is what I see on the hung process: Thread1 is doing a rename operation. Lets say file1 being moved from 'dir0' to 'dir1'. This calls mdcache_rename() - Holds content_lock for both parents - dir0 and dir1 - In m