[Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-01 Thread Satya Prakash GS
Hi, I am seeing "Permission denied" errors while running iozone on nfs client with kerberos enabled. Digging further, I found there are a lot of AUTH_REJECTEDCRED messages in nfs server log. NFS client tolerates 2 errors from server and tries to refresh the credentials. On the third call it would

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-06 Thread Satya Prakash GS
With libntirpc debugs enabled I could see all the three retries are failing because of the unavailability of the creds in the cache. The credentials are being removed by the reaper in the authgss_ctx_gc_idle because of this condition - abs(axp->gen - gd->gen) > __svc_params->gss.max_idle_gen >From

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-06 Thread Matt Benjamin
fs-ganesha-devel@lists.sourceforge.net > Sent: Monday, March 6, 2017 1:10:36 PM > Subject: Re: [Nfs-ganesha-devel] Permission denied error with Kerberos > enabled > > With libntirpc debugs enabled I could see all the three retries are > failing because of the unavailabil

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-07 Thread William Allen Simpson
On 3/6/17 6:58 PM, Matt Benjamin wrote: > Looking briefly at section 5.3.3.3 of rfc2203, it seems like that would be > correct. If the client has just refreshed its credentials, why is it > continuing to send with the expired context? > I don't know, but I'll take a look. Now that we always hav

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-07 Thread William Allen Simpson
On 3/7/17 4:56 AM, William Allen Simpson wrote: > On 3/6/17 6:58 PM, Matt Benjamin wrote: >> Looking briefly at section 5.3.3.3 of rfc2203, it seems like that would be >> correct. If the client has just refreshed its credentials, why is it >> continuing to send with the expired context? >> > I d

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-07 Thread Satya Prakash GS
"Satya Prakash GS" >> To: nfs-ganesha-devel@lists.sourceforge.net >> Sent: Monday, March 6, 2017 1:10:36 PM >> Subject: Re: [Nfs-ganesha-devel] Permission denied error with Kerberos >> enabled >> >> With libntirpc debugs enabled I could see al

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-09 Thread Satya Prakash GS
ike that would be >> correct. If the client has just refreshed its credentials, why is it >> continuing to send with the expired context? >> >> Matt >> >> - Original Message - >>> From: "Satya Prakash GS" >>> To: nf

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-10 Thread Satya Prakash GS
Is this a possibility : Server first rejects a client op with CREDPROBLEM/REJECTEDCRED, Client does an upcall and gssd initializes the context with the server. However the server recycles it immediately before the operation was retried (looks like there is a bug in the LRU implementation on Ganesh

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-10 Thread William Allen Simpson
I'm not familiar with this code, so not likely to be much help. Looks mostly written by Matt, but Malahal made the most recent changes in July 2016. On 3/10/17 9:35 AM, Satya Prakash GS wrote: > Is this a possibility : > > Server first rejects a client op with CREDPROBLEM/REJECTEDCRED, > Client do

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-10 Thread Satya Prakash GS
On Sat, Mar 11, 2017 at 12:37 AM, William Allen Simpson wrote: > I'm not familiar with this code, so not likely to be much help. > Looks mostly written by Matt, but Malahal made the most recent > changes in July 2016. > > On 3/10/17 9:35 AM, Satya Prakash GS wrote: >> >> Is this a possibility : >>

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-11 Thread Malahal Naineni
gd->gen is not used in the latest code. If I remember, there was a bug removing recent cached entries resulting in permission errors. What version are you using? Try using V2.5. Regards, Malahal. On Sat, Mar 11, 2017 at 12:54 AM, Satya Prakash GS wrote: > On Sat, Mar 11, 2017 at 12:37 AM, Willia

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-11 Thread Satya Prakash GS
We are using 2.3-stable. Given that most of our testing has been done it's a bit difficult for us to move to 2.5 now but we can take fixes from 2.5. I put a similar fix (to the existing one in 2.5) but I am running into another issue around the ticket renew time. Operations are failing on the clie

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-12 Thread William Allen Simpson
On 3/11/17 8:15 AM, Satya Prakash GS wrote: > We are using 2.3-stable. Given that most of our testing has been done > it's a bit difficult for us to move to 2.5 now but we can take fixes > from 2.5. > Sorry, I should have asked long ago what version you were using. On this list, I always assume th

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-12 Thread Malahal Naineni
>> Indeed, 2.4 was mostly a bug fix release Actually, 2.4 has couple big features as far as ganesha project is concerned, but Bill is probably indicating that libntirpc corresponding to ganesha2.4 is mostly bug fix release. Regards, Malahal. On Sun, Mar 12, 2017 at 8:15 PM, William Allen Simpso

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-13 Thread Satya Prakash GS
My bad, I should have mentioned the version in the original post. Mahalal was kind enough to share a list of relevant commits. With the patches I continued to see the issue. I suspect the client code is not handling GSS_S_CONTEXT_EXPIRED correctly on a call to gss_verify_mic. Instead I fixed the s

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-16 Thread Satya Prakash GS
Has anyone seen client ops failing with error -13 because of context expiry on client (gss_verify_mic fails). Surprisingly with little load, it's consistently reproducible on my setup. Can someone point me to the relevant commits if this has already been fixed. Thanks, Satya. On Mon, Mar 13, 2017

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-19 Thread Malahal Naineni
If I understand, you have renewable ticket and commands fail when the ticket expires? I will let our folks tests it. Any more details on reproducing this issue. On Fri, Mar 17, 2017 at 9:59 AM, Satya Prakash GS wrote: > Has anyone seen client ops failing with error -13 because of context > expiry

Re: [Nfs-ganesha-devel] Permission denied error with Kerberos enabled

2017-03-21 Thread Satya Prakash GS
Here are the reproduction steps: I have 3 different servers hosting nfs client, server and KDC. I set the ticket lifetime to 10 minutes on the client and server (in krb5.conf). When adding a principal I used specified "-maxlife "10 minutes" -maxrenew 2017-04-30". I specified max_life (to 10 mins)