Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Kinglong Mee]

On 2018/10/11 19:09, Soumya Koduri wrote:
> NFS-Ganesha's md-cache layer already does extensive caching of attributes and 
> ACLs of each file looked upon. Do you see any additional benefit with turning 
> on gluster md-cache as well?  More replies inline..

Yes, I think.

The logical is different between md-cache and ganesha's cache,
Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from 
back-end glusterfs;
Md-cache caches depneds on timeout too, but md-cache can delay the timeout for 
some cases.

> 
> On 10/11/18 7:47 AM, Kinglong Mee wrote:
>> Cc nfs-ganesha,
>>
>> Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
>> ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
>> ACLs
>> ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
>>
>> But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
>> requests.
>> So, md-cache caches bad virtual glusterfs ACLs.
>>
>> After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs 
>> client gets many EIO errors.
>>
>> https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
>>
>> There are two chooses for cache virtual glusterfs ACLs in md-cache,
>> 1. Cache it separately as posix ACLs (a new option maybe 
>> "cache-glusterfs-acl" is added);
>>     And make sure _posix_xattr_get_set fills them when lookup requests.
>>
> 
> I am not sure if posix layer can handle it. Virtual xattrs are in-memory and 
> not stored on disk. They are converted to/from posix-acl in posix-acl xlator. 
> So FWIU, posix-acl xlator should handle setting these attributes as part of 
> LOOKUP response if needed. Same shall apply for any virtual xattr cached in 
> md-cache. Request Poornima to comment.

Posix-acl can hand it correctly now.

> 
> At a time, any gfapi consumer would use either posix-acl or virtual glusterfs 
> ACLs. So having two options to selectively choose which one of them to cache 
> sounds better to me instead of unnecessarily storing two different 
> representations of the same ACL.

Make sense.
I will add another option for virtual glusterfs ACLs in md-cache.

thanks,
Kinglong Mee

> 
> Thanks,
> Soumya
> 
>> 2. Does not cache it, only cache posix ACLs;
>>     If gfapi request it, md-cache lookup according posix ACL at cache,
>>     if exist, make the virtual glusterfs ACL locally and return to gfapi;
>>     otherwise, send the request to glusterfsd.
>>
>> Virtual glusterfs ACLs are another format of posix ACLs, there are larger 
>> than posix ACLs,
>> and always exist no matter the really posix ACL exist or not.
> 
>>
>> So, I'd prefer #2.
>> Any comments are welcome.
>>
>> thanks,
>> Kinglong Mee
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Soumya Koduri]

NFS-Ganesha's md-cache layer already does extensive caching of 
attributes and ACLs of each file looked upon. Do you see any additional 
benefit with turning on gluster md-cache as well?  More replies inline..


On 10/11/18 7:47 AM, Kinglong Mee wrote:

Cc nfs-ganesha,

Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
ACLs
("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.

But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
requests.
So, md-cache caches bad virtual glusterfs ACLs.

After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client 
gets many EIO errors.

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

There are two chooses for cache virtual glusterfs ACLs in md-cache,
1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" 
is added);
And make sure _posix_xattr_get_set fills them when lookup requests.



I am not sure if posix layer can handle it. Virtual xattrs are in-memory 
and not stored on disk. They are converted to/from posix-acl in 
posix-acl xlator. So FWIU, posix-acl xlator should handle setting these 
attributes as part of LOOKUP response if needed. Same shall apply for 
any virtual xattr cached in md-cache. Request Poornima to comment.


At a time, any gfapi consumer would use either posix-acl or virtual 
glusterfs ACLs. So having two options to selectively choose which one of 
them to cache sounds better to me instead of unnecessarily storing two 
different representations of the same ACL.


Thanks,
Soumya


2. Does not cache it, only cache posix ACLs;
If gfapi request it, md-cache lookup according posix ACL at cache,
if exist, make the virtual glusterfs ACL locally and return to gfapi;
otherwise, send the request to glusterfsd.

Virtual glusterfs ACLs are another format of posix ACLs, there are larger than 
posix ACLs,
and always exist no matter the really posix ACL exist or not.




So, I'd prefer #2.
Any comments are welcome.

thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Kinglong Mee]

On 2018/10/11 12:10, Jiffin Tony Thottan wrote:> On Thursday 11 October 2018 
08:10 AM, Raghavendra Gowdappa wrote:
>> On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee > > wrote:
>>
>> Cc nfs-ganesha,
>>
>> Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
>> ("system.posix_acl_access"/"system.posix_acl_default") and virtual 
>> glusterfs ACLs
>> ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
>>
>>
>> Not sure how virtual xattrs are consumed or who generates them. +Raghavendra 
>> Talur  +Thottan, Jiffin 
>>  - acl maintainers.
> 
> The currently only consumers of this virtual xattr is nfs-ganesha. Nfsv4 acls 
> were sent from client and ganesha converts to posix acl
> 
> and sent as virtual xattr to glusterfs bricks using pub_glfs_h_acl_set/get 
> api's. AFAIR  in samba vfs module they convert windows acl to
> 
> posix acl and sent as actual getxattr/setxattr calls on "system.posixl-acl"
> 
> --
> 
> Jiffin
> 
>>
>>
>> But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when 
>> lookup requests.
>> So, md-cache caches bad virtual glusterfs ACLs.
>>
>> After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs 
>> client gets many EIO errors.
>>
>> https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

Hi Jiffin,

Thanks for your explain.

I'm testing the cache of virtual glusterfs ACLs in md-cache for ganesha's nfsv4.
I wanna to know the plan of ACLs in the future.
Are both posix ACLs (used by samba) and virtual glusterfs ACLs (used by 
ganesha's nfsv4) reserved?
Or Is only virtual glusterfs ACLs reserved?

If both are reserved, there are two chooses for md-cache caches virtual 
glusterfs ACLs.
>> There are two chooses for cache virtual glusterfs ACLs in md-cache,
>> 1. Cache it separately as posix ACLs (a new option maybe 
>> "cache-glusterfs-acl" is added);
>>    And make sure _posix_xattr_get_set fills them when lookup requests.
>>
>> 2. Does not cache it, only cache posix ACLs;
>>    If gfapi request it, md-cache lookup according posix ACL at cache,
>>    if exist, make the virtual glusterfs ACL locally and return to gfapi;
>>    otherwise, send the request to glusterfsd.
>>
>> Virtual glusterfs ACLs are another format of posix ACLs, there are 
>> larger than posix ACLs,
>> and always exist no matter the really posix ACL exist or not.
>>
>> So, I'd prefer #2.

If only virtual glusterfs ACLs is reserved, I'd like chose the #1 implement for 
md-cache.
Md-cache reserves caching posix acl until samba moving to the virtual glusterfs 
ACLs.

thanks,
Kinglong Mee
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Jiffin Tony Thottan]

On Thursday 11 October 2018 08:10 AM, Raghavendra Gowdappa wrote:



On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee > wrote:


Cc nfs-ganesha,

Md-cache has option "cache-posix-acl" that controls caching of
posix ACLs
("system.posix_acl_access"/"system.posix_acl_default") and virtual
glusterfs ACLs
("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.


Not sure how virtual xattrs are consumed or who generates them. 
+Raghavendra Talur  +Thottan, Jiffin 
 - acl maintainers.


The currently only consumers of this virtual xattr is nfs-ganesha. Nfsv4 
acls were sent from client and ganesha converts to posix acl


and sent as virtual xattr to glusterfs bricks using 
pub_glfs_h_acl_set/get api's. AFAIR  in samba vfs module they convert 
windows acl to


posix acl and sent as actual getxattr/setxattr calls on "system.posixl-acl"

--

Jiffin




But, _posix_xattr_get_set does not fill virtual glusterfs ACLs
when lookup requests.
So, md-cache caches bad virtual glusterfs ACLs.

After I turn on "cache-posix-acl" option to cache ACLs at
md-cache, nfs client gets many EIO errors.

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

There are two chooses for cache virtual glusterfs ACLs in md-cache,
1. Cache it separately as posix ACLs (a new option maybe
"cache-glusterfs-acl" is added);
   And make sure _posix_xattr_get_set fills them when lookup requests.

2. Does not cache it, only cache posix ACLs;
   If gfapi request it, md-cache lookup according posix ACL at cache,
   if exist, make the virtual glusterfs ACL locally and return to
gfapi;
   otherwise, send the request to glusterfsd.

Virtual glusterfs ACLs are another format of posix ACLs, there are
larger than posix ACLs,
and always exist no matter the really posix ACL exist or not.

So, I'd prefer #2.
Any comments are welcome.

thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-devel



___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Raghavendra Gowdappa]

On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee  wrote:

> Cc nfs-ganesha,
>
> Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
> ("system.posix_acl_access"/"system.posix_acl_default") and virtual
> glusterfs ACLs
> ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
>

Not sure how virtual xattrs are consumed or who generates them. +Raghavendra
Talur  +Thottan, Jiffin  - acl
maintainers.


> But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup
> requests.
> So, md-cache caches bad virtual glusterfs ACLs.
>
> After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs
> client gets many EIO errors.
>
> https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
>
> There are two chooses for cache virtual glusterfs ACLs in md-cache,
> 1. Cache it separately as posix ACLs (a new option maybe
> "cache-glusterfs-acl" is added);
>And make sure _posix_xattr_get_set fills them when lookup requests.
>
> 2. Does not cache it, only cache posix ACLs;
>If gfapi request it, md-cache lookup according posix ACL at cache,
>if exist, make the virtual glusterfs ACL locally and return to gfapi;
>otherwise, send the request to glusterfsd.
>
> Virtual glusterfs ACLs are another format of posix ACLs, there are larger
> than posix ACLs,
> and always exist no matter the really posix ACL exist or not.
>
> So, I'd prefer #2.
> Any comments are welcome.
>
> thanks,
> Kinglong Mee
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Kinglong Mee]

Cc nfs-ganesha,

Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
ACLs
("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.

But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
requests.
So, md-cache caches bad virtual glusterfs ACLs.

After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client 
gets many EIO errors.

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

There are two chooses for cache virtual glusterfs ACLs in md-cache,
1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" 
is added);
   And make sure _posix_xattr_get_set fills them when lookup requests.

2. Does not cache it, only cache posix ACLs;
   If gfapi request it, md-cache lookup according posix ACL at cache,
   if exist, make the virtual glusterfs ACL locally and return to gfapi;
   otherwise, send the request to glusterfsd.

Virtual glusterfs ACLs are another format of posix ACLs, there are larger than 
posix ACLs, 
and always exist no matter the really posix ACL exist or not.

So, I'd prefer #2.
Any comments are welcome.

thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Kinglong Mee]

Md-cache has option "cache-posix-acl" that controls caching of posix ACLs 
("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
ACLs ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.

But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
requests.
So, md-cache caches bad virtual glusterfs ACLs.

After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client 
gets many EIO errors.

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

There are two chooses for cache virtual glusterfs ACLs in md-cache,
1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" 
is added);
   And make sure _posix_xattr_get_set fills them when lookup requests.

2. Does not cache it, only cache posix ACLs;
   If gfapi request it, md-cache lookup according posix ACL at cache,
   if exist, make the virtual glusterfs ACL locally and return to gfapi;
   otherwise, send the request to glusterfsd.

Virtual glusterfs ACLs are another format of posix ACLs, there are larger than 
posix ACLs, 
and always exist no matter the really posix ACL exist or not.

So, I'd prefer #2.
Any comments are welcome.

thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel