Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Yan, Zheng]

On Tue, Mar 27, 2018 at 12:16 AM, Josh Haft  wrote:
> Here's what I'm seeing using basic owner/group permissions. Both
> directories are mounted on my NFS client with the same options. Only
> difference is underneath, from the NFS server, 'aclsupport' is mounted
> via ceph-fuse with fuse_default_permissions=0 (acls enabled), and
> 'noaclsupport' is mounted via ceph-fuse with
> fuse_default_permissions=1.
>
> 'user2' is part of 'group1' and should have r/w access to 'dir', but
> does not when trying to access the filesystem mounted with ACL
> support.
>
> [user2@test01 ]$ groups
> user2 group1
>
> [user2@test01 ]$ stat -c "%i" /mnt/cephfs/aclsupport/dir/
> 1099511790134
> [user2@test01 ]$ stat -c "%i" /mnt/cephfs/noaclsupport/dir/
> 1099511790134
>
> [user2@test01 ]$ ls -lh /mnt/cephfs/aclsupport
> total 1.5K
> drwxrws--- 1 user1   group1  0 Mar 22 15:32 dir
>
> [user2@test01 ]$ ls /mnt/cephfs/aclsupport/dir/
> ls: reading directory /mnt/cephfs/aclsupport/dir/: Permission denied
>
> [user2@test01 ]$ ls /mnt/cephfs/noaclsupport/dir/
> foo
>

This is expected behaviour. When fuse_default_permissions=0, all
permission checks are done in ceph-fuse. In your case, ceph-fuse can't
find which groups request initiator are in. This is due to limitation
of fuse API. I don't have idea how to fix it.

Regards
Yan, Zheng


> On Sat, Mar 24, 2018 at 3:26 AM, Yan, Zheng  wrote:
>> On Sat, Mar 24, 2018 at 11:34 AM, Josh Haft  wrote:
>>>
>>>
>>> On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng  wrote:

 On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft  wrote:
 > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
 >>
 >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
 >> > Hello!
 >> >
 >> > I'm running Ceph 12.2.2 with one primary and one standby MDS.
 >> > Mounting
 >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
 >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
 >> > options. I then export this mount via NFS and the clients mount
 >> > NFS4.1.
 >> >
 >> does fuse_default_permissions=0 work?
 >
 > Yes, ACLs work as expected when I set fuse_default_permissions=0.
 >
 >> > After doing some in-depth testing it seems I'm unable to allow access
 >> > from
 >> > the NFS clients to a directory/file based on group membership when
 >> > the
 >> > underlying CephFS was mounted with ACL support. This issue appears
 >> > using
 >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
 >> > ACLs do
 >> > work if the principal is a user instead of a group. If I disable ACL
 >> > support
 >> > on the ceph-fuse mount, things work as expected using fs permissions;
 >> > obviously I don't get ACL support.
 >> >
 >> > As an intermediate step I did check whether this works directly on
 >> > the
 >> > CephFS filesystem - on the NFS server - and it does. So it appears to
 >> > be an
 >> > issue re-exporting it via NFS.
 >> >
 >> > I do not see this issue when mounting CephFS via the kernel,
 >> > exporting via
 >> > NFS, and re-running these tests.
 >> >
 >> > I searched the ML and bug reports but only found this -
 >> > http://tracker.ceph.com/issues/12617 - which seems close to the issue
 >> > I'm
 >> > running into, but was closed as resolved 2+ years ago.
 >> >
 >> > Has anyone else run into this? Am I missing something obvious?
 >> >
 >>
 >> ceph-fuse does permission check according to localhost's config of
 >> supplement group. that's why you see this behavior.
 >
 > You're saying both the NFS client and server (where ceph-fuse is
 > running) need to use the same directory backend? (they are)
 > I should have mentioned I'm using LDAP/AD on client and server, so I
 > don't think that is the problem.
 >
 > Either way, I would not expect the behavior to change simply by
 > enabling ACLs, especially when I'm using filesystem permissions, and
 > ACLs aren't part of the equation.

 More specifically, ceph-fuse find which groups request initiator are
 in by function fuse_req_getgroups(). this function does tricks on
 "/proc/%lu/task/%lu/status".  It only works  when nfs client and
 ceph-fuse are running on the same machine.

>>> So why does this work when I'm using ceph-fuse but ACLs are disabled?

>>
>> Really?
>>
>> Please check if supplement groups work for inodes without ACL (mount
>> fuse with config option fuse_default_permissions=0)
>>
>>

 >> Yan, Zheng
 >>
 >> > Thanks!
 >> > Josh
 >> >
 >> >
 >> > ___
 >> > ceph-users mailing list
 >> > ceph-users@lists.ceph.com
 >> > 

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

Here's what I'm seeing using basic owner/group permissions. Both
directories are mounted on my NFS client with the same options. Only
difference is underneath, from the NFS server, 'aclsupport' is mounted
via ceph-fuse with fuse_default_permissions=0 (acls enabled), and
'noaclsupport' is mounted via ceph-fuse with
fuse_default_permissions=1.

'user2' is part of 'group1' and should have r/w access to 'dir', but
does not when trying to access the filesystem mounted with ACL
support.

[user2@test01 ]$ groups
user2 group1

[user2@test01 ]$ stat -c "%i" /mnt/cephfs/aclsupport/dir/
1099511790134
[user2@test01 ]$ stat -c "%i" /mnt/cephfs/noaclsupport/dir/
1099511790134

[user2@test01 ]$ ls -lh /mnt/cephfs/aclsupport
total 1.5K
drwxrws--- 1 user1   group1  0 Mar 22 15:32 dir

[user2@test01 ]$ ls /mnt/cephfs/aclsupport/dir/
ls: reading directory /mnt/cephfs/aclsupport/dir/: Permission denied

[user2@test01 ]$ ls /mnt/cephfs/noaclsupport/dir/
foo

On Sat, Mar 24, 2018 at 3:26 AM, Yan, Zheng  wrote:
> On Sat, Mar 24, 2018 at 11:34 AM, Josh Haft  wrote:
>>
>>
>> On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng  wrote:
>>>
>>> On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft  wrote:
>>> > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
>>> >>
>>> >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
>>> >> > Hello!
>>> >> >
>>> >> > I'm running Ceph 12.2.2 with one primary and one standby MDS.
>>> >> > Mounting
>>> >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
>>> >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
>>> >> > options. I then export this mount via NFS and the clients mount
>>> >> > NFS4.1.
>>> >> >
>>> >> does fuse_default_permissions=0 work?
>>> >
>>> > Yes, ACLs work as expected when I set fuse_default_permissions=0.
>>> >
>>> >> > After doing some in-depth testing it seems I'm unable to allow access
>>> >> > from
>>> >> > the NFS clients to a directory/file based on group membership when
>>> >> > the
>>> >> > underlying CephFS was mounted with ACL support. This issue appears
>>> >> > using
>>> >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
>>> >> > ACLs do
>>> >> > work if the principal is a user instead of a group. If I disable ACL
>>> >> > support
>>> >> > on the ceph-fuse mount, things work as expected using fs permissions;
>>> >> > obviously I don't get ACL support.
>>> >> >
>>> >> > As an intermediate step I did check whether this works directly on
>>> >> > the
>>> >> > CephFS filesystem - on the NFS server - and it does. So it appears to
>>> >> > be an
>>> >> > issue re-exporting it via NFS.
>>> >> >
>>> >> > I do not see this issue when mounting CephFS via the kernel,
>>> >> > exporting via
>>> >> > NFS, and re-running these tests.
>>> >> >
>>> >> > I searched the ML and bug reports but only found this -
>>> >> > http://tracker.ceph.com/issues/12617 - which seems close to the issue
>>> >> > I'm
>>> >> > running into, but was closed as resolved 2+ years ago.
>>> >> >
>>> >> > Has anyone else run into this? Am I missing something obvious?
>>> >> >
>>> >>
>>> >> ceph-fuse does permission check according to localhost's config of
>>> >> supplement group. that's why you see this behavior.
>>> >
>>> > You're saying both the NFS client and server (where ceph-fuse is
>>> > running) need to use the same directory backend? (they are)
>>> > I should have mentioned I'm using LDAP/AD on client and server, so I
>>> > don't think that is the problem.
>>> >
>>> > Either way, I would not expect the behavior to change simply by
>>> > enabling ACLs, especially when I'm using filesystem permissions, and
>>> > ACLs aren't part of the equation.
>>>
>>> More specifically, ceph-fuse find which groups request initiator are
>>> in by function fuse_req_getgroups(). this function does tricks on
>>> "/proc/%lu/task/%lu/status".  It only works  when nfs client and
>>> ceph-fuse are running on the same machine.
>>>
>> So why does this work when I'm using ceph-fuse but ACLs are disabled?
>>>
>
> Really?
>
> Please check if supplement groups work for inodes without ACL (mount
> fuse with config option fuse_default_permissions=0)
>
>
>>>
>>> >> Yan, Zheng
>>> >>
>>> >> > Thanks!
>>> >> > Josh
>>> >> >
>>> >> >
>>> >> > ___
>>> >> > ceph-users mailing list
>>> >> > ceph-users@lists.ceph.com
>>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >> >
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Yan, Zheng]

On Sat, Mar 24, 2018 at 11:34 AM, Josh Haft  wrote:
>
>
> On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng  wrote:
>>
>> On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft  wrote:
>> > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
>> >>
>> >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
>> >> > Hello!
>> >> >
>> >> > I'm running Ceph 12.2.2 with one primary and one standby MDS.
>> >> > Mounting
>> >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
>> >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
>> >> > options. I then export this mount via NFS and the clients mount
>> >> > NFS4.1.
>> >> >
>> >> does fuse_default_permissions=0 work?
>> >
>> > Yes, ACLs work as expected when I set fuse_default_permissions=0.
>> >
>> >> > After doing some in-depth testing it seems I'm unable to allow access
>> >> > from
>> >> > the NFS clients to a directory/file based on group membership when
>> >> > the
>> >> > underlying CephFS was mounted with ACL support. This issue appears
>> >> > using
>> >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
>> >> > ACLs do
>> >> > work if the principal is a user instead of a group. If I disable ACL
>> >> > support
>> >> > on the ceph-fuse mount, things work as expected using fs permissions;
>> >> > obviously I don't get ACL support.
>> >> >
>> >> > As an intermediate step I did check whether this works directly on
>> >> > the
>> >> > CephFS filesystem - on the NFS server - and it does. So it appears to
>> >> > be an
>> >> > issue re-exporting it via NFS.
>> >> >
>> >> > I do not see this issue when mounting CephFS via the kernel,
>> >> > exporting via
>> >> > NFS, and re-running these tests.
>> >> >
>> >> > I searched the ML and bug reports but only found this -
>> >> > http://tracker.ceph.com/issues/12617 - which seems close to the issue
>> >> > I'm
>> >> > running into, but was closed as resolved 2+ years ago.
>> >> >
>> >> > Has anyone else run into this? Am I missing something obvious?
>> >> >
>> >>
>> >> ceph-fuse does permission check according to localhost's config of
>> >> supplement group. that's why you see this behavior.
>> >
>> > You're saying both the NFS client and server (where ceph-fuse is
>> > running) need to use the same directory backend? (they are)
>> > I should have mentioned I'm using LDAP/AD on client and server, so I
>> > don't think that is the problem.
>> >
>> > Either way, I would not expect the behavior to change simply by
>> > enabling ACLs, especially when I'm using filesystem permissions, and
>> > ACLs aren't part of the equation.
>>
>> More specifically, ceph-fuse find which groups request initiator are
>> in by function fuse_req_getgroups(). this function does tricks on
>> "/proc/%lu/task/%lu/status".  It only works  when nfs client and
>> ceph-fuse are running on the same machine.
>>
> So why does this work when I'm using ceph-fuse but ACLs are disabled?
>>

Really?

Please check if supplement groups work for inodes without ACL (mount
fuse with config option fuse_default_permissions=0)


>>
>> >> Yan, Zheng
>> >>
>> >> > Thanks!
>> >> > Josh
>> >> >
>> >> >
>> >> > ___
>> >> > ceph-users mailing list
>> >> > ceph-users@lists.ceph.com
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

On Fri, Mar 23, 2018 at 8:49 PM, Yan, Zheng  wrote:

> On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft  wrote:
> > On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
> >>
> >> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
> >> > Hello!
> >> >
> >> > I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
> >> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
> >> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
> >> > options. I then export this mount via NFS and the clients mount
> NFS4.1.
> >> >
> >> does fuse_default_permissions=0 work?
> >
> > Yes, ACLs work as expected when I set fuse_default_permissions=0.
> >
> >> > After doing some in-depth testing it seems I'm unable to allow access
> from
> >> > the NFS clients to a directory/file based on group membership when the
> >> > underlying CephFS was mounted with ACL support. This issue appears
> using
> >> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However,
> ACLs do
> >> > work if the principal is a user instead of a group. If I disable ACL
> support
> >> > on the ceph-fuse mount, things work as expected using fs permissions;
> >> > obviously I don't get ACL support.
> >> >
> >> > As an intermediate step I did check whether this works directly on the
> >> > CephFS filesystem - on the NFS server - and it does. So it appears to
> be an
> >> > issue re-exporting it via NFS.
> >> >
> >> > I do not see this issue when mounting CephFS via the kernel,
> exporting via
> >> > NFS, and re-running these tests.
> >> >
> >> > I searched the ML and bug reports but only found this -
> >> > http://tracker.ceph.com/issues/12617 - which seems close to the
> issue I'm
> >> > running into, but was closed as resolved 2+ years ago.
> >> >
> >> > Has anyone else run into this? Am I missing something obvious?
> >> >
> >>
> >> ceph-fuse does permission check according to localhost's config of
> >> supplement group. that's why you see this behavior.
> >
> > You're saying both the NFS client and server (where ceph-fuse is
> > running) need to use the same directory backend? (they are)
> > I should have mentioned I'm using LDAP/AD on client and server, so I
> > don't think that is the problem.
> >
> > Either way, I would not expect the behavior to change simply by
> > enabling ACLs, especially when I'm using filesystem permissions, and
> > ACLs aren't part of the equation.
>
> More specifically, ceph-fuse find which groups request initiator are
> in by function fuse_req_getgroups(). this function does tricks on
> "/proc/%lu/task/%lu/status".  It only works  when nfs client and
> ceph-fuse are running on the same machine.
>
> So why does this work when I'm using ceph-fuse but ACLs are disabled?

>
> >> Yan, Zheng
> >>
> >> > Thanks!
> >> > Josh
> >> >
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Yan, Zheng]

On Fri, Mar 23, 2018 at 9:50 PM, Josh Haft  wrote:
> On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
>>
>> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
>> > Hello!
>> >
>> > I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
>> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
>> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
>> > options. I then export this mount via NFS and the clients mount NFS4.1.
>> >
>> does fuse_default_permissions=0 work?
>
> Yes, ACLs work as expected when I set fuse_default_permissions=0.
>
>> > After doing some in-depth testing it seems I'm unable to allow access from
>> > the NFS clients to a directory/file based on group membership when the
>> > underlying CephFS was mounted with ACL support. This issue appears using
>> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However, ACLs do
>> > work if the principal is a user instead of a group. If I disable ACL 
>> > support
>> > on the ceph-fuse mount, things work as expected using fs permissions;
>> > obviously I don't get ACL support.
>> >
>> > As an intermediate step I did check whether this works directly on the
>> > CephFS filesystem - on the NFS server - and it does. So it appears to be an
>> > issue re-exporting it via NFS.
>> >
>> > I do not see this issue when mounting CephFS via the kernel, exporting via
>> > NFS, and re-running these tests.
>> >
>> > I searched the ML and bug reports but only found this -
>> > http://tracker.ceph.com/issues/12617 - which seems close to the issue I'm
>> > running into, but was closed as resolved 2+ years ago.
>> >
>> > Has anyone else run into this? Am I missing something obvious?
>> >
>>
>> ceph-fuse does permission check according to localhost's config of
>> supplement group. that's why you see this behavior.
>
> You're saying both the NFS client and server (where ceph-fuse is
> running) need to use the same directory backend? (they are)
> I should have mentioned I'm using LDAP/AD on client and server, so I
> don't think that is the problem.
>
> Either way, I would not expect the behavior to change simply by
> enabling ACLs, especially when I'm using filesystem permissions, and
> ACLs aren't part of the equation.

More specifically, ceph-fuse find which groups request initiator are
in by function fuse_req_getgroups(). this function does tricks on
"/proc/%lu/task/%lu/status".  It only works  when nfs client and
ceph-fuse are running on the same machine.


>> Yan, Zheng
>>
>> > Thanks!
>> > Josh
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Josh Haft]

On Fri, Mar 23, 2018 at 12:14 AM, Yan, Zheng  wrote:
>
> On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
> > Hello!
> >
> > I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
> > CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
> > fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
> > options. I then export this mount via NFS and the clients mount NFS4.1.
> >
> does fuse_default_permissions=0 work?

Yes, ACLs work as expected when I set fuse_default_permissions=0.

> > After doing some in-depth testing it seems I'm unable to allow access from
> > the NFS clients to a directory/file based on group membership when the
> > underlying CephFS was mounted with ACL support. This issue appears using
> > both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However, ACLs do
> > work if the principal is a user instead of a group. If I disable ACL support
> > on the ceph-fuse mount, things work as expected using fs permissions;
> > obviously I don't get ACL support.
> >
> > As an intermediate step I did check whether this works directly on the
> > CephFS filesystem - on the NFS server - and it does. So it appears to be an
> > issue re-exporting it via NFS.
> >
> > I do not see this issue when mounting CephFS via the kernel, exporting via
> > NFS, and re-running these tests.
> >
> > I searched the ML and bug reports but only found this -
> > http://tracker.ceph.com/issues/12617 - which seems close to the issue I'm
> > running into, but was closed as resolved 2+ years ago.
> >
> > Has anyone else run into this? Am I missing something obvious?
> >
>
> ceph-fuse does permission check according to localhost's config of
> supplement group. that's why you see this behavior.

You're saying both the NFS client and server (where ceph-fuse is
running) need to use the same directory backend? (they are)
I should have mentioned I'm using LDAP/AD on client and server, so I
don't think that is the problem.

Either way, I would not expect the behavior to change simply by
enabling ACLs, especially when I'm using filesystem permissions, and
ACLs aren't part of the equation.

> Regards
> Yan, Zheng
>
> > Thanks!
> > Josh
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Group-based permissions issue when using ACLs on CephFS

[Yan, Zheng]

On Fri, Mar 23, 2018 at 5:14 AM, Josh Haft  wrote:
> Hello!
>
> I'm running Ceph 12.2.2 with one primary and one standby MDS. Mounting
> CephFS via ceph-fuse (to leverage quotas), and enabled ACLs by adding
> fuse_default_permissions=0 and client_acl_type=posix_acl to the mount
> options. I then export this mount via NFS and the clients mount NFS4.1.
>
does fuse_default_permissions=0 work?

> After doing some in-depth testing it seems I'm unable to allow access from
> the NFS clients to a directory/file based on group membership when the
> underlying CephFS was mounted with ACL support. This issue appears using
> both filesystem permissions (e.g. chgrp) and NFSv4 ACLs. However, ACLs do
> work if the principal is a user instead of a group. If I disable ACL support
> on the ceph-fuse mount, things work as expected using fs permissions;
> obviously I don't get ACL support.
>
> As an intermediate step I did check whether this works directly on the
> CephFS filesystem - on the NFS server - and it does. So it appears to be an
> issue re-exporting it via NFS.
>
> I do not see this issue when mounting CephFS via the kernel, exporting via
> NFS, and re-running these tests.
>
> I searched the ML and bug reports but only found this -
> http://tracker.ceph.com/issues/12617 - which seems close to the issue I'm
> running into, but was closed as resolved 2+ years ago.
>
> Has anyone else run into this? Am I missing something obvious?
>

ceph-fuse does permission check according to localhost's config of
supplement group. that's why you see this behavior.

Regards
Yan, Zheng

> Thanks!
> Josh
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com