Re: [lustre-discuss] seclabel
On Tue, May 23, 2017 at 08:08:54PM +, Dilger, Andreas wrote: >On May 19, 2017, at 08:47, Robin Humble wrote: >> On Wed, May 17, 2017 at 02:37:31PM +, Sebastien Buisson wrote: >>> Le 17 mai 2017 à 16:16, Robin Humble a écrit : I took a gander at the source and noticed that llite/xattr.c deliberately filters out 'security.capability' and returns 0/-ENODATA for setcap/getcap, which is indeed what strace sees. so setcap/getcap is never even sent to the MDS. if I remove that filter (see patch on lustre-devel) then setcap/getcap works -> >> ... 'b15587' is listed as the reason for the filtering. I don't know what that refers to. is it still relevant? >>> b15587 refers to the old Lustre Bugzilla tracking tool: >>> https://projectlava.xyratex.com/show_bug.cgi?id=15587 >>> >>> Reading the discussion in the ticket, supporting xattr at the time of >>> Lustre 1.8 and 2.0 was causing issues on MDS side in some situations. So it >>> was decided to discard security.capability xattr on Lustre client side. I >>> think Andreas might have some insight, as he apparently participated in >>> b15587. >> >> my word that's a long time ago... >> I don't see much in the way of jira tickets about getxattr issues on >> MDS in recent times, and they're much more heavily used these days, so >> I hope that particular problem has long since been fixed. >> >> should I open a jira ticket to track re-enabling of security.capabilities? LU-9562 thanks for everyone's help! >I don't recall the details of b=15587 off the top of my head, but the >high-level issue is >that the security labels added a significant performance overhead, since they >were retrieved >on every file access, but not cached on the client, even if most systems never >used them. > >Seagate implemented the client-side xattr cache for Lustre 2.5, so this should >work a lot >better these days. I'm not 100% positive if we also cache negative xattr >lookups or not, >so this would need some testing/tracing to see if it generates a large number >of RPCs. fair enough. cheers, robin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
On May 19, 2017, at 08:47, Robin Humble wrote: > > Hi Sebastien, > > On Wed, May 17, 2017 at 02:37:31PM +, Sebastien Buisson wrote: >> Le 17 mai 2017 à 16:16, Robin Humble a écrit : >>> I took a gander at the source and noticed that llite/xattr.c >>> deliberately filters out 'security.capability' and returns 0/-ENODATA >>> for setcap/getcap, which is indeed what strace sees. so setcap/getcap >>> is never even sent to the MDS. >>> >>> if I remove that filter (see patch on lustre-devel) then setcap/getcap >>> works -> > ... >>> 'b15587' is listed as the reason for the filtering. >>> I don't know what that refers to. >>> is it still relevant? >> b15587 refers to the old Lustre Bugzilla tracking tool: >> https://projectlava.xyratex.com/show_bug.cgi?id=15587 >> >> Reading the discussion in the ticket, supporting xattr at the time of Lustre >> 1.8 and 2.0 was causing issues on MDS side in some situations. So it was >> decided to discard security.capability xattr on Lustre client side. I think >> Andreas might have some insight, as he apparently participated in b15587. > > my word that's a long time ago... > I don't see much in the way of jira tickets about getxattr issues on > MDS in recent times, and they're much more heavily used these days, so > I hope that particular problem has long since been fixed. > > should I open a jira ticket to track re-enabling of security.capabilities? I don't recall the details of b=15587 off the top of my head, but the high-level issue is that the security labels added a significant performance overhead, since they were retrieved on every file access, but not cached on the client, even if most systems never used them. Seagate implemented the client-side xattr cache for Lustre 2.5, so this should work a lot better these days. I'm not 100% positive if we also cache negative xattr lookups or not, so this would need some testing/tracing to see if it generates a large number of RPCs. > >> In any case, it is important to make clear that file capabilities, the >> feature you want to use, is completely distinct from SELinux. >> On the one hand, Capabilities are a Linux mechanism to refine permissions >> granted to privileged processes, by dividing the privileges traditionally >> associated with superuser into distinct units (known as capabilities). >> On the other hand, SELinux is the Linux implementation of Mandatory Access >> Control. >> Both Capabilities and SELinux rely on values stored into file extended >> attributes, but this is the only thing they have in common. > > 10-4. thanks. > > 'ls --color' requests the security.capability xattr so this would > be heavily accessed. do you think this is handled well enough currently > to not affect performance significantly? Good old "ls --color". The support for statx() _just_ got into the kernel after 10 years of trying, so that in another 5 years once "ls" is updated and in distros, then "ls --color" will not do an extra RPC for each file to get the file permission to color a file differently based on the executable status. I guess if this xattr is cached from open time (or negative cached if missing), then this wouldn't add significant overhead for a local getxattr call. It should be possible to turn on RPC tracing with "lctl set_param debug=+rpctrace" and count the RPCs for some operations with the patch applied and removed, and see what the differences are. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel Corporation ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
> Le 19 mai 2017 à 08:47, Robin Humble a écrit : > On Wed, May 17, 2017 at 02:37:31PM +, Sebastien Buisson wrote: >> >> Reading the discussion in the ticket, supporting xattr at the time of Lustre >> 1.8 and 2.0 was causing issues on MDS side in some situations. So it was >> decided to discard security.capability xattr on Lustre client side. I think >> Andreas might have some insight, as he apparently participated in b15587. > > my word that's a long time ago... > I don't see much in the way of jira tickets about getxattr issues on > MDS in recent times, and they're much more heavily used these days, so > I hope that particular problem has long since been fixed. > > should I open a jira ticket to track re-enabling of security.capabilities? Yes please. At the same time, would you mind pushing to review.whamcloud.com the patch you sent to lustre-devel? > >> In any case, it is important to make clear that file capabilities, the >> feature you want to use, is completely distinct from SELinux. >> On the one hand, Capabilities are a Linux mechanism to refine permissions >> granted to privileged processes, by dividing the privileges traditionally >> associated with superuser into distinct units (known as capabilities). >> On the other hand, SELinux is the Linux implementation of Mandatory Access >> Control. >> Both Capabilities and SELinux rely on values stored into file extended >> attributes, but this is the only thing they have in common. > > 10-4. thanks. > > 'ls --color' requests the security.capability xattr so this would > be heavily accessed. do you think this is handled well enough currently > to not affect performance significantly? > > setxattr would be minimal and not performance critical, unlike with eg. > selinux and creat. > For sure retrieving this xattr adds an overhead. But with recent versions of Lustre I am not aware of bugs in xattr handling. I think it would be helpful, in the Jira ticket you would open, to have comments from people that participated in the resolution of Bugzilla #15587. Thanks, Sebastien. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
Hi Sebastien, On Wed, May 17, 2017 at 02:37:31PM +, Sebastien Buisson wrote: > Le 17 mai 2017 à 16:16, Robin Humble a écrit : >> I took a gander at the source and noticed that llite/xattr.c >> deliberately filters out 'security.capability' and returns 0/-ENODATA >> for setcap/getcap, which is indeed what strace sees. so setcap/getcap >> is never even sent to the MDS. >> >> if I remove that filter (see patch on lustre-devel) then setcap/getcap >> works -> ... >> 'b15587' is listed as the reason for the filtering. >> I don't know what that refers to. >> is it still relevant? >b15587 refers to the old Lustre Bugzilla tracking tool: >https://projectlava.xyratex.com/show_bug.cgi?id=15587 > >Reading the discussion in the ticket, supporting xattr at the time of Lustre >1.8 and 2.0 was causing issues on MDS side in some situations. So it was >decided to discard security.capability xattr on Lustre client side. I think >Andreas might have some insight, as he apparently participated in b15587. my word that's a long time ago... I don't see much in the way of jira tickets about getxattr issues on MDS in recent times, and they're much more heavily used these days, so I hope that particular problem has long since been fixed. should I open a jira ticket to track re-enabling of security.capabilities? >In any case, it is important to make clear that file capabilities, the feature >you want to use, is completely distinct from SELinux. >On the one hand, Capabilities are a Linux mechanism to refine permissions >granted to privileged processes, by dividing the privileges traditionally >associated with superuser into distinct units (known as capabilities). >On the other hand, SELinux is the Linux implementation of Mandatory Access >Control. >Both Capabilities and SELinux rely on values stored into file extended >attributes, but this is the only thing they have in common. 10-4. thanks. 'ls --color' requests the security.capability xattr so this would be heavily accessed. do you think this is handled well enough currently to not affect performance significantly? setxattr would be minimal and not performance critical, unlike with eg. selinux and creat. cheers, robin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
Hi Robin, b15587 refers to the old Lustre Bugzilla tracking tool: https://projectlava.xyratex.com/show_bug.cgi?id=15587 Reading the discussion in the ticket, supporting xattr at the time of Lustre 1.8 and 2.0 was causing issues on MDS side in some situations. So it was decided to discard security.capability xattr on Lustre client side. I think Andreas might have some insight, as he apparently participated in b15587. In any case, it is important to make clear that file capabilities, the feature you want to use, is completely distinct from SELinux. On the one hand, Capabilities are a Linux mechanism to refine permissions granted to privileged processes, by dividing the privileges traditionally associated with superuser into distinct units (known as capabilities). On the other hand, SELinux is the Linux implementation of Mandatory Access Control. Both Capabilities and SELinux rely on values stored into file extended attributes, but this is the only thing they have in common. Cheers, Sebastien. > Le 17 mai 2017 à 16:16, Robin Humble a écrit : > > I setup a couple of VMs with 2.9 clients and servers (ldiskfs) and > unfortunately setcap/getcap still are unhappy - same as with my > previous 2.9 clients with 2.8 servers (ZFS). > > hmm. > I took a gander at the source and noticed that llite/xattr.c > deliberately filters out 'security.capability' and returns 0/-ENODATA > for setcap/getcap, which is indeed what strace sees. so setcap/getcap > is never even sent to the MDS. > > if I remove that filter (see patch on lustre-devel) then setcap/getcap > works -> > > # df . > Filesystem1K-blocks Used Available Use% Mounted on > 10.122.1.5@tcp:/test8 4797904 33992 4491480 1% /mnt/test8 > # touch blah > # setcap cap_net_admin,cap_net_raw+p blah > # getcap blah > blah = cap_net_admin,cap_net_raw+p > > and I also tested that the 'ping' binary run as unprivileged user works > from lustre. > success! > > 'b15587' is listed as the reason for the filtering. > I don't know what that refers to. > is it still relevant? > > cheers, > robin > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
I setup a couple of VMs with 2.9 clients and servers (ldiskfs) and unfortunately setcap/getcap still are unhappy - same as with my previous 2.9 clients with 2.8 servers (ZFS). hmm. I took a gander at the source and noticed that llite/xattr.c deliberately filters out 'security.capability' and returns 0/-ENODATA for setcap/getcap, which is indeed what strace sees. so setcap/getcap is never even sent to the MDS. if I remove that filter (see patch on lustre-devel) then setcap/getcap works -> # df . Filesystem1K-blocks Used Available Use% Mounted on 10.122.1.5@tcp:/test8 4797904 33992 4491480 1% /mnt/test8 # touch blah # setcap cap_net_admin,cap_net_raw+p blah # getcap blah blah = cap_net_admin,cap_net_raw+p and I also tested that the 'ping' binary run as unprivileged user works from lustre. success! 'b15587' is listed as the reason for the filtering. I don't know what that refers to. is it still relevant? cheers, robin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
Hi Eli et al, >> Le 15 mai 2017 à 14:39, E.S. Rosenberg a écrit : >> Hi Robin, >> Did you ever solve this? >> We are considering trying root-on-lustre but that would be a deal-breaker. no. instead I started down the track of layering overlayfs on top of lustre. tmpfs (used by overlayfs's upper layer) has a working seclabel mount option. so I just 'copy up' the 3 or 4 exe's that have seclabels, 'setcap' them with the right label, and they work fine. I'm not sure overlayfs is going to work out though, so I'd really like seclabel in lustre. On Tue, May 16, 2017 at 08:17:48AM +, Sebastien Buisson wrote: >From Lustre 2.8, we have basic support of SELinux on Lustre client side. It >means Lustre stores the security context of files in extended attributes. In >this way Lustre supports seclabel. >In Lustre 2.9, an additional enhancement for SELinux support was landed. > >Which version are you using? 2.9 clients, 2.8 servers on ZFS. centos7 x86_64 everywhere. sestatus disabled everywhere. zfs has xattr=sa on osts, mdt, mgs Andreas wrote (a while ago): >> I try to stay away from that myself, but newer Lustre clients support SELinux >> and similar things. You probably need to strace and/or collect some kernel >> debug logs (maybe with debug=-1 set) to see where the error is being >> generated. a debug=-1 trace is here -> https://rjh.org/~rjh/lustre/dk.log.-1.txt.gz command line was -> lctl set_param debug=-1 ; usleep 5; lctl clear; usleep 5 ; /usr/sbin/setcap cap_net_admin,cap_net_raw+p /mnt/oneSIS-overlay/lowerdir/usr/bin/ping ; /usr/sbin/getcap /mnt/oneSIS-overlay/lowerdir/usr/bin/ping ; lctl dk /lfs/data0/system/log/dk.log.-1 ; lctl set_param debug='ioctl neterror warning error emerg ha config console lfsck' /mnt/oneSIS-overlay/lowerdir is the lustre root filesystem image (usually mounted read-only, but read-write for this debugging) expected output is nothing for setcap. expected output for getcap is # getcap /mnt/oneSIS-overlay/lowerdir/usr/bin/ping /mnt/oneSIS-overlay/lowerdir/usr/bin/ping = cap_net_admin,cap_net_raw+p but actual output is nothing -> # getcap /mnt/oneSIS-overlay/lowerdir/usr/bin/ping # to the copy of 'ping' on the tmpfs/overlayfs getcap/setcap works fine -> # getcap /usr/bin/ping /usr/bin/ping = cap_net_admin,cap_net_raw+p cheers, robin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
I hope to move us to 2.9 in the very near future, currently 2.8 On Tue, May 16, 2017 at 11:17 AM, Sebastien Buisson wrote: > From Lustre 2.8, we have basic support of SELinux on Lustre client side. > It means Lustre stores the security context of files in extended > attributes. In this way Lustre supports seclabel. > In Lustre 2.9, an additional enhancement for SELinux support was landed. > > Which version are you using? > > Cheers, > Sebastien. > > > Le 15 mai 2017 à 14:39, E.S. Rosenberg a > écrit : > > > > Hi Robin, > > Did you ever solve this? > > We are considering trying root-on-lustre but that would be a > deal-breaker. > > Thanks, > > Eli > > > > On Sat, Mar 4, 2017 at 9:38 AM, Dilger, Andreas < > andreas.dil...@intel.com> wrote: > > On Mar 2, 2017, at 05:55, Robin Humble > wrote: > > > > > > Hiya, > > > > > > I'm updating an image for a root-on-lustre cluster from centos6 to 7 > > > and I've hit a little snag. I can't seem to mount lustre so that it > > > understands seclabel. ie. setcap/getcap don't work. the upshot is that > > > root can use ping (and a few other tools), but users can't. > > > > > > any idea what I'm doing wrong? > > > > > > from what little I understand about it I think seclabel is a form of > > > xattr. > > > > I try to stay away from that myself, but newer Lustre clients support > SELinux > > and similar things. You probably need to strace and/or collect some > kernel > > debug logs (maybe with debug=-1 set) to see where the error is being > generated. > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Lustre Principal Architect > > Intel Corporation > > > > > > > > > > > > > > > > ___ > > lustre-discuss mailing list > > lustre-discuss@lists.lustre.org > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > > > ___ > > lustre-discuss mailing list > > lustre-discuss@lists.lustre.org > > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
From Lustre 2.8, we have basic support of SELinux on Lustre client side. It means Lustre stores the security context of files in extended attributes. In this way Lustre supports seclabel. In Lustre 2.9, an additional enhancement for SELinux support was landed. Which version are you using? Cheers, Sebastien. > Le 15 mai 2017 à 14:39, E.S. Rosenberg a écrit : > > Hi Robin, > Did you ever solve this? > We are considering trying root-on-lustre but that would be a deal-breaker. > Thanks, > Eli > > On Sat, Mar 4, 2017 at 9:38 AM, Dilger, Andreas > wrote: > On Mar 2, 2017, at 05:55, Robin Humble wrote: > > > > Hiya, > > > > I'm updating an image for a root-on-lustre cluster from centos6 to 7 > > and I've hit a little snag. I can't seem to mount lustre so that it > > understands seclabel. ie. setcap/getcap don't work. the upshot is that > > root can use ping (and a few other tools), but users can't. > > > > any idea what I'm doing wrong? > > > > from what little I understand about it I think seclabel is a form of > > xattr. > > I try to stay away from that myself, but newer Lustre clients support SELinux > and similar things. You probably need to strace and/or collect some kernel > debug logs (maybe with debug=-1 set) to see where the error is being > generated. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Intel Corporation > > > > > > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
Hi Robin, Did you ever solve this? We are considering trying root-on-lustre but that would be a deal-breaker. Thanks, Eli On Sat, Mar 4, 2017 at 9:38 AM, Dilger, Andreas wrote: > On Mar 2, 2017, at 05:55, Robin Humble > wrote: > > > > Hiya, > > > > I'm updating an image for a root-on-lustre cluster from centos6 to 7 > > and I've hit a little snag. I can't seem to mount lustre so that it > > understands seclabel. ie. setcap/getcap don't work. the upshot is that > > root can use ping (and a few other tools), but users can't. > > > > any idea what I'm doing wrong? > > > > from what little I understand about it I think seclabel is a form of > > xattr. > > I try to stay away from that myself, but newer Lustre clients support > SELinux > and similar things. You probably need to strace and/or collect some kernel > debug logs (maybe with debug=-1 set) to see where the error is being > generated. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Intel Corporation > > > > > > > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] seclabel
On Mar 2, 2017, at 05:55, Robin Humble wrote: > > Hiya, > > I'm updating an image for a root-on-lustre cluster from centos6 to 7 > and I've hit a little snag. I can't seem to mount lustre so that it > understands seclabel. ie. setcap/getcap don't work. the upshot is that > root can use ping (and a few other tools), but users can't. > > any idea what I'm doing wrong? > > from what little I understand about it I think seclabel is a form of > xattr. I try to stay away from that myself, but newer Lustre clients support SELinux and similar things. You probably need to strace and/or collect some kernel debug logs (maybe with debug=-1 set) to see where the error is being generated. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Intel Corporation ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] seclabel
Hiya, I'm updating an image for a root-on-lustre cluster from centos6 to 7 and I've hit a little snag. I can't seem to mount lustre so that it understands seclabel. ie. setcap/getcap don't work. the upshot is that root can use ping (and a few other tools), but users can't. any idea what I'm doing wrong? from what little I understand about it I think seclabel is a form of xattr. cheers, robin ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
