Re: [lustre-discuss] Disable identity_upcall and ACL
Thanks for the clarifications. Aurélien Le 14/01/2019 01:36, « Andreas Dilger » a écrit : On Jan 10, 2019, at 04:52, Degremont, Aurelien wrote: > > > Le 09/01/2019 21:39, « Andreas Dilger » a écrit : > >> If admins completely trust the client nodes (e.g. they are on a secure >> network) or they completely _distrust_ them (e.g. subdirectory mounts >> with nodemaps/idmaps and Kerberos/SSK to identify them), or the data >> just isn't that secret, then allowing the client to handle the group >> lookups instead of the MDS is mostly OK. >> >> The main issue is for new, uncached lookups from the client. Since the >> RPC only includes the UID, GID, and maybe one supplementary GID, it is >> possible that the MDS (without the identity_upcall) may deny the lookup >> because the request does not contain any IDs that would allow file access. > > According to struct mdt_body there is room for only one suppgid. > But the value is not always packed in mdc, depending on the call. > So that means that hopefully between 0 and 1 supplementary group will be passed to MDT, if I read the code correctly. > >> I guess the other question is why you are interested to get rid of it, >> or what issue you are seeing with it enabled? > > If identity_upcall is enabled, you need an up to date group database available on MDS. This is not always the case. I'm trusting the clients in this case. I would be interesting in having the MDT doing no credential checks and letting the clients (VFS) do all the validations. MDT is already trusting client when it is sending uid and gid. > > So, coming back to my original question, the ACL warning message in MDT is not really limited to ACL but more generally to any supplementary groups checks. Some accesses could be denied if they rely on supplementary groups (likely not the first one) and could be wrongly granted or denied if based on ACL. Correct? Correct. > Permission checks for primary uid/gid is always correct, whatever identity_upcall value? Yes, definitely. If the client "knows" the right suppgid then it will send it to the MDS as well, but otherwise it just picks the first one. Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Disable identity_upcall and ACL
On Jan 10, 2019, at 04:52, Degremont, Aurelien wrote: > > > Le 09/01/2019 21:39, « Andreas Dilger » a écrit : > >> If admins completely trust the client nodes (e.g. they are on a secure >> network) or they completely _distrust_ them (e.g. subdirectory mounts >> with nodemaps/idmaps and Kerberos/SSK to identify them), or the data >> just isn't that secret, then allowing the client to handle the group >> lookups instead of the MDS is mostly OK. >> >> The main issue is for new, uncached lookups from the client. Since the >> RPC only includes the UID, GID, and maybe one supplementary GID, it is >> possible that the MDS (without the identity_upcall) may deny the lookup >> because the request does not contain any IDs that would allow file access. > > According to struct mdt_body there is room for only one suppgid. > But the value is not always packed in mdc, depending on the call. > So that means that hopefully between 0 and 1 supplementary group will be > passed to MDT, if I read the code correctly. > >> I guess the other question is why you are interested to get rid of it, >> or what issue you are seeing with it enabled? > > If identity_upcall is enabled, you need an up to date group database > available on MDS. This is not always the case. I'm trusting the clients in > this case. I would be interesting in having the MDT doing no credential > checks and letting the clients (VFS) do all the validations. MDT is already > trusting client when it is sending uid and gid. > > So, coming back to my original question, the ACL warning message in MDT is > not really limited to ACL but more generally to any supplementary groups > checks. Some accesses could be denied if they rely on supplementary groups > (likely not the first one) and could be wrongly granted or denied if based on > ACL. Correct? Correct. > Permission checks for primary uid/gid is always correct, whatever > identity_upcall value? Yes, definitely. If the client "knows" the right suppgid then it will send it to the MDS as well, but otherwise it just picks the first one. Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Disable identity_upcall and ACL
On Jan 9, 2019, at 07:52, Daniel Kobras wrote: > > Hi Aurélien! > > Am 09.01.19 um 14:30 schrieb Degremont, Aurelien: >> The secondary group thing was ok to me. I got this idea even if there is >> some weird results during my tests. Looks like you can overwrite MDT checks >> if user and group is properly defined on client node. Cache effects? > > In a talk I gave a decade ago, I described a problem with authorization > due to inconsistencies between client and MDT, depending on whether > metadata was in the client cache or not (see p. 23 of > http://wiki.lustre.org/images/b/ba/Tuesday_lustre_automotive.pdf -- you > really managed to challenge my memory ;-) I faintly remember Andreas > commenting that the MDT was always supposed to be authoritative, even > for cached content, and the experienced behaviour was a bug. Indeed, > other than those prehistoric versions, I'm not aware of any > inconsistencies in authorization due to cache effects. If admins completely trust the client nodes (e.g. they are on a secure network) or they completely _distrust_ them (e.g. subdirectory mounts with nodemaps/idmaps and Kerberos/SSK to identify them), or the data just isn't that secret, then allowing the client to handle the group lookups instead of the MDS is mostly OK. The main issue is for new, uncached lookups from the client. Since the RPC only includes the UID, GID, and maybe one supplementary GID, it is possible that the MDS (without the identity_upcall) may deny the lookup because the request does not contain any IDs that would allow file access. If the file is cached on the client, then the MDS doesn't get involved at all, and the VFS has full access to the IDs and can make the decision locally. This may lead to inconsistent behaviour as the file moves in and out of the client cache (e.g. a shared file that is accessed by multiple users, some of which have direct UID/GID access, others which only have access via supp GID). This access pattern is fairly uncommon, however. >> ACL is really the thing I was interested in. Who is validating the ACLs? >> MDT, client or both? Do you think ACL could be properly applied if >> user/groups are only defined on client side and identity_upcall is disabled >> on MDT side? > > Posix ACLs use numeric uids and gids, just like ordinary permission > bits. Evaluation is supposed to happens on the MDT for both. If you can > do without secondary groups, there's no need for user and group > databases on the MDT--numeric id will work fine. (Unless you use > Kerberos, which will typically require user names for proper id mapping. I believe the MDS is also verifying the file access via ACL, so if the only reason a user can access the file is because of a supp GID that is missing (due to no identity_upcall or inconsistent /etc/groups) then access would be denied. The other (maybe lesser?) risk is if access should be denied because of a supplementary group, the MDS will actually allow access to the inode to the client. I'm not sure if the client will revalidate the ACLs locally and deny this on first open or not. In summary, no identity_upcall would _mostly_ work, especially for simple usage modes. I guess the other question is why you are interested to get rid of it, or what issue you are seeing with it enabled? Cheers, Andreas --- Andreas Dilger Principal Lustre Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Disable identity_upcall and ACL
Hi Aurélien! Am 09.01.19 um 14:30 schrieb Degremont, Aurelien: > The secondary group thing was ok to me. I got this idea even if there is some > weird results during my tests. Looks like you can overwrite MDT checks if > user and group is properly defined on client node. Cache effects? In a talk I gave a decade ago, I described a problem with authorization due to inconsistencies between client and MDT, depending on whether metadata was in the client cache or not (see p. 23 of http://wiki.lustre.org/images/b/ba/Tuesday_lustre_automotive.pdf -- you really managed to challenge my memory ;-) I faintly remember Andreas commenting that the MDT was always supposed to be authoritative, even for cached content, and the experienced behaviour was a bug. Indeed, other than those prehistoric versions, I'm not aware of any inconsistencies in authorization due to cache effects. > ACL is really the thing I was interested in. Who is validating the ACLs? MDT, > client or both? Do you think ACL could be properly applied if user/groups are > only defined on client side and identity_upcall is disabled on MDT side? Posix ACLs use numeric uids and gids, just like ordinary permission bits. Evaluation is supposed to happens on the MDT for both. If you can do without secondary groups, there's no need for user and group databases on the MDT--numeric id will work fine. (Unless you use Kerberos, which will typically require user names for proper id mapping.) Kind regards, Daniel -- Daniel Kobras Principal Architect Puzzle ITC Deutschland +49 7071 14316 0 www.puzzle-itc.de -- Puzzle ITC Deutschland GmbH Sitz der Gesellschaft: Jurastr. 27/1, 72072 Tübingen Eingetragen am Amtsgericht Stuttgart HRB 765802 Geschäftsführer: Lukas Kallies, Daniel Kobras, Mark Pröhl ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Disable identity_upcall and ACL
Hi Daniel! The secondary group thing was ok to me. I got this idea even if there is some weird results during my tests. Looks like you can overwrite MDT checks if user and group is properly defined on client node. Cache effects? ACL is really the thing I was interested in. Who is validating the ACLs? MDT, client or both? Do you think ACL could be properly applied if user/groups are only defined on client side and identity_upcall is disabled on MDT side? Thanks Aurélien Le 09/01/2019 12:22, « lustre-discuss au nom de Daniel Kobras » a écrit : Hi Aurélien! Am 09.01.19 um 11:48 schrieb Degremont, Aurelien: > When disabling identity_upcall on a MDT, you get this message in system > logs: > > lustre-MDT: disable "identity_upcall" with ACL enabled maybe cause > unexpected "EACCESS" > > I’m trying to understand what could be a scenario that shows this problem? > What is the implication, or rather, how identity_upcall works? Without an identity_upcall, all Lustre users effectively lose their secondary group memberships. These are not passed in the RPCs, but evaluated on the MDS instead. The default l_getidentity receives a numeric uid, queries NSS to obtain the corresponding account's list of gids, and passes the list back to the kernel. As a test scenario, just try to access a file or directory from an account that only has access permissions via one of its secondardy groups. (The log message is a bit misleading--you don't actually need to use ACLs, ordinary group permissions are sufficient.) Kind regards, Daniel -- Daniel Kobras Principal Architect Puzzle ITC Deutschland +49 7071 14316 0 www.puzzle-itc.de -- Puzzle ITC Deutschland GmbH Sitz der Gesellschaft: Jurastr. 27/1, 72072 Tübingen Eingetragen am Amtsgericht Stuttgart HRB 765802 Geschäftsführer: Lukas Kallies, Daniel Kobras, Mark Pröhl ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Disable identity_upcall and ACL
Hi Aurélien! Am 09.01.19 um 11:48 schrieb Degremont, Aurelien: > When disabling identity_upcall on a MDT, you get this message in system > logs: > > lustre-MDT: disable "identity_upcall" with ACL enabled maybe cause > unexpected "EACCESS" > > I’m trying to understand what could be a scenario that shows this problem? > What is the implication, or rather, how identity_upcall works? Without an identity_upcall, all Lustre users effectively lose their secondary group memberships. These are not passed in the RPCs, but evaluated on the MDS instead. The default l_getidentity receives a numeric uid, queries NSS to obtain the corresponding account's list of gids, and passes the list back to the kernel. As a test scenario, just try to access a file or directory from an account that only has access permissions via one of its secondardy groups. (The log message is a bit misleading--you don't actually need to use ACLs, ordinary group permissions are sufficient.) Kind regards, Daniel -- Daniel Kobras Principal Architect Puzzle ITC Deutschland +49 7071 14316 0 www.puzzle-itc.de -- Puzzle ITC Deutschland GmbH Sitz der Gesellschaft: Jurastr. 27/1, 72072 Tübingen Eingetragen am Amtsgericht Stuttgart HRB 765802 Geschäftsführer: Lukas Kallies, Daniel Kobras, Mark Pröhl ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Disable identity_upcall and ACL
Hello all, When disabling identity_upcall on a MDT, you get this message in system logs: lustre-MDT: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" I’m trying to understand what could be a scenario that shows this problem? What is the implication, or rather, how identity_upcall works? Thanks for your help Aurélien ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
