[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
when *not* using -partial_chain ** Changed in: sssd (Ubuntu) Status: New => In Progress ** Changed in: sssd (Ubuntu) Assignee: (unassigned) => Marco Trevisan (Treviño) (3v1n0) ** Changed in: sssd (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
Yeah, sure... As per man page: -partial_chain Allow verification to succeed even if a complete chain cannot be built to a self-signed trust-anchor, provided it is possible to construct a chain to a trusted certificate that might not be self-signed. And you can test it quite easily with the attached generated certs using: openssl verify [-partial_chain] \ -CAfile test_CA/intermediate_CA/SSSD_test_intermediate_CA.pem \ test_CA/intermediate_CA/SSSD_test_intermediate_CA_cert_x509_0001.pem While when using -partial_chain will only match when using test_CA/intermediate_CA/SSSD_test_intermediate_CA_full_db.pem as CAfile ** Attachment added: "Test CA certificates chain" https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+attachment/5481720/+files/test_CA.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
So, I've done some work on SSSD upstream to make this to happen: https://github.com/SSSD/sssd/pull/5558 With that we'll just be able to set on upgraders the option `certification_verification = partial_chain`, and this will just make the SSSD's PEM ring to work as the NSS db used to work: and so verify a certificate if its only its issuer is in the SSSD's CA certificates DB. This comes with unit tests covering the case with generated certificates, not sure if I can personally test this with real hardware (for SRU purposes) though... We may still need to simulate it. At the end, it's just as doing: openssl verify -partial_chain -CAfile intermediate_CA.pem intermediate_CA_issued_cert.pem Karl, will this be enough for you? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
re conversion / upgrades => we should really find the full chain if we can to inject it into openssl. I'm not sure if there are any ways to force openssl to be happy with trusted issuer without a full chain. I would have thought there is a way to make openssl do that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
Re: certs. Ideally we should be shipping a bundle of certificates, which are well known roots of trust for smarcards. Aka the DOD, National ID cards/passports, etc. In a new path locations. Because the smartcard roots of trusts are not the same as for https:// connections. But that's not immediately fixable. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
I think that is a long standing openssl bug that it demands full chains, and more so it trips up not only when the chain is incomplete, but also where there are extra chains, which are redundant; and if any of them have untrusted certs, or certs of small sizes / old hashes (aka legacy chains) it also refuses to establish connections. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
> While this would technically work, it would be really bad news. This would allow anyone with any user cert issued by a CA in the system wide cert store (by any CA in the world) to be trusted and pass authorization checks by p11_child. (Albeit, some directory attributes would have to line up, depending on your match rules) Well, that's just partially true since as you said: - Without a match rule (that has to be configured) there's no access anyways However as I was saying, maybe the other way around can be safer? I mean, SSSD will still use /etc/sssd/pki/sssd_auth_ca_db.pem for the trusted certs, but we will populate it adding also the ones trusted by the system. Maybe providing a way to filter them out. I'm talking only of upgrades from NSS installs though, for new installations people will have to manually add their trusted CAs to /etc/sssd/pki/sssd_auth_ca_db.pem. The point here is, I suppose, that if the system trusts a CA, then we can't just not trust it for some specific operation, this can be still filtered out (if needed) by using proper sssd config parameters. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
On Thu, Mar 18, 2021 at 02:14:23AM -, Karl Grindley wrote: > I'll also comment, (and perhaps a bit of scope creap, but...) we've > found a number of unfixed issues with sssd, specifically with PKINIT and > LDAP optimizations. We're working with the upstream maintainers to help > address these. We would like to see these brought into 20.04 LTS, as > all directory users can benefit here. Are you or Marco the best to help > us bring these into a general release down the road? Your assistance in getting fixes landed would be most welcome! We do have to take great care to minimise risk of regression to existing users of course, as you've very well demonstrated in this bug :) Our process for getting fixes in is documented at https://wiki.ubuntu.com/StableReleaseUpdates. Assuming that based on that policy and the specifics of the fixes, cherry-picks are most appropriate, the main steps would be to have a bug for each issue, get the Ubuntu development release fixed, and then prepare an update for 20.04. https://wiki.ubuntu.com/StableReleaseUpdates#Procedure has the details. You can do most of this yourself, and we'd be happy to help with sponsorship and any rough edges in the process. You can reach us on the ubuntu-devel@ mailing list (https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel) or #ubuntu-devel on Freenode on IRC. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
Karl, I'm the responsible for what it concerns this specific change and so feel free to use me a reference for this (maybe ping me on IRC so we can have some better interaction regarding the best solution we can take). However, the SSSD maintainer in ubuntu is Sergio Durigan Junior, so he's the one can help you better with the other relevant issues. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
I agree and concur. Just need some checks here, as this is a pretty big change in behavior for a mid-life LTS release. That said, the new configuration is in line with RHEL8, and will help reduce the configuration scope for a working solution. I'll also comment, (and perhaps a bit of scope creap, but...) we've found a number of unfixed issues with sssd, specifically with PKINIT and LDAP optimizations. We're working with the upstream maintainers to help address these. We would like to see these brought into 20.04 LTS, as all directory users can benefit here. Are you or Marco the best to help us bring these into a general release down the road? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
Karl, thank you for the detailed writeup? This looks very useful. I'll leave Marco to respond as he drove the change in question, but a couple of less technical comments: On Thu, Mar 18, 2021 at 01:16:28AM -, Karl Grindley wrote: > I don't discourage this change, in fact, will help push along the MFA > adoption. [I rearranged ordering of your sentences a bit for context] > I'd also suggest that MFA support in general should be considered a core > requirement for future versions of the LTS, and well tested, supported > and documented. Adoption will only grow with time, and become more > critical. This will help reduce the variations of implementations, and > help drive folks to a known and supported configuration. Thank you for the support! I believe this was exactly Marco's intention. > However, I think perhaps some preflight checks in the package could > solve someone bricking their machine. (or a large quantity of machines). It sounds to me that there's some scope for improvement then, if that can be figured out between you and Marco, and that a revert isn't required. So to ensure there's no misunderstanding about expectations about this bug, the way I see it now is that we're going to keep what we have. Marco will (presumably) consider your suggestion and that might lead to a further upload to add some further sanity checks depending on how the details pan out. We can use this bug to track and communicate about that. Does that work for everyone? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
> Karl, thank you for the detailed writeup? That was intended to say: Karl, thank you for the detailed writeup! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
So, if I didn't get it wrong, if we'd just use /etc/ssl/certs/ca- certificates.crt as the SSSD pam certificate in such case would work? I mean having this in /etc/sssd/sssd.conf [pam] pam_cert_db_path = /etc/ssl/certs/ca-certificates.crt And then what was into /etc/sssd/pki/sssd_auth_ca_db.pem to be added to .crt's under /usr/local/share/ca-certificates/sssd_auth_ca_db/ and eventually calling update-ca-certificates maybe? We could even do the other way around probably, by adding an hook to /etc/ca-certificates/update.d/ so that we ensure that /etc/ssl/certs/ca- certificates.crt is always in sync with the system ring? As Robie said, we could revert this change but this would not be ideal for various reasons IMHO: 1. As you said this is going to be used more and more, and so we'll have to end up to keep supporting a growing number of systems with an outdated method that is going to be dropped in future (i.e. better to do it now that its usage is limited than having to do it in future when the audience is bigger) 2. We would like to have a single documented method to have smartcard auth in ubuntu using SSSD that can be validated from 20.04 onward and that keep working in future LTSs (and for sure next LTS will have to drop NSS anyways, so it's just about delaying a problem making it bigger). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
To speak to real world assessment here - there's a big push across many (US) gov't orgs and industry to deploy MFA. These requirements are not new, but many have not been enforced due to lack of compliance checks/certifications. This is changing with new efforts in the US Gov't industry circles with regard to CMMC. This is an assessment/certification that industry must meet and maintain for contactual compliance, starting to roll out in the next year or so. Likewise there's been a lot of focus lately on unclassified compliance with NIST policy. We have a number of customers, working toward or maintaining an MFA solution. All are struggling. Many have lagged with pam_pkcs11 providing/satisfying most compliance requirements. But with RHEL8 and Ubuntu 20.04 adoption underway (with RHEL6 and 14.04 end of life) many are stuck working to cobble together an implementation. Of course with the uptick in remote work, MFA has also resurged, also pushing along adoption of sssd MFA. We noticed with the latest round of patching something was a-miss. and today tracked it down to this change. We're working with our customers to come up with a workaround. I think there's a larger number of folks impacted here, but unfortunately, the number of possible ways to do MFA is very large, and because no one maintainer has completely documented/supported MFA well, sysadmins typically develop their MFA craft using what they can. I don't discourage this change, in fact, will help push along the MFA adoption. However, I think perhaps some preflight checks in the package could solve someone bricking their machine. (or a large quantity of machines). I'd also suggest that MFA support in general should be considered a core requirement for future versions of the LTS, and well tested, supported and documented. Adoption will only grow with time, and become more critical. This will help reduce the variations of implementations, and help drive folks to a known and supported configuration. Reproduction of the issue: In our circles, we see a fully Microsoft AD integrated Smartcard (with kerberos and PKINIT) implementation. This also bleeds over into pam_sss configuration issues with U20.04, (for which I should file another ticket) Based on my diagnosis today, I think this is isolated to p11_child, and those with a nssdb with only issuing CA certs populated in the database. I don't think this issue matters for which directory is being used and if PKINIT is functioning, since all the MFA magic happens within p11_child. I'm going to assume that you folks have some way test AD with MFA, and will try to summarize. To reproduce, you'll need (at least) a 2 tier CA PKI chain. Root -> Issuing CA -> End user cert (with old sssd version) configure for smart card auth * do as you always do to join/setup sssd to a directory service * verify user ID lookups, and login works as expected with password * add any mapping/filter rules to the /etc/sssd/sssd.conf for p11_child * upadte /usr/share/pam-configs/sss to Priority 800, rebuild pam stack, dpkg-divert /usr/share/pam-configs/sss * add the root and issuing certs to /usr/local/share/ca-certificates, rebuild system trust store * generate a new, empty nssdb /usr/bin/certutil -N -d sql:/etc/pki/nssdb --empty-password * when adding the certs to nssdb, only add the Issuing CA WITH CT,C,C flags certutil -A -d /etc/pki/nssdb -n issuingCA.crt -t "CT,C,C" -i /usr/local/share/ca-certificates/issuingCA.pem * enable openSC modutil -force -dbdir /etc/pki/nssdb -add "OpenSC" -libfile opensc-pkcs11.so * test PKI auth works * login or: /usr/libexec/sssd/p11_child --nssdb=/etc/pki/nssdb --pre -d 10 --debug-fd=1 --verify no_ocsp * perform upgrade to latest sssd * verify the /etc/sssd/pki/sssd_auth_ca_db.pem is populated only with the issuingCA * test p11_child to see if it breaks /usr/libexec/sssd/p11_child --debug-microseconds=0 --debug-timestamps=1 --debug-fd=23 --debug-level=0xf7f0 --pre --verify no_ocsp --nssdb /etc/sssd/pki/sssd_auth_ca_db.pem fix it: * add the /usr/local/share/ca-certificates/rootCA.pem >> /etc/sssd/pki/sssd_auth_ca_db.pem * run p11_child again, observe that it works * try to login Brick your system procedure: After above test procedure works: * configure for MFA on old sssd * populate the below to /usr/share/pam-configs/sss-smartcardonly * pam-auth-update --package --enable sss-smartcardonly --remove sss --force * verify only smart card is allowed to login * apt upgrade * reboot, login no longer allowed Note that SSHing into the system may be allowed, depending on ssh configuration and if sss_ssh_authroizedkeys is enabled. Name: SSS authentication - Requires Smartcard Default: yes Conflicts: sss Priority: 800 Auth-Type: Primary Auth: [success=end default=ignore]pam_sss.so use_first_pass require_cert_auth Auth-Initial: [success=end default=ignore]pam_sss.so forward_pass require_cert_auth Account-Type: Additional Account: sufficient
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
Karl, thank you for your report. We now need to decide an appropriate course of action here. An update to sssd to revert the change is a possibility, but there's also risk there that we will break users twice. Do you have a deployment affected by this? How many other users might be affected by this? In bug 1905790, our assessment was that it was unlikely that anyone was using NSS+sssd -based smartcards on 20.04. Was this assessment wrong? What's the real world impact here? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
Could you maybe provide an example setup we can reuse to simulate such scenario? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1919563] Re: updated sssd with smart cards now brick systems without full cert chain
** Information type changed from Private Security to Public ** Tags added: regression-update -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1919563 Title: updated sssd with smart cards now brick systems without full cert chain To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1919563/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
