Re: Server certs expired higher up the chain, imaps and https
On Sat, Oct 02, 2021 at 05:25:17PM +0200, Marcus MERIGHI wrote: > > I've nominated you for the "most helpful person around" award. > There just have to be clones! I don't see how he has enough time. So +1 or +2 or +3.. Chris
Re: Server certs expired higher up the chain, imaps and https
Hello! benoit-li...@fb12.de (Sebastian Benoit), 2021.09.30 (Thu) 21:42 (CEST): > Chris Bennett(cpb_m...@bennettconstruction.us) on 2021.09.30 10:02:17 -0700: > > I'm getting that the certs are expired, but https works fine in Firefox, > > including when looking at the full chain. > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > > mail.strengthcouragewisdom.rocks:https > > This is an issue with an expired root/intermediate certificate (DST Root X3) > in use by Let's Encrypt. > > Stuart Henderson (sthen@) summarized it like this: > > LibreSSL in OpenBSD 6.9/earlier is having problems with the expiry of a > CA certificate used to cross-sign Let's Encrypt certs. > > LE decided not to switch to using their own root fully, rather they > are continuing to use the expired cross-signer to increase compatibility > with old Android devices, which is tickling this problem. > https://letsencrypt.org/2020/12/21/extending-android-compatibility.html > > An errata has just been published, you can install it using syspatch. I've syspatch(8)-ed a machine that now delivers the following error: $ ftp -VMo /dev/null \ "https://shop.theater-phoenix.at/Events.aspx?msg=0=1; TLS handshake failure: certificate verification failed: unable to get local issuer certificate $ openssl s_client -servername shop.theater-phoenix.at -connect \ shop.theater-phoenix.at:https Verify return code: 21 (unable to verify the first certificate) The server "shop.theater-phoenix.at" runs under Windows and uses letsencrypt certificates. Does this issue have the same root cause or is this something different? Marcus
Re: Server certs expired higher up the chain, imaps and https
Hello! stu.li...@spacehopper.org (Stuart Henderson), 2021.10.02 (Sat) 16:13 (CEST): > On 2021-10-02, Marcus MERIGHI wrote: > > benoit-li...@fb12.de (Sebastian Benoit), 2021.09.30 (Thu) 21:42 (CEST): > >> Chris Bennett(cpb_m...@bennettconstruction.us) on 2021.09.30 10:02:17 > >> -0700: > >> > I'm getting that the certs are expired, but https works fine in Firefox, > >> > including when looking at the full chain. > >> > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > >> > mail.strengthcouragewisdom.rocks:https > >> > >> This is an issue with an expired root/intermediate certificate (DST Root > >> X3) > >> in use by Let's Encrypt. > > I've syspatch(8)-ed a machine that now delivers the following error: > > $ openssl s_client -servername shop.theater-phoenix.at -connect \ > > shop.theater-phoenix.at:https > > Verify return code: 21 (unable to verify the first certificate) > > Does this issue have the same root cause or is this something different? > > Different. They are using the wrong *intermediate* cert (which expired on > *Wednesday*): > > Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 > Validity > Not Before: Oct 7 19:21:40 2020 GMT > Not After : Sep 29 19:21:40 2021 GMT > Subject: C=US, O=Let's Encrypt, CN=R3 > > Specifically, at present they should be using this instead: > https://letsencrypt.org/certs/lets-encrypt-r3.pem > However it may change in future so they should use the one fetched by > their ACME client (generally this > means using the "fullchain" file) rather than fetching a separate one. I've nominated you for the "most helpful person around" award. Thanks! Marcus
Re: Server certs expired higher up the chain, imaps and https
On 2021-10-02, Marcus MERIGHI wrote: > Hello! > > benoit-li...@fb12.de (Sebastian Benoit), 2021.09.30 (Thu) 21:42 (CEST): >> Chris Bennett(cpb_m...@bennettconstruction.us) on 2021.09.30 10:02:17 -0700: >> > I'm getting that the certs are expired, but https works fine in Firefox, >> > including when looking at the full chain. >> > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect >> > mail.strengthcouragewisdom.rocks:https >> >> This is an issue with an expired root/intermediate certificate (DST Root X3) >> in use by Let's Encrypt. >> >> Stuart Henderson (sthen@) summarized it like this: >> >> LibreSSL in OpenBSD 6.9/earlier is having problems with the expiry of a >> CA certificate used to cross-sign Let's Encrypt certs. >> >> LE decided not to switch to using their own root fully, rather they >> are continuing to use the expired cross-signer to increase compatibility >> with old Android devices, which is tickling this problem. >> https://letsencrypt.org/2020/12/21/extending-android-compatibility.html >> >> An errata has just been published, you can install it using syspatch. > > I've syspatch(8)-ed a machine that now delivers the following error: > > $ ftp -VMo /dev/null \ > "https://shop.theater-phoenix.at/Events.aspx?msg=0=1; > TLS handshake failure: certificate verification failed: unable to get > local issuer certificate > > $ openssl s_client -servername shop.theater-phoenix.at -connect \ > shop.theater-phoenix.at:https > Verify return code: 21 (unable to verify the first certificate) > > The server "shop.theater-phoenix.at" runs under Windows and uses > letsencrypt certificates. > > Does this issue have the same root cause or is this something different? Different. They are using the wrong *intermediate* cert (which expired on *Wednesday*): Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 Validity Not Before: Oct 7 19:21:40 2020 GMT Not After : Sep 29 19:21:40 2021 GMT Subject: C=US, O=Let's Encrypt, CN=R3 Specifically, at present they should be using this instead: https://letsencrypt.org/certs/lets-encrypt-r3.pem However it may change in future so they should use the one fetched by their ACME client (generally this means using the "fullchain" file) rather than fetching a separate one. -- Please keep replies on the mailing list.
Re: Server certs expired higher up the chain, imaps and https
On 2021-10-01, Andrew Daugherity wrote: > On Thu, Sep 30, 2021 at 4:00 PM Sebastian Benoit wrote: >> This is an issue with an expired root/intermediate certificate (DST Root X3) >> in use by Let's Encrypt. >> >> [...] >> An errata has just been published, you can install it using syspatch. > > Thanks for the quick patch! I can verify this fixes fetching with > ftp(1) from https servers which use Let's Encrypt certificates. It > looks like this is "workaround 2" as described in this OpenSSL blog > [1]? It's like the last paragraph of workaround 2, but without the build flag to enable it. OpenSSL 1.1 doesn't have the problem due to the same change on their side. LibreSSL in -current/7.0 uses the new verifier which is better at evaluating the graph of certificates and doesn't have this problem. > I'm surprised this was even needed, as I thought this problem was > "fixed" last year after the AddTrust External CA Root expiration. It > seems to be a similar case of "while waiting for widespread acceptance > of a new root, the intermediate is cross-signed". In that case the > cert chain configured on your web server was: > - host cert, signed by: > - InCommon RSA Server CA (or several other intermediates), signed by: > - USERTrust RSA Certification Authority, signed by: > - AddTrust External CA Root (which expired 2020-05-30; recommended not > to send the top-level root since it's in the client store and thus > redundant). > > A few years before that expiration, they got a new USERTrust root into > browser/OS certificate stores, and intermediates like InCommon were > also signed by this new root. Browsers would ignore that last > USERTrust intermediate cert since it wasn't needed, but OpenSSL before > 1.1 would still consider it expired because of that intermediate. > > This seems to be a similar scenario, with: > - host cert, signed by: > - Let's Encrypt R3, signed by: > - ISRG Root X1, signed by: > - DST Root CA X3 (which just expired) IIUC the usertrust workaround handles cases where the server-supplied certificate has expired, in this case the expired cert is in the trust store but is not supplied by the server. > Likewise, there's a new ISRG Root X1. The "alternate" or "short" > chain on your server would consist of host + R3, but certbot etc. are > defaulting to the "long" chain which adds the X1 intermediate. Unlike > the USERTrust/AddTrust scenario, where the intermediate USERTrust cert > expired the same time as the AddTrust root, the intermediate X1 cert > doesn't expire until 2024-09-30. That seems to go against accepted > standards of not issuing certificates expiring after the issuer > expires, but maybe they got special dispensation. (And apparently > Android doesn't care if the root expired, if everything else is valid? Yes, and I see the same with at least one of the common JDK crypto libraries (not entirely surprising given the relationship between Android and Java). > Also, apparently there was a different, older R3 intermediate which > _also_ expired a couple days ago.) Correct, I saw some issues with this too which I think might have been down to client caching. (Also some server operators might be using manually downloaded intermediates that they didn't update. "Normal" GUI web browsers wouldn't notice this as they mostly fetch missing intermediates automatically). > The immediate fix last year was to stop sending the unnecessary > expired intermediate cert (i.e. only send host cert and InCommon RSA, > not the USERTrust intermediate), but I thought a fix went into > LibreSSL then to not consider a host "expired" if it was possible to > generate a valid chain of trust, regardless of "extra" certs sent by > the server? > > Indeed, Let's Encrypt's own documentation [2] thinks that only > LibreSSL < 3.2.0 is affected, but that is not the case. LibreSSL > 3.3.2 in OpenBSD 6.9 (before the new syspatch) still considered > servers expired, as does the somewhat older version bundled in macOS. Difficult situation and whether LE defaulted to the openssl 1.0.x-compatible chain or whether they defaulted to the one taking advantage of the Android/Java security bug a large number of people would have problems. But I think they did pick the one which is better for the ecosystem in general and where the people who are affected have some hope of fixing it. > -Andrew > > [1] https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/ > [2] > https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 > > -- Please keep replies on the mailing list.
Re: Server certs expired higher up the chain, imaps and https
On Thu, Sep 30, 2021 at 4:00 PM Sebastian Benoit wrote: > This is an issue with an expired root/intermediate certificate (DST Root X3) > in use by Let's Encrypt. > > [...] > An errata has just been published, you can install it using syspatch. Thanks for the quick patch! I can verify this fixes fetching with ftp(1) from https servers which use Let's Encrypt certificates. It looks like this is "workaround 2" as described in this OpenSSL blog [1]? I'm surprised this was even needed, as I thought this problem was "fixed" last year after the AddTrust External CA Root expiration. It seems to be a similar case of "while waiting for widespread acceptance of a new root, the intermediate is cross-signed". In that case the cert chain configured on your web server was: - host cert, signed by: - InCommon RSA Server CA (or several other intermediates), signed by: - USERTrust RSA Certification Authority, signed by: - AddTrust External CA Root (which expired 2020-05-30; recommended not to send the top-level root since it's in the client store and thus redundant). A few years before that expiration, they got a new USERTrust root into browser/OS certificate stores, and intermediates like InCommon were also signed by this new root. Browsers would ignore that last USERTrust intermediate cert since it wasn't needed, but OpenSSL before 1.1 would still consider it expired because of that intermediate. This seems to be a similar scenario, with: - host cert, signed by: - Let's Encrypt R3, signed by: - ISRG Root X1, signed by: - DST Root CA X3 (which just expired) Likewise, there's a new ISRG Root X1. The "alternate" or "short" chain on your server would consist of host + R3, but certbot etc. are defaulting to the "long" chain which adds the X1 intermediate. Unlike the USERTrust/AddTrust scenario, where the intermediate USERTrust cert expired the same time as the AddTrust root, the intermediate X1 cert doesn't expire until 2024-09-30. That seems to go against accepted standards of not issuing certificates expiring after the issuer expires, but maybe they got special dispensation. (And apparently Android doesn't care if the root expired, if everything else is valid? Also, apparently there was a different, older R3 intermediate which _also_ expired a couple days ago.) The immediate fix last year was to stop sending the unnecessary expired intermediate cert (i.e. only send host cert and InCommon RSA, not the USERTrust intermediate), but I thought a fix went into LibreSSL then to not consider a host "expired" if it was possible to generate a valid chain of trust, regardless of "extra" certs sent by the server? Indeed, Let's Encrypt's own documentation [2] thinks that only LibreSSL < 3.2.0 is affected, but that is not the case. LibreSSL 3.3.2 in OpenBSD 6.9 (before the new syspatch) still considered servers expired, as does the somewhat older version bundled in macOS. -Andrew [1] https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/ [2] https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
Re: Server certs expired higher up the chain, imaps and https
On 21-09-30 19:45:38, James Cook wrote: > On Thu, Sep 30, 2021 at 10:02:17AM -0700, Chris Bennett wrote: > > Hi, > > > > I'm getting that the certs are expired, but https works fine in Firefox, > > including when looking at the full chain. > > > > > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > > mail.strengthcouragewisdom.rocks:imaps > > > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > > mail.strengthcouragewisdom.rocks:https > > > > However are not happy. I force updated my ssl certs, syspatch, pkg_add > > -u and rebooted. > > > > I didn't rebuild dh.pem for dovecot. > > > > Is this just a DNS propagation issue? > > Or should I do something further myself? > > > > Thanks > > Chris Bennett > > A certificate in LetsEncrypt's chain expired today or yesterday. The > issue is a bit complicated. > > > There's a page here: > > https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ > > and a forum thread here: > > https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190 > > > Summary: generally, newer clients and web browsers will not give the > cert expired error, because the middle certificate on the chain is a > root cert in its own right. Other clients, including as far as I can > tell the LibreSSL version included in OpenBSD 6.9, are more strict and > reject the whole chain because the last certificate in the chain > expired. > > E.g. I just tried "ftp -o x > 'https://mail.strengthcouragewisdom.rocks/'" on -current and it > worked. Current uses the new X.509 verifier and is not impacted by this problem. > LetsEncrypt does not want to remove that last one from the chain > because older Android phones don't have that middle certificate as a > root CA. > > Some post(s) in the thread claim it is possible to request an alternate > chain from LetsEncrypt, if you want one that doesn't end with the > expired one. I couldn't find this functionality in OpenBSD's > acme-client. However, I tried manually editing the fullchain pem file > downloaded by acme-client, deleting the third of three certificates in > the file, and now my older clients are happy (but presumably old > Android phones will not be happy). Let's Encrypt is supplying alternate certificate chains. When you fetch the certificate from the ACME order, HTTP Link headers with 'rel="alternate"' are potentially provided and these can be fetched in the same way as the primary certificate chain. Unfortunately, acme-client does not currently support this (not to mention that chain selection also becomes messy from a application/configuration stand point).
Re: Server certs expired higher up the chain, imaps and https
I think I read in some news (slashdot? HN?) semi-recently that a bunch of old-style (?) Let's Encrypt certificates are expiring today. Different software packages may handle it differently, as to how they determine what to accept...? Sorry vague, but I something on my phone with one site that I'm guessing is from the same cause. On 2021-09-30 10:02:17-0700, Chris Bennett wrote: > Hi, > > I'm getting that the certs are expired, but https works fine in Firefox, > including when looking at the full chain. > > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:imaps > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:https > > However are not happy. I force updated my ssl certs, syspatch, pkg_add > -u and rebooted. > > I didn't rebuild dh.pem for dovecot. > > Is this just a DNS propagation issue? > Or should I do something further myself? > > Thanks > Chris Bennett >
Re: Server certs expired higher up the chain, imaps and https
On Thu, Sep 30, 2021 at 10:02:17AM -0700, Chris Bennett wrote: > Hi, > > I'm getting that the certs are expired, but https works fine in Firefox, > including when looking at the full chain. > > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:imaps > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:https > > However are not happy. I force updated my ssl certs, syspatch, pkg_add > -u and rebooted. > > I didn't rebuild dh.pem for dovecot. > > Is this just a DNS propagation issue? > Or should I do something further myself? > > Thanks > Chris Bennett A certificate in LetsEncrypt's chain expired today or yesterday. The issue is a bit complicated. There's a page here: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ and a forum thread here: https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190 Summary: generally, newer clients and web browsers will not give the cert expired error, because the middle certificate on the chain is a root cert in its own right. Other clients, including as far as I can tell the LibreSSL version included in OpenBSD 6.9, are more strict and reject the whole chain because the last certificate in the chain expired. E.g. I just tried "ftp -o x 'https://mail.strengthcouragewisdom.rocks/'" on -current and it worked. LetsEncrypt does not want to remove that last one from the chain because older Android phones don't have that middle certificate as a root CA. Some post(s) in the thread claim it is possible to request an alternate chain from LetsEncrypt, if you want one that doesn't end with the expired one. I couldn't find this functionality in OpenBSD's acme-client. However, I tried manually editing the fullchain pem file downloaded by acme-client, deleting the third of three certificates in the file, and now my older clients are happy (but presumably old Android phones will not be happy). -- James
Re: Server certs expired higher up the chain, imaps and https
Chris Bennett(cpb_m...@bennettconstruction.us) on 2021.09.30 10:02:17 -0700: > Hi, > > I'm getting that the certs are expired, but https works fine in Firefox, > including when looking at the full chain. > > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:imaps > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:https > > However are not happy. I force updated my ssl certs, syspatch, pkg_add > -u and rebooted. > > I didn't rebuild dh.pem for dovecot. > > Is this just a DNS propagation issue? > Or should I do something further myself? This is an issue with an expired root/intermediate certificate (DST Root X3) in use by Let's Encrypt. Stuart Henderson (sthen@) summarized it like this: LibreSSL in OpenBSD 6.9/earlier is having problems with the expiry of a CA certificate used to cross-sign Let's Encrypt certs. LE decided not to switch to using their own root fully, rather they are continuing to use the expired cross-signer to increase compatibility with old Android devices, which is tickling this problem. https://letsencrypt.org/2020/12/21/extending-android-compatibility.html An errata has just been published, you can install it using syspatch. Best, Benno
Re: Server certs expired higher up the chain, imaps and https
O 30/09/21 ás 19:02, Chris Bennett escribiu: > Hi, > > I'm getting that the certs are expired, but https works fine in Firefox, > including when looking at the full chain. > > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:imaps > > openssl s_client -servername mail.strengthcouragewisdom.rocks -connect > mail.strengthcouragewisdom.rocks:https > > However are not happy. I force updated my ssl certs, syspatch, pkg_add > -u and rebooted. > > I didn't rebuild dh.pem for dovecot. > > Is this just a DNS propagation issue? > Or should I do something further myself? > > Thanks > Chris Bennett > I just saw a similar thread on freebsd-questions[1], you might have the same problem that they had. [1] https://lists.freebsd.org/pipermail/freebsd-questions/2021-September/294839.html
Server certs expired higher up the chain, imaps and https
Hi, I'm getting that the certs are expired, but https works fine in Firefox, including when looking at the full chain. openssl s_client -servername mail.strengthcouragewisdom.rocks -connect mail.strengthcouragewisdom.rocks:imaps openssl s_client -servername mail.strengthcouragewisdom.rocks -connect mail.strengthcouragewisdom.rocks:https However are not happy. I force updated my ssl certs, syspatch, pkg_add -u and rebooted. I didn't rebuild dh.pem for dovecot. Is this just a DNS propagation issue? Or should I do something further myself? Thanks Chris Bennett
