Dear OpenSSL developers, We recently experienced an issue with our internal Mercurial repositories where Mercurial will refuse to connect to the repository due to an SSL certificate error. The problem appeared to show up randomly on some machines, but not others.
The repository is hosted on an SCM-Manager instance and served over https with a certificate signed by our internal root CA, which is in turn deployed to the machines on our network using Active Directory group policies. Recently, our IT department has pushed out a new root CA certificate through Active Directory. This CA is a renewal of the old one, with the validity time extended through 2021, and the signature algorithm changed to SHA256+RSA (was SHA1+RSA before), this was when the problem started happening. The old root CA was not revoked or uninstalled. I tracked down the issue to a problem with OpenSSL, which Mercurial uses for SSL support. Specifically, if OpenSSL is asked to verify an SSL certificate using a CA certificate store containing both a valid CA certificate and an expired one, both correct CA certs for the SSL one, it will _sometimes_ reject the certificate with the following error message: secondary.pem: CN = testing error 10 at 1 depth lookup:certificate has expired The behavior is dependent on the physical ordering of the CA certificates in the store. Specifically, if the only certificates in the store are the valid and expired CAs, then the error occurs only if the valid one comes first. With more than two certificates in the store, the behavior appears to be completely random (but shows up consistently for the same orderings). E.g. with two valid CAs and an expired one, all 3 signing the same SSL cert, the following orders work: older valid CA, expired CA, newer valid CA older valid CA, newer valid CA, expired CA newer valid CA, expired CA, older valid CA but these fail with "certificate has expired": expired CA, older valid CA, newer valid CA (logical ordering, this is the order in which Windows returns certificates deployed using AD) expired CA, newer valid CA, older valid CA newer valid CA, older valid CA, expired CA As a simple testcase, I created the following 3 PEM files: goodcacerts.pem: -----BEGIN CERTIFICATE----- MIIEwzCCAqugAwIBAgIEAwvYVTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0 ZXN0aW5nMB4XDTA3MDcwNzA1MDcwN1oXDTA4MDcwNjA1MDcwN1owEjEQMA4GA1UE AxMHdGVzdGluZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAIHAVog+ GiK2M/QqsnCDTH4U9C2Bp47CEGNvHBWLVJh2ikfnpydmQUgo2JPeAn7UldRZKtS0 ORQt9mSvFdL9llxfbQVn4XnJM3GzKewQjrvBvBm0c9v99k37F0F0rlDrm1wWzs8C Jzw2xiBSeznpLWCWtOKvWy7P9kGNlN32xNNw7P9hSfMjwXeXICCWlj0nD1rngoyp jvrrobXGpZ+5gUUBg40AspLlN0bsaMjBSBmpFIZs7g9iPmVP4ANlzkPOwjLfXlk3 1CFef8F48NrlprjAzRAvSSgpixDCeYu/yjK+1IKCo1g9bYTVv41RlCs9f2Q6JjQ8 t/CtHkFnin6wKgnEQwpkwPd8DIRpgKzcYQkFym2kPolP1BPE6vvuLEwb2k1sokjQ U35HEL9UvLzhEPlIPqtzaOcV3eQb08GpLv8CihHHubtWXV4DIrkBHwLDwC9inM/l NSc3JefFy8I6BWxxzmXu8Wcvn9EsheJKswnJB+/0IgBuXT5RFdpBNCFRI/mdb33N bIpS27J7CZW3sbXT2UVMuYTU7862nymd0m/bY8PQHypqep0xCCzEd35qC4jaY+RA DJZQGRbEXfOkRGkxxzvES8m87pThT5SIJzutuiiufmbtlqdZWv+a3v/L+rpjIBet cRF2Sb44aTmc9RvAJ802s7XxlrYuyqy4D/MzAgMBAAGjITAfMB0GA1UdDgQWBBSY /19/f9b2cU+d4GT8hVS7LWD+ejANBgkqhkiG9w0BAQsFAAOCAgEAV+PnySD6Nkct SsohAvra8Xuski5ZwHfymAICRM9ZWD6s/2/7y652vegMw40wxW3ZeT7NTzoi+MiT LgYsbdu7upRGvS6dnB7ye8udOtIPhk/QPqDgH0SkxB2imeUaprX8EJUYFQX0AUd7 MAqMcqVFrpYo7xjXCxsWpWPzT6USsG9gjcSbWoukxIGmhlNvtKycSY3+L0HFZKyr mMHZxYsUxVyPKBwfhwQTC/RQccCNLRdW68eaJSjxrbFjkMCLtppGqKHfRr/KBMvf 7K5q52vZ2O324yVdQdQjPuLk84pyJmyZ8Ew48XK06ebHgbx+fBDsNYq27IU4GRm+ FMl/Rp0q/fxZ3CphBNBdyDRRIGzIJN4e3ocSeNDcQt6uD7h88D0FrCXrlkxYGW51 qvaxFmUXlAjuv0K2iRDzvVbP2A6JhB2PBUw/lkEDcy2WP0WctLdL/GVeatRTUlzw jxQRSi21Azwq9I4fqG/cwDDiWPGmyP2/xJoZROhky0zm/0A9mWEU9R7tB+FppZZN 3Un9HtSIRt30mqC1AgcIdiFlXpS5RiJElgIEuJVma3P2BxFK2glycy12UYy9gcOw VV9D2OrLNxKp9icmy/D2b9Z2udLji9jI0MIhl/xDlshsTtkDmOshgqnKIyNzF/gF 8fgqQZm0AezWcLtrnbFTSG2f6Rpdjec= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIExTCCAq2gAwIBAgIEKKekeTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0 ZXN0aW5nMCAXDTA3MTIzMTIzMDAwMFoYDzIwOTcwOTE2MjMwMDAwWjASMRAwDgYD VQQDEwd0ZXN0aW5nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgcBW iD4aIrYz9CqycINMfhT0LYGnjsIQY28cFYtUmHaKR+enJ2ZBSCjYk94CftSV1Fkq 1LQ5FC32ZK8V0v2WXF9tBWfheckzcbMp7BCOu8G8GbRz2/32TfsXQXSuUOubXBbO zwInPDbGIFJ7OektYJa04q9bLs/2QY2U3fbE03Ds/2FJ8yPBd5cgIJaWPScPWueC jKmO+uuhtcaln7mBRQGDjQCykuU3RuxoyMFIGakUhmzuD2I+ZU/gA2XOQ87CMt9e WTfUIV5/wXjw2uWmuMDNEC9JKCmLEMJ5i7/KMr7UgoKjWD1thNW/jVGUKz1/ZDom NDy38K0eQWeKfrAqCcRDCmTA93wMhGmArNxhCQXKbaQ+iU/UE8Tq++4sTBvaTWyi SNBTfkcQv1S8vOEQ+Ug+q3No5xXd5BvTwaku/wKKEce5u1ZdXgMiuQEfAsPAL2Kc z+U1Jzcl58XLwjoFbHHOZe7xZy+f0SyF4kqzCckH7/QiAG5dPlEV2kE0IVEj+Z1v fc1silLbsnsJlbextdPZRUy5hNTvzrafKZ3Sb9tjw9AfKmp6nTEILMR3fmoLiNpj 5EAMllAZFsRd86REaTHHO8RLybzulOFPlIgnO626KK5+Zu2Wp1la/5re/8v6umMg F61xEXZJvjhpOZz1G8AnzTaztfGWti7KrLgP8zMCAwEAAaMhMB8wHQYDVR0OBBYE FJj/X39/1vZxT53gZPyFVLstYP56MA0GCSqGSIb3DQEBCwUAA4ICAQBP1bWTxa0L UkBHMMBvnRkof+qE7hms/mJslKFVoYxD+fRoBKU2vPPCqYqbENwMGZtHxBQ2CZFU y4gUJQClrzqT1Pk5IcSVkAwe4RFQ7+IalITvyF0QRQZ1qaLvo893lBdQH2RfLyZG hFl6z+dPjWBvh8w9Oo/LpzyV1LEYd25LtPL3ZJsH5Xbh1RP2oESVkd/qacxat3kn ljmpxsuj4xJz1/VOin8xRJN97bz9kUO76Fs9ICCHSPunaVoYucCzuOG8JYPaIuUG 9I8ymNPDjcI48Gf+mxS4cs89NuSYGJ2CLB2/Knx2ViebcKnx79x62QmYco8vF5iK G08yGJ4E4J+ERj5j7EfnUfKFDUSowzt97l7cJ1hDOx8qoeWSbaQdC2PlF5NnBZLW mkCmMttlCs75V15Bw662AB1vGzEzKn79dTv2XsLOf7105Oq+0wgPH+iKXRYLYnDR s+uGTY5pCU5K37Uv4T7m761JDqkipMobhWSDjzaTI4LE8qmJGWb3jwmW4nZBZmrT FddmZyOMIQUpkLcNHea2ESdbFdo2JsUnfMGtY3XxAoeIW6+JYSBwlfjkpECM9gPH 4L27x01NmQL5yVxNHjN0EjittE/HHMXgntIRTYktjN5V8eZla85vwR72PSafUtpr x51x19J+pGfSpXRPO48tGi8+YUckS7qN4w== -----END CERTIFICATE----- badcacerts.pem: -----BEGIN CERTIFICATE----- MIIExTCCAq2gAwIBAgIEKKekeTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0 ZXN0aW5nMCAXDTA3MTIzMTIzMDAwMFoYDzIwOTcwOTE2MjMwMDAwWjASMRAwDgYD VQQDEwd0ZXN0aW5nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgcBW iD4aIrYz9CqycINMfhT0LYGnjsIQY28cFYtUmHaKR+enJ2ZBSCjYk94CftSV1Fkq 1LQ5FC32ZK8V0v2WXF9tBWfheckzcbMp7BCOu8G8GbRz2/32TfsXQXSuUOubXBbO zwInPDbGIFJ7OektYJa04q9bLs/2QY2U3fbE03Ds/2FJ8yPBd5cgIJaWPScPWueC jKmO+uuhtcaln7mBRQGDjQCykuU3RuxoyMFIGakUhmzuD2I+ZU/gA2XOQ87CMt9e WTfUIV5/wXjw2uWmuMDNEC9JKCmLEMJ5i7/KMr7UgoKjWD1thNW/jVGUKz1/ZDom NDy38K0eQWeKfrAqCcRDCmTA93wMhGmArNxhCQXKbaQ+iU/UE8Tq++4sTBvaTWyi SNBTfkcQv1S8vOEQ+Ug+q3No5xXd5BvTwaku/wKKEce5u1ZdXgMiuQEfAsPAL2Kc z+U1Jzcl58XLwjoFbHHOZe7xZy+f0SyF4kqzCckH7/QiAG5dPlEV2kE0IVEj+Z1v fc1silLbsnsJlbextdPZRUy5hNTvzrafKZ3Sb9tjw9AfKmp6nTEILMR3fmoLiNpj 5EAMllAZFsRd86REaTHHO8RLybzulOFPlIgnO626KK5+Zu2Wp1la/5re/8v6umMg F61xEXZJvjhpOZz1G8AnzTaztfGWti7KrLgP8zMCAwEAAaMhMB8wHQYDVR0OBBYE FJj/X39/1vZxT53gZPyFVLstYP56MA0GCSqGSIb3DQEBCwUAA4ICAQBP1bWTxa0L UkBHMMBvnRkof+qE7hms/mJslKFVoYxD+fRoBKU2vPPCqYqbENwMGZtHxBQ2CZFU y4gUJQClrzqT1Pk5IcSVkAwe4RFQ7+IalITvyF0QRQZ1qaLvo893lBdQH2RfLyZG hFl6z+dPjWBvh8w9Oo/LpzyV1LEYd25LtPL3ZJsH5Xbh1RP2oESVkd/qacxat3kn ljmpxsuj4xJz1/VOin8xRJN97bz9kUO76Fs9ICCHSPunaVoYucCzuOG8JYPaIuUG 9I8ymNPDjcI48Gf+mxS4cs89NuSYGJ2CLB2/Knx2ViebcKnx79x62QmYco8vF5iK G08yGJ4E4J+ERj5j7EfnUfKFDUSowzt97l7cJ1hDOx8qoeWSbaQdC2PlF5NnBZLW mkCmMttlCs75V15Bw662AB1vGzEzKn79dTv2XsLOf7105Oq+0wgPH+iKXRYLYnDR s+uGTY5pCU5K37Uv4T7m761JDqkipMobhWSDjzaTI4LE8qmJGWb3jwmW4nZBZmrT FddmZyOMIQUpkLcNHea2ESdbFdo2JsUnfMGtY3XxAoeIW6+JYSBwlfjkpECM9gPH 4L27x01NmQL5yVxNHjN0EjittE/HHMXgntIRTYktjN5V8eZla85vwR72PSafUtpr x51x19J+pGfSpXRPO48tGi8+YUckS7qN4w== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIEwzCCAqugAwIBAgIEAwvYVTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0 ZXN0aW5nMB4XDTA3MDcwNzA1MDcwN1oXDTA4MDcwNjA1MDcwN1owEjEQMA4GA1UE AxMHdGVzdGluZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAIHAVog+ GiK2M/QqsnCDTH4U9C2Bp47CEGNvHBWLVJh2ikfnpydmQUgo2JPeAn7UldRZKtS0 ORQt9mSvFdL9llxfbQVn4XnJM3GzKewQjrvBvBm0c9v99k37F0F0rlDrm1wWzs8C Jzw2xiBSeznpLWCWtOKvWy7P9kGNlN32xNNw7P9hSfMjwXeXICCWlj0nD1rngoyp jvrrobXGpZ+5gUUBg40AspLlN0bsaMjBSBmpFIZs7g9iPmVP4ANlzkPOwjLfXlk3 1CFef8F48NrlprjAzRAvSSgpixDCeYu/yjK+1IKCo1g9bYTVv41RlCs9f2Q6JjQ8 t/CtHkFnin6wKgnEQwpkwPd8DIRpgKzcYQkFym2kPolP1BPE6vvuLEwb2k1sokjQ U35HEL9UvLzhEPlIPqtzaOcV3eQb08GpLv8CihHHubtWXV4DIrkBHwLDwC9inM/l NSc3JefFy8I6BWxxzmXu8Wcvn9EsheJKswnJB+/0IgBuXT5RFdpBNCFRI/mdb33N bIpS27J7CZW3sbXT2UVMuYTU7862nymd0m/bY8PQHypqep0xCCzEd35qC4jaY+RA DJZQGRbEXfOkRGkxxzvES8m87pThT5SIJzutuiiufmbtlqdZWv+a3v/L+rpjIBet cRF2Sb44aTmc9RvAJ802s7XxlrYuyqy4D/MzAgMBAAGjITAfMB0GA1UdDgQWBBSY /19/f9b2cU+d4GT8hVS7LWD+ejANBgkqhkiG9w0BAQsFAAOCAgEAV+PnySD6Nkct SsohAvra8Xuski5ZwHfymAICRM9ZWD6s/2/7y652vegMw40wxW3ZeT7NTzoi+MiT LgYsbdu7upRGvS6dnB7ye8udOtIPhk/QPqDgH0SkxB2imeUaprX8EJUYFQX0AUd7 MAqMcqVFrpYo7xjXCxsWpWPzT6USsG9gjcSbWoukxIGmhlNvtKycSY3+L0HFZKyr mMHZxYsUxVyPKBwfhwQTC/RQccCNLRdW68eaJSjxrbFjkMCLtppGqKHfRr/KBMvf 7K5q52vZ2O324yVdQdQjPuLk84pyJmyZ8Ew48XK06ebHgbx+fBDsNYq27IU4GRm+ FMl/Rp0q/fxZ3CphBNBdyDRRIGzIJN4e3ocSeNDcQt6uD7h88D0FrCXrlkxYGW51 qvaxFmUXlAjuv0K2iRDzvVbP2A6JhB2PBUw/lkEDcy2WP0WctLdL/GVeatRTUlzw jxQRSi21Azwq9I4fqG/cwDDiWPGmyP2/xJoZROhky0zm/0A9mWEU9R7tB+FppZZN 3Un9HtSIRt30mqC1AgcIdiFlXpS5RiJElgIEuJVma3P2BxFK2glycy12UYy9gcOw VV9D2OrLNxKp9icmy/D2b9Z2udLji9jI0MIhl/xDlshsTtkDmOshgqnKIyNzF/gF 8fgqQZm0AezWcLtrnbFTSG2f6Rpdjec= -----END CERTIFICATE----- site.pem: -----BEGIN CERTIFICATE----- MIIE5TCCAs2gAwIBAgIEBxm5gTANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwd0 ZXN0aW5nMB4XDTE2MDYyMDIwMDU1NVoXDTM4MTEyNDIwMDU1NVowEzERMA8GA1UE AxMIdGVzdGluZzIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCfeSXR qF0VCSZ3cT6bnEP4BkVqnM09RI+Ycn/Oh5A/FGDqHc2T2OcdvkTU4KnuwPqShW5x gvNGSeHDBHyYcHzrcu/unyVgt8u/247g4egq/Ep8edNkSkdws5N9JTxwm74H31af ndSRYeyC2m2pwZx1sVd0gOK1NodlpP/hcm7NPrcAE6HFzkemzJrsCgxQFhWsRvba GlEA40ZhOjGvSKfAb2oCfpW5pFYx7kSI1ii/VfX8H7CU7ZJUmj7stuc0pXZvu11v FsssenVX7XBxBwrDDoJdjCeEz1zfl68P7QhuCLS541SgHg/ZQPSTyMsQlzhR05Z1 qaiP6oUFQSFvO7j3e6UbIgSxyXK1tRevbhwGLPBzfUgUzeKiRrMAkIDj1dC9NjK2 DAKO5VuvvQKhCLD21a5XhFMDjV1Ig2TUMMT1n55f4I6EXadcBXcBRPszZNzEgVRd lsNGmsESCQ60bdRAnzxbbuP3A7Ufi0ASljzu3wwNGlOoEGDR3aOl341GO5x7z/gB sMAaN0LqZnb3ACjKW6NE1jsYPWtW37y6xjnTliWA7wmcUoOZwYgwTOGuF+3lmyto vJT2XVnli3wWUT2TPh9t4MgGzYePYiinpem3S9Ylsw7JNR0r4EDqyLF+8r4+Wg2V AhpOKkzcOfRaxXjxWNiJ8CqLpb6E8P7p2cb1QwIDAQABo0IwQDAfBgNVHSMEGDAW gBSY/19/f9b2cU+d4GT8hVS7LWD+ejAdBgNVHQ4EFgQU/+XByYexoutQKmN8aKwm 9ihiokYwDQYJKoZIhvcNAQELBQADggIBADX9+ihl9fGZelPZ426NYBWHvhr2LjB2 gVVPQpGG53AD5HOSRD1Lf0atfOXWnM0DXqN15cEp4jhB5ui16Vp8dYbU+7TSMpjX W3emVBk09oHgYA23W8OSdi93rx5JARGKY36aY1Si79MnJ40mR3LlZqnx54IOujX0 k8kLEkZ+MzbZnxxYfupr8wqihAgpNFJCTuW7QT9tlHATdlUN1zsdh9RtPal4Wngi dKk/L02VvvCLLfs37HCf8X/D89YDCH6XvXYok2TdhOe0VIiaa1JB0e0G/4rraPqw e80FRr+5tEJFRMgG0QYgZo61l73UMNNB0MQ6xKN0UR6K9W05B7ILwAMiRyZFCJ9f E+voJix+IWNtb495XfQGKpc/ALK+ifCdo1s6gIdE3RtEMZAwkdDjut/wtjaDfH78 8HUYJwAyrfQERrX5InF6pC9i5GasUcFnrq3zgq2jjddNMCQVbPZYxiabXq0qwDho Mevk5tkPunZEfE3Npbb85vxPnn7z1uE3C6CDr0ZIRC2frlD6ShFqw6FhFQYnNvXd Phs8fKQEwPN/PSb46rY/iLbASMNvh14Ut971Ta4ZKW/JHemfeRPUzqCECbE0jiKq k5KEX5Ge7lLQ6i1q6jEPizR9z/HPbkkSNEIJN+UaTXW78i+0WGP8cLQIzGZFRNZB uxRMPjdZ/XpR -----END CERTIFICATE----- goodcacerts.pem and badcacerts.pem contain exactly the same certificates (only ordered differently), yet: C:\Users\gstefanik\Documents\buggycerts>openssl verify -CAfile goodcacerts.pem site.pem site.pem: OK C:\Users\gstefanik\Documents\buggycerts>openssl verify -CAfile badcacerts.pem site.pem site.pem: CN = testing error 10 at 1 depth lookup:certificate has expired OK Even worse, some 32-bit Windows builds of openssl actually crash when verifying using badcacerts.pem (e.g. https://indy.fulgan.com/SSL/openssl-1.0.2h-i386-win32.zip) Unfortunately I was not able to compile a 32-bit Windows build myself. Sincerely, Gábor Stefanik -------------------------------------------------------------------------- This message, including its attachments, is confidential. For more information please read NNG's email policy here: http://www.nng.com/emailpolicy/ By responding to this email you accept the email policy. -- Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4580 Please log in as guest with password guest if prompted -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev