All,
Off-topic, but I know there are plenty of Spring users on this list who
can probably help me figure this out.
Recently, Let's Encrypt switched from using their soon-to-be-expiring
intermediate certificate:
Owner: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: a0141420000015385736a0b85eca708
Valid from: Thu Mar 17 12:40:46 EDT 2016 until: Wed Mar 17 12:40:46 EDT 2021
To this new one:
Owner: CN=R3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 400175048314a4c8218c84a90c16cddf
Valid from: Wed Oct 07 15:21:40 EDT 2020 until: Wed Sep 29 15:21:40 EDT 2021
We are using LE for our LDAPS server and our latest-issued certificate
is signed by and includes the above ("R3") certificate in the server's
certificate chain when contacted by a client.
As soon as this switch occurred on Saturday, two of my dependent
services stopped working. It took me quite a while to figure out the
underlying issue, but I got one of them back up and running by simply
adding the new R3 intermediate certificate to the trust store of a
non-Java service (actually, it was Apache httpd mod_authz_ldap). That
was fun tracking-down, but I digress.
The other service is JasperReports Server, which uses Spring Security
for authentication against our LDAPS database. It's running on Tomcat,
and here is the important part (IMO) of the command-line of the running
process:
-Djavax.net.ssl.trustStrore=conf/truststore.jks
-Djavax.net.ssl.trustStrorePassword=[the password]
The "conf" directory refers to $CATALINA_BASE/conf.
Dumping the existing conf/truststore.jks file reveals the contents:
Alias name: letsencrypt
Creation date: Dec 12, 2016
Entry type: trustedCertEntry
Owner: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: a0141420000015385736a0b85eca708
Valid from: Thu Mar 17 12:40:46 EDT 2016 until: Wed Mar 17 12:40:46 EDT 2021
Nice. So just add the new certificate to the trust store and restart the
service, right?
$ keytool -importcert -alias 'letsencrypt2020' -keystore conf/truststore.jks
Enter keystore password:
-----BEGIN CERTIFICATE-----
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-----END CERTIFICATE-----
Owner: CN=R3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 400175048314a4c8218c84a90c16cddf
Valid from: Wed Oct 07 15:21:40 EDT 2020 until: Wed Sep 29 15:21:40 EDT 2021
Certificate fingerprints:
MD5: 31:21:28:F5:A0:ED:7B:A5:4B:65:82:92:87:56:BA:83
SHA1: 48:50:4E:97:4C:0D:AC:5B:5C:D4:76:C8:20:22:74:B2:4C:8C:71:72
SHA256:
73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
[
accessMethod: caIssuers
accessLocation: URIName: http://apps.identrust.com/roots/dstrootcax3.p7c
]
]
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: C4 A7 B1 A4 7B 2C 71 FA DB E1 4B 90 75 FF C4 15 .....,q...K.u...
0010: 60 85 89 10 `...
]
]
#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:0
]
#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: http://crl.identrust.com/DSTROOTCAX3CRL.crl]
]]
#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
[CertificatePolicyId: [2.23.140.1.2.1]
[] ]
[CertificatePolicyId: [1.3.6.1.4.1.44947.1.1.1]
[PolicyQualifierInfo: [
qualifierID: 1.3.6.1.5.5.7.2.1
qualifier: 0000: 16 22 68 74 74 70 3A 2F 2F 63 70 73 2E 72 6F 6F
."http://cps.roo
0010: 74 2D 78 31 2E 6C 65 74 73 65 6E 63 72 79 70 74 t-x1.letsencrypt
0020: 2E 6F 72 67 .org
]] ]
]
#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]
#7: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_CertSign
Crl_Sign
]
#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 14 2E B3 17 B7 58 56 CB AE 50 09 40 E6 1F AF 9D .....XV..P.@....
0010: 8B 14 C2 C6 ....
]
]
Trust this certificate? [no]:
yes
Certificate was added to keystore
Now, I restart the service.
I try to login and I get a failure in the logs with the stack trace:
EncryptionAuthenticationProcessingFilter,http-nio-8443-exec-10:218 - An
internal erro
r occurred while trying to authenticate the user.
org.springframework.security.authentication.InternalAuthenticationServiceException:
simple bind failed: intranet.ch
ildhealthcare.org:636; nested exception is
javax.naming.CommunicationException: simple bind failed: intranet.childh
ealthcare.org:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException
: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]
[......]
[......]
Caused by: javax.naming.CommunicationException: simple bind failed:
intranet.childhealthcare.org:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:218)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
at
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
at
com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
at
javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at
javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at
javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:153)
at
org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(LdapContextSource.java:43)
at
org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:254)
... 70 more
[......]
[......]
Okay, so it can't validate the server's certificate. Hmm. I thought I
would have fixed that by adding the new cert to the trust store. Maybe
it's using a different trust store.
$ find / -name truststore.jks
/home/jasperreports/jasperserver-tomcat/conf/truststore.jks
Okay, that's the one and only file.
Maybe it's not using the file at all. Let's try changing to
-Djavax.net.ssl.trustStrorePassword=WRONGPASSWORD.
Still got the same PKIX error. So it looks like it's not even trying to
use that file for trust. Maybe it's using the JVM's cacerts file for that?
keytool -list -v -keystore ${JAVA_HOME}jre/lib/security/cacerts | grep 'Let'
[nothing]
Wait, what? How was it working before?
Maybe it's in the configuration somewhere.
$ grep -i '\(trust\|store\)'
webapps/jasperserver/WEB-INF/applicationContext-externalAuth-LDAP.xml
[nothing]
I'm at the limit of my knowledge about Spring Security, here. Can anyone
suggest what might be happening, here? This was definitely working on
Friday, and since Saturday I can't make a connection. The only thing
that changed was the certificate chain from Let's Encrypt and me
(ineffectively) importing the new intermediate certificate into the
conf/truststore.jks file.
Any ideas?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
