All,
On 1/25/21 11:10, Christopher Schultz wrote:
All,
Off-topic, but I know there are plenty of Spring users on this list who
can probably help me figure this out.
Recently, Let's Encrypt switched from using their soon-to-be-expiring
intermediate certificate:
Owner: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: a0141420000015385736a0b85eca708
Valid from: Thu Mar 17 12:40:46 EDT 2016 until: Wed Mar 17 12:40:46 EDT
2021
To this new one:
Owner: CN=R3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 400175048314a4c8218c84a90c16cddf
Valid from: Wed Oct 07 15:21:40 EDT 2020 until: Wed Sep 29 15:21:40 EDT
2021
We are using LE for our LDAPS server and our latest-issued certificate
is signed by and includes the above ("R3") certificate in the server's
certificate chain when contacted by a client.
As soon as this switch occurred on Saturday, two of my dependent
services stopped working. It took me quite a while to figure out the
underlying issue, but I got one of them back up and running by simply
adding the new R3 intermediate certificate to the trust store of a
non-Java service (actually, it was Apache httpd mod_authz_ldap). That
was fun tracking-down, but I digress.
The other service is JasperReports Server, which uses Spring Security
for authentication against our LDAPS database. It's running on Tomcat,
and here is the important part (IMO) of the command-line of the running
process:
-Djavax.net.ssl.trustStrore=conf/truststore.jks
-Djavax.net.ssl.trustStrorePassword=[the password]
The "conf" directory refers to $CATALINA_BASE/conf.
Dumping the existing conf/truststore.jks file reveals the contents:
Alias name: letsencrypt
Creation date: Dec 12, 2016
Entry type: trustedCertEntry
Owner: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: a0141420000015385736a0b85eca708
Valid from: Thu Mar 17 12:40:46 EDT 2016 until: Wed Mar 17 12:40:46 EDT
2021
Nice. So just add the new certificate to the trust store and restart the
service, right?
$ keytool -importcert -alias 'letsencrypt2020' -keystore
conf/truststore.jks
Enter keystore password:
-----BEGIN CERTIFICATE-----
MIIEZTCCA02gAwIBAgIQQAF1BIMUpMghjISpDBbN3zANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIwMTAwNzE5MjE0MFoXDTIxMDkyOTE5MjE0MFow
MjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxCzAJBgNVBAMT
AlIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuwIVKMz2oJTTDxLs
jVWSw/iC8ZmmekKIp10mqrUrucVMsa+Oa/l1yKPXD0eUFFU1V4yeqKI5GfWCPEKp
Tm71O8Mu243AsFzzWTjn7c9p8FoLG77AlCQlh/o3cbMT5xys4Zvv2+Q7RVJFlqnB
U840yFLuta7tj95gcOKlVKu2bQ6XpUA0ayvTvGbrZjR8+muLj1cpmfgwF126cm/7
gcWt0oZYPRfH5wm78Sv3htzB2nFd1EbjzK0lwYi8YGd1ZrPxGPeiXOZT/zqItkel
/xMY6pgJdz+dU/nPAeX1pnAXFK9jpP+Zs5Od3FOnBv5IhR2haa4ldbsTzFID9e1R
oYvbFQIDAQABo4IBaDCCAWQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8E
BAMCAYYwSwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5p
ZGVudHJ1c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTE
p7Gkeyxx+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEE
AYLfEwEBATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0
LmNvbS9EU1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYf
r52LFMLGMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0B
AQsFAAOCAQEA2UzgyfWEiDcx27sT4rP8i2tiEmxYt0l+PAK3qB8oYevO4C5z70kH
ejWEHx2taPDY/laBL21/WKZuNTYQHHPD5b1tXgHXbnL7KqC401dk5VvCadTQsvd8
S8MXjohyc9z9/G2948kLjmE6Flh9dDYrVYA9x2O+hEPGOaEOa1eePynBgPayvUfL
qjBstzLhWVQLGAkXXmNs+5ZnPBxzDJOLxhF2JIbeQAcH5H0tZrUlo5ZYyOqA7s9p
O5b85o3AM/OJ+CktFBQtfvBhcJVd9wvlwPsk+uyOy2HI7mNxKKgsBTt375teA2Tw
UdHkhVNcsAKX1H7GNNLOEADksd86wuoXvg==
-----END CERTIFICATE-----
Owner: CN=R3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: 400175048314a4c8218c84a90c16cddf
Valid from: Wed Oct 07 15:21:40 EDT 2020 until: Wed Sep 29 15:21:40 EDT
2021
Certificate fingerprints:
MD5: 31:21:28:F5:A0:ED:7B:A5:4B:65:82:92:87:56:BA:83
SHA1: 48:50:4E:97:4C:0D:AC:5B:5C:D4:76:C8:20:22:74:B2:4C:8C:71:72
SHA256:
73:0C:1B:DC:D8:5F:57:CE:5D:C0:BB:A7:33:E5:F1:BA:5A:92:5B:2A:77:1D:64:0A:26:F7:A4:54:22:4D:AD:3B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 1.3.6.1.5.5.7.1.1 Criticality=false
AuthorityInfoAccess [
[
accessMethod: caIssuers
accessLocation: URIName:
http://apps.identrust.com/roots/dstrootcax3.p7c
]
]
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: C4 A7 B1 A4 7B 2C 71 FA DB E1 4B 90 75 FF C4 15 .....,q...K.u...
0010: 60 85 89 10 `...
]
]
#3: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:0
]
#4: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: http://crl.identrust.com/DSTROOTCAX3CRL.crl]
]]
#5: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
[CertificatePolicyId: [2.23.140.1.2.1]
[] ]
[CertificatePolicyId: [1.3.6.1.4.1.44947.1.1.1]
[PolicyQualifierInfo: [
qualifierID: 1.3.6.1.5.5.7.2.1
qualifier: 0000: 16 22 68 74 74 70 3A 2F 2F 63 70 73 2E 72 6F 6F
."http://cps.roo
0010: 74 2D 78 31 2E 6C 65 74 73 65 6E 63 72 79 70 74 t-x1.letsencrypt
0020: 2E 6F 72 67 .org
]] ]
]
#6: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]
#7: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_CertSign
Crl_Sign
]
#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 14 2E B3 17 B7 58 56 CB AE 50 09 40 E6 1F AF 9D .....XV..P.@....
0010: 8B 14 C2 C6 ....
]
]
Trust this certificate? [no]:
yes
Certificate was added to keystore
Now, I restart the service.
I try to login and I get a failure in the logs with the stack trace:
EncryptionAuthenticationProcessingFilter,http-nio-8443-exec-10:218 - An
internal erro
r occurred while trying to authenticate the user.
org.springframework.security.authentication.InternalAuthenticationServiceException:
simple bind failed: intranet.ch
ildhealthcare.org:636; nested exception is
javax.naming.CommunicationException: simple bind failed: intranet.childh
ealthcare.org:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException
: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]
[......]
[......]
Caused by: javax.naming.CommunicationException: simple bind failed:
intranet.childhealthcare.org:636 [Root exception is
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target]
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:218)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
at
com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
at
com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
at
com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
at
javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at
javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at
javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:153)
at
org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(LdapContextSource.java:43)
at
org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:254)
... 70 more
[......]
[......]
Okay, so it can't validate the server's certificate. Hmm. I thought I
would have fixed that by adding the new cert to the trust store. Maybe
it's using a different trust store.
$ find / -name truststore.jks
/home/jasperreports/jasperserver-tomcat/conf/truststore.jks
Okay, that's the one and only file.
Maybe it's not using the file at all. Let's try changing to
-Djavax.net.ssl.trustStrorePassword=WRONGPASSWORD.
Still got the same PKIX error. So it looks like it's not even trying to
use that file for trust. Maybe it's using the JVM's cacerts file for that?
keytool -list -v -keystore ${JAVA_HOME}jre/lib/security/cacerts | grep
'Let'
[nothing]
Wait, what? How was it working before?
Maybe it's in the configuration somewhere.
$ grep -i '\(trust\|store\)'
webapps/jasperserver/WEB-INF/applicationContext-externalAuth-LDAP.xml
[nothing]
I'm at the limit of my knowledge about Spring Security, here. Can anyone
suggest what might be happening, here? This was definitely working on
Friday, and since Saturday I can't make a connection. The only thing
that changed was the certificate chain from Let's Encrypt and me
(ineffectively) importing the new intermediate certificate into the
conf/truststore.jks file.
Any ideas?
I continued to fight with this after I posted my message. I went for the
nuclear option: modify cacerts to include the new LE intermediate
certificate, and that worked.
But why had it worked before, when cacerts didn't include the *previous*
intermediate certificate?
Well, I have no idea when I introduced this, but looking back carefully
at the copy/paste command line from my initial post, I find this:
-Djavax.net.ssl.trustStrore=conf/truststore.jks
Misppelled "trustStore". *facepalm*
I don't believe I changed my CATALINA_OPTS during all of this, but it
may have happened.
I restored my backup copy of JAVA_HOME/jre/lib/security/cacerts,
corrected the misspelling in that system property name, and restarted
JasperReports Server.
I'm now able to authenticate against my LDAPS server. Well, I was always
able to authenticate. I'm able to *handshake* with it, again :)
I'm still curious as to wy this worked before Saturday given:
1. cacerts doesn't contain anything Let's Encrypt-related
2. My own trust store wasn't in use due to the misspelling
So either I was confused about when my cacerts backup was made (maybe
long ago, matching its file date; or maybe on Saturday) and the older LE
intermediate cert *was* in there the whole time, or I somehow
fat-fingered the trustStore system property when I wasn't even intending
to change it while looking at the file. (I typically use vi when editing
config/script files, so stray edits are not typical.)
Anyhow, I hope this helps someone in the future: Spring Security's LDAP
authenrticator does indeed use the global trust store (unless it's been
overridden, which looks like it's a fairly involved process[1]).
-chris
[1] https://github.com/spring-projects/spring-ldap/issues/494
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
