RE: Client Cert TLS issue
Thanks Mark, will do! -Original Message- From: Mark Thomas Sent: Thursday, October 31, 2019 3:04 PM To: Tomcat Users List ; George Stanchev Subject: Re: Client Cert TLS issue On 16/10/2019 18:55, George Stanchev wrote: > And this is not where it hangs. I stepped through the code through the > handshaker but still haven't been able to figure out the hang point as I am > not familiar with the details of that portion of TC code . I've attached two > cacerts that you can plug in as truststore to try out the issue. George, The mailing list strips most attachments. Please open a BZ issue for this. One, that means it won't get forgotten and two you will be able to attach those cacert files so we can easily reproduce the issue you are seeing. Thanks, Mark
Re: Client Cert TLS issue
On 16/10/2019 18:55, George Stanchev wrote: > And this is not where it hangs. I stepped through the code through the > handshaker but still haven't been able to figure out the hang point as I am > not familiar with the details of that portion of TC code . I've attached two > cacerts that you can plug in as truststore to try out the issue. George, The mailing list strips most attachments. Please open a BZ issue for this. One, that means it won't get forgotten and two you will be able to attach those cacert files so we can easily reproduce the issue you are seeing. Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Client Cert TLS issue
Thanks Mark,
I was going to try to build tcnative with debug this weekend to try to help out
but life begged to disagree...i will try to step in that code next week as
well...
Appreciate it!
George
-Original Message-
From: Mark Thomas
Sent: Sunday, October 20, 2019 2:38 PM
To: users@tomcat.apache.org
Subject: Re: Client Cert TLS issue
Just a note to say I haven't forgotten this. I hope to look at this this week
unless someone beats me to it.
Mark
On 16/10/2019 17:55, George Stanchev wrote:
>
> On 15/10/2019 22:15, George Stanchev wrote:
>> Hi,
>>
>> I would need some help with tracking an issue with TC 8.5.47 (windows x64,
>> java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple
>> client tries to connect to the server, the server hangs on SSL handshake
>> until either the client times out on read or the server times out (if I set
>> the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I
>> have enabled the client side SSL trace and everything goes well until ECDH
>> key exchange (for brevity I have enabled only one TLS suite
>> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if
>> requested. The cacerts we use is whatever comes with Azul's Java distro
>> which has ~150 entries + the custom cert for testing. The issue comes from
>> how the connector deals with trusted certs because if I reduce the cacerts
>> to contain only the test certificate, the request is being served without an
>> issue. Also if I remove the tcnative-1.dll from TC there is no issue either.
>>
>> Perhaps I am missing something. Any help is appreciated.
>
>
> This sounds like it is repeatable and that you have a system you can test
> with. On that basis here are a few things to try:
>
> 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake
> is hanging.
>
> 2. Try a binary search to try and determine if the issue is the number of
> certificates in the trust store or is caused by a specific certificate.
>
> It sounds like there might be an issue converting one or more of the trusted
> certs in the trust store to a format OpenSSL can work with.
>
>
>
> So the thread dumps didn't prove to be very useful - at least I couldn't see
> anything. I attached one of them (dump3.txt). But the trial and error in the
> binary searched proved it is not any certificate but a magic number of
> certificates in cacerts - 125 hang and 124 work. I don't know if it is a
> memory issue or buffer size issue - I tried removing and adding random certs
> around the 124/125 boundary and which one is out and in doesn't make a
> difference. I realize that a cert is rather large blob so it still could be
> memory issue with the size of the cert doesn't matter that much. In
> 8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms,
> tms, sr) we pass the DER-encoded certs to OpenSSL:
>
> // Pass along the DER encoded certificates of the accepted
> client
> // certificate issuers, so that their subjects can be
> presented
> // by the server during the handshake to allow the client
> choosing
> // an acceptable certificate
> for (X509Certificate caCert :
> x509TrustManager.getAcceptedIssuers()) {
> SSLContext.addClientCACertificateRaw(ctx,
> caCert.getEncoded());
> if (log.isDebugEnabled())
> log.debug(sm.getString("openssl.addedClientCaCert",
> caCert.toString()));
> }
>
> And this is not where it hangs. I stepped through the code through the
> handshaker but still haven't been able to figure out the hang point as I am
> not familiar with the details of that portion of TC code . I've attached two
> cacerts that you can plug in as truststore to try out the issue.
>
> George
>
>
>
>>
>> George
>>
>> [1]
>>
>> > port="8443" SSLEnabled="true" maxHttpHeaderSize="8192"
>> maxThreads="150" acceptCount="100" enableLookups="false"
>> disableUploadTimeout="true"
>> scheme="https" secure="true" clientAuth="true" sslProtocol="TLS"
>> sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> keystoreType="JKS"
>> keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit"
Re: Client Cert TLS issue
Just a note to say I haven't forgotten this. I hope to look at this this
week unless someone beats me to it.
Mark
On 16/10/2019 17:55, George Stanchev wrote:
>
> On 15/10/2019 22:15, George Stanchev wrote:
>> Hi,
>>
>> I would need some help with tracking an issue with TC 8.5.47 (windows x64,
>> java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple
>> client tries to connect to the server, the server hangs on SSL handshake
>> until either the client times out on read or the server times out (if I set
>> the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I
>> have enabled the client side SSL trace and everything goes well until ECDH
>> key exchange (for brevity I have enabled only one TLS suite
>> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if
>> requested. The cacerts we use is whatever comes with Azul's Java distro
>> which has ~150 entries + the custom cert for testing. The issue comes from
>> how the connector deals with trusted certs because if I reduce the cacerts
>> to contain only the test certificate, the request is being served without an
>> issue. Also if I remove the tcnative-1.dll from TC there is no issue either.
>>
>> Perhaps I am missing something. Any help is appreciated.
>
>
> This sounds like it is repeatable and that you have a system you can test
> with. On that basis here are a few things to try:
>
> 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake
> is hanging.
>
> 2. Try a binary search to try and determine if the issue is the number of
> certificates in the trust store or is caused by a specific certificate.
>
> It sounds like there might be an issue converting one or more of the trusted
> certs in the trust store to a format OpenSSL can work with.
>
>
>
> So the thread dumps didn't prove to be very useful - at least I couldn't see
> anything. I attached one of them (dump3.txt). But the trial and error in the
> binary searched proved it is not any certificate but a magic number of
> certificates in cacerts - 125 hang and 124 work. I don't know if it is a
> memory issue or buffer size issue - I tried removing and adding random certs
> around the 124/125 boundary and which one is out and in doesn't make a
> difference. I realize that a cert is rather large blob so it still could be
> memory issue with the size of the cert doesn't matter that much. In
> 8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms,
> tms, sr) we pass the DER-encoded certs to OpenSSL:
>
> // Pass along the DER encoded certificates of the accepted
> client
> // certificate issuers, so that their subjects can be
> presented
> // by the server during the handshake to allow the client
> choosing
> // an acceptable certificate
> for (X509Certificate caCert :
> x509TrustManager.getAcceptedIssuers()) {
> SSLContext.addClientCACertificateRaw(ctx,
> caCert.getEncoded());
> if (log.isDebugEnabled())
> log.debug(sm.getString("openssl.addedClientCaCert",
> caCert.toString()));
> }
>
> And this is not where it hangs. I stepped through the code through the
> handshaker but still haven't been able to figure out the hang point as I am
> not familiar with the details of that portion of TC code . I've attached two
> cacerts that you can plug in as truststore to try out the issue.
>
> George
>
>
>
>>
>> George
>>
>> [1]
>>
>> > port="8443" SSLEnabled="true" maxHttpHeaderSize="8192"
>> maxThreads="150" acceptCount="100" enableLookups="false"
>> disableUploadTimeout="true"
>> scheme="https" secure="true" clientAuth="true" sslProtocol="TLS"
>> sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> keystoreType="JKS"
>> keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit"
>> keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
>> ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
>> TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
>> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
>> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
>> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
>> TLS_ECDHE_RSA_WIT
RE: Client Cert TLS issue
Hi Chris, This didn't make a difference and no difference in CPU utilization was observed. The lockup happens before server request the client cert so you can drop my big fat cacerts in a TC instance create a client-cert connector, in our case we set the " javax.net.ssl.trustStore" to explicitly point to cacerts and hit it with curl -k https://localhost:8443/ George -Original Message- From: Christopher Schultz Sent: Wednesday, October 16, 2019 1:16 PM To: users@tomcat.apache.org Subject: Re: Client Cert TLS issue -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 George, On 10/16/19 12:55, George Stanchev wrote: > > On 15/10/2019 22:15, George Stanchev wrote: >> Hi, >> >> I would need some help with tracking an issue with TC 8.5.47 (windows >> x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. >> When a simple client tries to connect to the server, the server hangs >> on SSL handshake until either the client times out on read or the >> server times out (if I set the >> HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I >> have enabled the client side SSL trace and everything goes well until >> ECDH key exchange (for brevity I have enabled only one TLS suite >> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs >> if requested. The cacerts we use is whatever comes with Azul's Java >> distro which has ~150 entries + the custom cert for testing. The >> issue comes from how the connector deals with trusted certs because >> if I reduce the cacerts to contain only the test certificate, the >> request is being served without an issue. >> Also if I remove the tcnative-1.dll from TC there is no issue either. >> >> Perhaps I am missing something. Any help is appreciated. > > This sounds like it is repeatable and that you have a > system you can test with. On that basis here are a few things to > try: > > 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS > handshake is hanging. > > 2. Try a binary search to try and determine if the issue is the number > of certificates in the trust store or is caused by a specific > certificate. > > It sounds like there might be an issue converting one or more of the > trusted certs in the trust store to a format OpenSSL can work with. > > > > So the thread dumps didn't prove to be very useful - at least I > couldn't see anything. Me, either. Can you try again using: pollerThreadCount="1" and acceptorThreadCount="1" on your to see if that changes anything? Do you notice a spike in CPU when the connection hangs? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl2nbIYACgkQHPApP6U8 pFguXQ//cFd9lOb665cZRC68AB/0p869DBPRN9Cie8EALUgHicYSdEiRCWAUofuJ MZx8/Sds3mrB965nmfcNLWRQBBkDqVRrN39yqOMumAZeohZq9phUbGT8chsdHD8C tI8T96bEvKIlOtHoWK9qIYoEKZQ0nKotc0Rz49xZXTNmHmIred4nmR5fCQG4R5qD xoqXOo5wui3aN7y9VBk+sWBQh0HGpOXvGAHaQ2NbSKE6VTitiiEM92n6Dkz60wnG QbK2KrflW5E36NdVwNvFkqR/H3WVCrABBJ6puGHAL3nmlhg/n+MTpyqd7nkg1WSU j1U+hqxN4EHPlTcBUtaeb6DriwaQUIHNMH3h0J8H/UyfvoIRVPA570LF5Cycj7oK zlVgVmsZIZjIzt+qP6xzKkvhXzPLpemIOheDOZO4opgPdHIXGPAI9XVzwrARxMfv KeqyA16XrU6pM7GKvkDEnSDiMye/pPGbq/U3mnYdlRs4Lwn9PvmnzBasSXbrsg6i qeU1v6lSWPx18/9Qq1Qyfjxfgu3SkBpvHypwdv3MNMBk6Y2Gp/pg917FyqfNvoxX l0TaIYYf5xL6bHsxj1uopUoCnl4KxaTAaQ3qYg4+hFO3nzDNTB1k+Xu+pbuePr3U CyaIdmM3sqN7fNIwbfQ1slXrckavl+z/ZZmTZn2zIfa1yeMmP4s= =vkfG -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Client Cert TLS issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 George, On 10/16/19 12:55, George Stanchev wrote: > > On 15/10/2019 22:15, George Stanchev wrote: >> Hi, >> >> I would need some help with tracking an issue with TC 8.5.47 >> (windows x64, java: azul 1.8.0_222) configured with [1] and >> tcnative-1.dll. When a simple client tries to connect to the >> server, the server hangs on SSL handshake until either the client >> times out on read or the server times out (if I set the >> HttpsURLConnection#setConnectTimeout(0) and >> ...#setReadTimeout(0)). I have enabled the client side SSL trace >> and everything goes well until ECDH key exchange (for brevity I >> have enabled only one TLS suite >> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug >> logs if requested. The cacerts we use is whatever comes with >> Azul's Java distro which has ~150 entries + the custom cert for >> testing. The issue comes from how the connector deals with >> trusted certs because if I reduce the cacerts to contain only the >> test certificate, the request is being served without an issue. >> Also if I remove the tcnative-1.dll from TC there is no issue >> either. >> >> Perhaps I am missing something. Any help is appreciated. > > This sounds like it is repeatable and that you have a > system you can test with. On that basis here are a few things to > try: > > 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS > handshake is hanging. > > 2. Try a binary search to try and determine if the issue is the > number of certificates in the trust store or is caused by a > specific certificate. > > It sounds like there might be an issue converting one or more of > the trusted certs in the trust store to a format OpenSSL can work > with. > > > > So the thread dumps didn't prove to be very useful - at least I > couldn't see anything. Me, either. Can you try again using: pollerThreadCount="1" and acceptorThreadCount="1" on your to see if that changes anything? Do you notice a spike in CPU when the connection hangs? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl2nbIYACgkQHPApP6U8 pFguXQ//cFd9lOb665cZRC68AB/0p869DBPRN9Cie8EALUgHicYSdEiRCWAUofuJ MZx8/Sds3mrB965nmfcNLWRQBBkDqVRrN39yqOMumAZeohZq9phUbGT8chsdHD8C tI8T96bEvKIlOtHoWK9qIYoEKZQ0nKotc0Rz49xZXTNmHmIred4nmR5fCQG4R5qD xoqXOo5wui3aN7y9VBk+sWBQh0HGpOXvGAHaQ2NbSKE6VTitiiEM92n6Dkz60wnG QbK2KrflW5E36NdVwNvFkqR/H3WVCrABBJ6puGHAL3nmlhg/n+MTpyqd7nkg1WSU j1U+hqxN4EHPlTcBUtaeb6DriwaQUIHNMH3h0J8H/UyfvoIRVPA570LF5Cycj7oK zlVgVmsZIZjIzt+qP6xzKkvhXzPLpemIOheDOZO4opgPdHIXGPAI9XVzwrARxMfv KeqyA16XrU6pM7GKvkDEnSDiMye/pPGbq/U3mnYdlRs4Lwn9PvmnzBasSXbrsg6i qeU1v6lSWPx18/9Qq1Qyfjxfgu3SkBpvHypwdv3MNMBk6Y2Gp/pg917FyqfNvoxX l0TaIYYf5xL6bHsxj1uopUoCnl4KxaTAaQ3qYg4+hFO3nzDNTB1k+Xu+pbuePr3U CyaIdmM3sqN7fNIwbfQ1slXrckavl+z/ZZmTZn2zIfa1yeMmP4s= =vkfG -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Client Cert TLS issue
On 15/10/2019 22:15, George Stanchev wrote:
> Hi,
>
> I would need some help with tracking an issue with TC 8.5.47 (windows x64,
> java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple
> client tries to connect to the server, the server hangs on SSL handshake
> until either the client times out on read or the server times out (if I set
> the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I
> have enabled the client side SSL trace and everything goes well until ECDH
> key exchange (for brevity I have enabled only one TLS suite
> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if
> requested. The cacerts we use is whatever comes with Azul's Java distro which
> has ~150 entries + the custom cert for testing. The issue comes from how the
> connector deals with trusted certs because if I reduce the cacerts to contain
> only the test certificate, the request is being served without an issue. Also
> if I remove the tcnative-1.dll from TC there is no issue either.
>
> Perhaps I am missing something. Any help is appreciated.
This sounds like it is repeatable and that you have a system you can test with.
On that basis here are a few things to try:
1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake is
hanging.
2. Try a binary search to try and determine if the issue is the number of
certificates in the trust store or is caused by a specific certificate.
It sounds like there might be an issue converting one or more of the trusted
certs in the trust store to a format OpenSSL can work with.
So the thread dumps didn't prove to be very useful - at least I couldn't see
anything. I attached one of them (dump3.txt). But the trial and error in the
binary searched proved it is not any certificate but a magic number of
certificates in cacerts - 125 hang and 124 work. I don't know if it is a memory
issue or buffer size issue - I tried removing and adding random certs around
the 124/125 boundary and which one is out and in doesn't make a difference. I
realize that a cert is rather large blob so it still could be memory issue with
the size of the cert doesn't matter that much. In
8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms,
tms, sr) we pass the DER-encoded certs to OpenSSL:
// Pass along the DER encoded certificates of the accepted
client
// certificate issuers, so that their subjects can be presented
// by the server during the handshake to allow the client
choosing
// an acceptable certificate
for (X509Certificate caCert :
x509TrustManager.getAcceptedIssuers()) {
SSLContext.addClientCACertificateRaw(ctx,
caCert.getEncoded());
if (log.isDebugEnabled())
log.debug(sm.getString("openssl.addedClientCaCert",
caCert.toString()));
}
And this is not where it hangs. I stepped through the code through the
handshaker but still haven't been able to figure out the hang point as I am not
familiar with the details of that portion of TC code . I've attached two
cacerts that you can plug in as truststore to try out the issue.
George
>
> George
>
> [1]
>
> port="8443" SSLEnabled="true" maxHttpHeaderSize="8192"
> maxThreads="150" acceptCount="100" enableLookups="false"
> disableUploadTimeout="true"
> scheme="https" secure="true" clientAuth="true" sslProtocol="TLS"
> sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> keystoreType="JKS"
> keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit"
> keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
> ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
> TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256,
> TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256,
> TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA,
> TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_
Re: Client Cert TLS issue
On 15/10/2019 22:15, George Stanchev wrote:
> Hi,
>
> I would need some help with tracking an issue with TC 8.5.47 (windows x64,
> java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple
> client tries to connect to the server, the server hangs on SSL handshake
> until either the client times out on read or the server times out (if I set
> the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I
> have enabled the client side SSL trace and everything goes well until ECDH
> key exchange (for brevity I have enabled only one TLS suite
> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if
> requested. The cacerts we use is whatever comes with Azul's Java distro which
> has ~150 entries + the custom cert for testing. The issue comes from how the
> connector deals with trusted certs because if I reduce the cacerts to contain
> only the test certificate, the request is being served without an issue. Also
> if I remove the tcnative-1.dll from TC there is no issue either.
>
> Perhaps I am missing something. Any help is appreciated.
This sounds like it is repeatable and that you have a system you can
test with. On that basis here are a few things to try:
1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS
handshake is hanging.
2. Try a binary search to try and determine if the issue is the number
of certificates in the trust store or is caused by a specific certificate.
It sounds like there might be an issue converting one or more of the
trusted certs in the trust store to a format OpenSSL can work with.
Mark
>
> George
>
> [1]
>
> port="8443" SSLEnabled="true" maxHttpHeaderSize="8192"
> maxThreads="150" acceptCount="100" enableLookups="false"
> disableUploadTimeout="true"
> scheme="https" secure="true" clientAuth="true" sslProtocol="TLS"
> sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> keystoreType="JKS"
> keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit"
> keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
> ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
> TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256,
> TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256,
> TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA,
> TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA,
> TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8,
> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_CCM,
> TLS_ECDHE_ECDSA_WITH_AES_256_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8,
> TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256,
> TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384,
> TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" />
>
>
