Looking at our production environment, I see that we use an absolute path in proxy.config.ssl.CA.cert.filename. Could you try setting that to /ssl/ca.pem? Perhaps a bug came in with the path manipulation.
On Wed, Jan 6, 2021 at 3:01 PM Zack Bartel <[email protected]> wrote: > Thanks for the help! > Yes, the client certificate is directly signed by the certificate in > ca.pem. There is no intermediary. I did not know about the "ssl" debug flag > but this is very helpful. It seems your theory about failing the client > cert verification was correct. > > [Jan 6 08:21:22.635] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=> > (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: > SSL_ERROR_WANT_READ (2), errno=0 > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1139 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1139&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=2e8RrV4SJTDNVqVPg9Jqsd0kS3SFkE-W4WWwQbYSwdE&e=> > (sslServerHandShakeEvent)> (ssl) Go on with the handshake state=5 > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=> > (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: > 8193 ret: 1 State: SSLv3/TLS write server done > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:401 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A401&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Vs4SLGP4rxroDg7Rbt7qkTKLaMA6Ya15ER4leCdjttY&e=> > (ssl_verify_client_callback)> (ssl) Callback: verify client cert > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1603 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1603&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=KJEEYFXd7M33amuR5k3ehukmfIPWBLHNpjXtoK0VPf8&e=> > (callHooks)> (ssl) callHooks sslHandshakeHookState=5 > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1675 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1675&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=7E5pSrM1JfOjHcvAzZcn1oNDErDsn4l9_qZKkXOG3tE&e=> > (callHooks)> (ssl) callHooks iterated to curHook=(nil) > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=> > (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: > 16392 ret: 592 State: error > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:1510 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A1510&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=Qw7t3wWHr5fpoD4lyUqEFXK7FOv3eHdmTYbq_Ppx84g&e=> > (ssl_callback_info)> (ssl) ssl_callback_info ssl: 0x7ff228028ab0 where: > 8194 ret: -1 State: error > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLUtils.cc:2344 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslutils.cc-3A2344&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=bYSr4Zeblv-W2acIqCvJDwdjbv4Yd6hfhMfUKGV__Ls&e=> > (SSLAccept)> (ssl.error.accept) SSL accept returned -1, ssl_error=1, > ERR_get_error=337100934 (error:1417C086:SSL > routines:tls_process_client_certificate:certificate verify failed) > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=> > (sslServerHandShakeEvent)> (ssl-diag) > SSL::140678460933888:error:1417C086:SSL > routines:tls_process_client_certificate:certificate verify > failed:../ssl/statem/statem_srvr.c:3701: peer address is 192.168.1.4 > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1213 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1213&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=NeldsD4eLgkhoyTV-xMLLQXlHwznW8HxcVoSqwV9Xj0&e=> > (sslServerHandShakeEvent)> (ssl-diag) SSL handshake error: SSL_ERROR_SSL > (1), errno=0 > [Jan 6 08:21:22.636] {0x7ff241b10700} DEBUG: <SSLNetVConnection.cc:1339 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__sslnetvconnection.cc-3A1339&d=DwMFAg&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=Ct9BRY8473_NWxRBvSW0ZOHBpyExcu9qgij7MorbZuA&m=eMNTvVzsIr4oMX8X4S6-8igbwFdenlRaU7SzsGbsuN4&s=5R00uQMyJ9YmmMafXRE9xHQzKiK1yEVJf-X57vJ0jGg&e=> > (sslServerHandShakeEvent)> (ssl-diag) > SSLNetVConnection::sslServerHandShakeEvent, SSL_ERROR_SSL errno=0 > > > The certificate is quite old but valid. > Validity > Not Before: Aug 18 14:29:55 2017 GMT > Not After : Aug 16 14:29:55 2027 GMT > > There is nothing noteworthy in diags.log nor any startup errors or > warnings. > > Thanks again, > Zack > > > > On Jan 6, 2021, at 7:38 AM, Susan Hinrichs <[email protected]> > wrote: > > The config looks good to me. There are autests that exercise ATS > requiring certs from the client with a variety of CA verification > configurations. I haven't worked with the 8.x branch, but those tests are > run against that branch I believe. > > Is your client certificate directly signed by the certificate in ca.pem? > Is there an intermediate cert floating around someplace? Assuming the > certs are exchanged as in the working case, it seems likely that the > failure is occurring in the client certificate verification. I assume that > ATS is sending the SSL alert. > > Any interesting start up failure messages in diags.log? > > The next thing to check is turning on debugging and setting the ssl debug > tag. > > On Tue, Jan 5, 2021 at 4:49 PM Zack Bartel <[email protected]> wrote: > >> Hello, >> We are using client -> TS SSL termination and using client certificates >> and a CA. While upgrading to 8.x this has stopped working and I'm having >> trouble figuring out why. I was wondering if there may have been some >> change in 8.x or error with my config which would be obvious to someone. >> >> The error is "SSL alert number 80" internal error. >> >> SSL_connect:SSLv3/TLS read server certificate request >> SSL_connect:SSLv3/TLS read server done >> SSL_connect:SSLv3/TLS write client certificate >> SSL_connect:SSLv3/TLS write client key exchange >> SSL_connect:SSLv3/TLS write certificate verify >> SSL_connect:SSLv3/TLS write change cipher spec >> >> >> SSL_connect:SSLv3/TLS write finished >> read from 0x564393d04170 [0x564393d0e6a3] (5 bytes => 5 (0x5)) >> 0000 - 15 03 03 00 02 ..... >> read from 0x564393d04170 [0x564393d0e6a8] (2 bytes => 2 (0x2)) >> 0000 - 02 50 .P >> >> *SSL3 alert read:fatal:internal error* >> *SSL_connect:error in error* >> *139713745085760:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert >> internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80* >> >> >> >> I've inspected the traffic with Wireshark to confirm the exchange is >> identical between the versions up until the "Internal Error" response. >> Tried up to 7.1.12-rc0 which works and 8.0.0 -> 8.1.x which does not. I've >> also tried on Ubuntu and Centos8 with the same results. >> >> Relevant config: >> >> >> records.config: >> CONFIG proxy.config.ssl.CA.cert.path STRING /ssl/ >> CONFIG proxy.config.ssl.CA.cert.filename STRING ca.pem >> >> CONFIG proxy.config.ssl.client.verify.server INT 1 >> CONFIG proxy.config.ssl.client.certification_level INT 2 >> CONFIG proxy.config.ssl.client.CA.cert.path STRING /etc/pki/tls/certs/ >> CONFIG proxy.config.ssl.client.CA.cert.filename STRING ca-bundle.crt >> >> CONFIG proxy.config.ssl.client.cipher_suite STRING >> TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:DES-CBC3-SHA >> >> CONFIG proxy.config.ssl.server.private_key.path STRING /ssl/ >> CONFIG proxy.config.ssl.server.cert.path STRING /ssl/ >> >> CONFIG proxy.config.ssl.server.cipher_suite STRING >> ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA >> >> ssl_multicert.config: >> ssl_cert_name=server.pem >> >> >> Any help on this greatly appreciated, thank you, >> >> Zack >> > >
