[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Aburadeh updated CASSANDRA-19508:
------------------------------------------
Description:
We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started
seeing thousands of msgs "Failed to get peer certificates for peer
/x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
causing a huge problem for us because cassandra log files are growing very fast
as our connections are short live connections, we open more than 1K connections
per second and they stay live for 1-2 seconds.
{code:java}
DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
ServerConnection.java:140 - Failed to get peer certificates for peer
/172.31.2.23:45796
javax.net.ssl.SSLPeerUnverifiedException: peer not verified
at
io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
at
io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
at
org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
at
org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
at
org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
at
org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
at
org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
at
org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
at
org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
{code}
*Our SSL config:*
{code:java}
client_encryption_options:
enabled: true
keystore: /path/to/keystore
keystore_password: xxxxx
optional: false
require_client_auth: false {code}
We should stop throwing this msg when require_client_auth is set to false. Or
at least it should be logged in TRACE not DEBUG.
I'm working on preparing a PR.
was:
We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started
seeing thousands of msgs "Failed to get peer certificates for peer
/x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
causing a huge problem for us because cassandra log files are growing very fast
as our connections are short live connections, we open more than 1K connections
per second and they stay live for 1-2 seconds.
{code:java}
DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
ServerConnection.java:140 - Failed to get peer certificates for peer
/172.31.2.23:45796
javax.net.ssl.SSLPeerUnverifiedException: peer not verified
at
io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
at
io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
at
org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
at
org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
at
org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
at
org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
at
org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
at
org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
at
org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
{code}
*Our SSL config:*
{code:java}
client_encryption_options:
enabled: true
keystore: /path/to/keystore
keystore_password: xxxxx
optional: false
require_client_auth: false {code}
We should stop throwing this msg when require_client_auth is set to false. Or
at least it should be logged in TRACE not DEBUG.
I'm working on preparing a PR.
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Reporter: Mohammad Aburadeh
> Priority: Urgent
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: xxxxx
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
