[
https://issues.apache.org/jira/browse/KAFKA-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sridhar updated KAFKA-7376:
---------------------------
Description:
Hi ,
We upgraded our Kafka cluster (3x nodes running on AWS cloud) to 2.0.0 version
and enabled security with SASL_SSL (plain) encryption for Inter-broker and
Client connection .
But there are lot of errors in the controller log for the inter-broker
communication .I have the followed exactly same steps as mentioned in the
document and set all kafka brokers fqdn hostname in the SAN
(SubjectAlternativeName) of my server certificate (selfsigned) .
[http://kafka.apache.org/documentation.html#security|http://example.com/]
openssl s_client -connect kafka-3:9093
CONNECTED(00000003)
depth=1
Noticed someone else also facing the similar problem .
[https://github.com/confluentinc/common/issues/158]
{noformat}
Server Configuration :
listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093
advertised.listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093
#Security
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=false
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
super.users=User:admin
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
ssl.truststore.location=/etc/kafka/ssl/kafka.server.truststore.jks
ssl.truststore.password=
ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.jks
ssl.keystore.password=
ssl.key.password=
#Zookeeper
zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
zookeeper.connection.timeout.ms=6000
{noformat}
{code:java}
[2018-09-04 12:02:57,289] WARN [RequestSendThread controllerId=2] Controller
2's connection to broker kafka-3:9093 (id: 3 rack: eu-central-1c) was
unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
at
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
at
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at
org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
at
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
at
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
... 9 more
Caused by: java.security.cert.CertificateException: No name matching kafka-3
found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at
sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at
sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
... 18 more
{code}
was:
Hi ,
We upgraded our Kafka cluster (3x nodes running on AWS cloud) to 2.0.0 version
and enabled security with SASL_SSL (plain) encryption for Inter-broker and
Client connection .
But there are lot of errors in the controller log for the inter-broker
communication .I have the followed exactly same steps as mentioned in the
document and set all kafka brokers fqdn hostname in the SAN
(SubjectAlternativeName) of my server certificate (selfsigned) .
[http://kafka.apache.org/documentation.html#security|http://example.com/]
openssl s_client -connect kafka-3:9093
CONNECTED(00000003)
depth=1
Noticed someone else also facing the similar problem .
[https://github.com/confluentinc/common/issues/158]
{noformat}
Server Configuration :
listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093
advertised.listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093
#Security
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=false
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
super.users=User:admin
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
ssl.truststore.location=/etc/kafka/ssl/kafka.server.truststore.jks
ssl.truststore.password=
ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.jks
ssl.keystore.password=
ssl.key.password=
#Zookeeper
zookeeper.connect=zk-reghub-d2-1:2181,zk-reghub-d2-2:2181,zk-reghub-d2-3:2181
zookeeper.connection.timeout.ms=6000
{noformat}
{code:java}
[2018-09-04 12:02:57,289] WARN [RequestSendThread controllerId=2] Controller
2's connection to broker kafka-3:9093 (id: 3 rack: eu-central-1c) was
unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
at
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
at
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at
org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
at
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at
org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
at
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
... 9 more
Caused by: java.security.cert.CertificateException: No name matching kafka-3
found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at
sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at
sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
... 18 more
{code}
> After Kafka upgrade to v2.0.0 , Controller unable to communicate with brokers
> on SASL_SSL
> -----------------------------------------------------------------------------------------
>
> Key: KAFKA-7376
> URL: https://issues.apache.org/jira/browse/KAFKA-7376
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 2.0.0
> Reporter: Sridhar
> Priority: Major
>
> Hi ,
> We upgraded our Kafka cluster (3x nodes running on AWS cloud) to 2.0.0
> version and enabled security with SASL_SSL (plain) encryption for
> Inter-broker and Client connection .
> But there are lot of errors in the controller log for the inter-broker
> communication .I have the followed exactly same steps as mentioned in the
> document and set all kafka brokers fqdn hostname in the SAN
> (SubjectAlternativeName) of my server certificate (selfsigned) .
> [http://kafka.apache.org/documentation.html#security|http://example.com/]
>
> openssl s_client -connect kafka-3:9093
> CONNECTED(00000003)
> depth=1
> Noticed someone else also facing the similar problem .
> [https://github.com/confluentinc/common/issues/158]
>
>
> {noformat}
> Server Configuration :
> listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093
> advertised.listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093
> #Security
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> allow.everyone.if.no.acl.found=false
> security.inter.broker.protocol=SASL_SSL
> sasl.mechanism.inter.broker.protocol=PLAIN
> sasl.enabled.mechanisms=PLAIN
> super.users=User:admin
> ssl.client.auth=required
> ssl.endpoint.identification.algorithm=
> ssl.truststore.location=/etc/kafka/ssl/kafka.server.truststore.jks
> ssl.truststore.password=
> ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.jks
> ssl.keystore.password=
> ssl.key.password=
> #Zookeeper
> zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181
> zookeeper.connection.timeout.ms=6000
> {noformat}
>
>
> {code:java}
>
> [2018-09-04 12:02:57,289] WARN [RequestSendThread controllerId=2] Controller
> 2's connection to broker kafka-3:9093 (id: 3 rack: eu-central-1c) was
> unsuccessful (kafka.controller.RequestSendThread)
> org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake
> failed
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> at
> org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
> at
> org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
> at
> org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
> at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
> at
> org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
> at
> kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
> at
> kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> at
> org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
> at
> org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
> at
> org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
> ... 9 more
> Caused by: java.security.cert.CertificateException: No name matching kafka-3
> found
> at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231)
> at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
> at
> sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
> at
> sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
> at
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
> at
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
> ... 18 more
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
