[ https://issues.apache.org/jira/browse/KAFKA-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sridhar updated KAFKA-7376: --------------------------- Description: Hi , We upgraded our Kafka cluster (3x nodes running on AWS cloud) to 2.0.0 version and enabled security with SASL_SSL (plain) encryption for Inter-broker and Client connection . But there are lot of errors in the controller log for the inter-broker communication .I have the followed exactly same steps as mentioned in the document and set all kafka brokers fqdn hostname in the SAN (SubjectAlternativeName) of my server certificate (selfsigned) . [http://kafka.apache.org/documentation.html#security|http://example.com/] openssl s_client -connect kafka-3:9093 CONNECTED(00000003) depth=1 Noticed someone else also facing the similar problem . [https://github.com/confluentinc/common/issues/158] {noformat} Server Configuration : listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093 advertised.listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093 #Security authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer allow.everyone.if.no.acl.found=false security.inter.broker.protocol=SASL_SSL sasl.mechanism.inter.broker.protocol=PLAIN sasl.enabled.mechanisms=PLAIN super.users=User:admin ssl.client.auth=required ssl.endpoint.identification.algorithm= ssl.truststore.location=/etc/kafka/ssl/kafka.server.truststore.jks ssl.truststore.password= ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.jks ssl.keystore.password= ssl.key.password= #Zookeeper zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181 zookeeper.connection.timeout.ms=6000 {noformat} {code:java} [2018-09-04 12:02:57,289] WARN [RequestSendThread controllerId=2] Controller 2's connection to broker kafka-3:9093 (id: 3 rack: eu-central-1c) was unsuccessful (kafka.controller.RequestSendThread) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487) at org.apache.kafka.common.network.Selector.poll(Selector.java:425) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331) ... 9 more Caused by: java.security.cert.CertificateException: No name matching kafka-3 found at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231) at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ... 18 more {code} was: Hi , We upgraded our Kafka cluster (3x nodes running on AWS cloud) to 2.0.0 version and enabled security with SASL_SSL (plain) encryption for Inter-broker and Client connection . But there are lot of errors in the controller log for the inter-broker communication .I have the followed exactly same steps as mentioned in the document and set all kafka brokers fqdn hostname in the SAN (SubjectAlternativeName) of my server certificate (selfsigned) . [http://kafka.apache.org/documentation.html#security|http://example.com/] openssl s_client -connect kafka-3:9093 CONNECTED(00000003) depth=1 Noticed someone else also facing the similar problem . [https://github.com/confluentinc/common/issues/158] {noformat} Server Configuration : listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093 advertised.listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093 #Security authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer allow.everyone.if.no.acl.found=false security.inter.broker.protocol=SASL_SSL sasl.mechanism.inter.broker.protocol=PLAIN sasl.enabled.mechanisms=PLAIN super.users=User:admin ssl.client.auth=required ssl.endpoint.identification.algorithm= ssl.truststore.location=/etc/kafka/ssl/kafka.server.truststore.jks ssl.truststore.password= ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.jks ssl.keystore.password= ssl.key.password= #Zookeeper zookeeper.connect=zk-reghub-d2-1:2181,zk-reghub-d2-2:2181,zk-reghub-d2-3:2181 zookeeper.connection.timeout.ms=6000 {noformat} {code:java} [2018-09-04 12:02:57,289] WARN [RequestSendThread controllerId=2] Controller 2's connection to broker kafka-3:9093 (id: 3 rack: eu-central-1c) was unsuccessful (kafka.controller.RequestSendThread) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487) at org.apache.kafka.common.network.Selector.poll(Selector.java:425) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510) at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73) at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331) ... 9 more Caused by: java.security.cert.CertificateException: No name matching kafka-3 found at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231) at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ... 18 more {code} > After Kafka upgrade to v2.0.0 , Controller unable to communicate with brokers > on SASL_SSL > ----------------------------------------------------------------------------------------- > > Key: KAFKA-7376 > URL: https://issues.apache.org/jira/browse/KAFKA-7376 > Project: Kafka > Issue Type: Bug > Components: controller > Affects Versions: 2.0.0 > Reporter: Sridhar > Priority: Major > > Hi , > We upgraded our Kafka cluster (3x nodes running on AWS cloud) to 2.0.0 > version and enabled security with SASL_SSL (plain) encryption for > Inter-broker and Client connection . > But there are lot of errors in the controller log for the inter-broker > communication .I have the followed exactly same steps as mentioned in the > document and set all kafka brokers fqdn hostname in the SAN > (SubjectAlternativeName) of my server certificate (selfsigned) . > [http://kafka.apache.org/documentation.html#security|http://example.com/] > > openssl s_client -connect kafka-3:9093 > CONNECTED(00000003) > depth=1 > Noticed someone else also facing the similar problem . > [https://github.com/confluentinc/common/issues/158] > > > {noformat} > Server Configuration : > listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093 > advertised.listeners=PLAINTEXT://kafka-3:9092,SASL_SSL://kafka-3:9093 > #Security > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer > allow.everyone.if.no.acl.found=false > security.inter.broker.protocol=SASL_SSL > sasl.mechanism.inter.broker.protocol=PLAIN > sasl.enabled.mechanisms=PLAIN > super.users=User:admin > ssl.client.auth=required > ssl.endpoint.identification.algorithm= > ssl.truststore.location=/etc/kafka/ssl/kafka.server.truststore.jks > ssl.truststore.password= > ssl.keystore.location=/etc/kafka/ssl/kafka.server.keystore.jks > ssl.keystore.password= > ssl.key.password= > #Zookeeper > zookeeper.connect=zk-1:2181,zk-2:2181,zk-3:2181 > zookeeper.connection.timeout.ms=6000 > {noformat} > > > {code:java} > > [2018-09-04 12:02:57,289] WARN [RequestSendThread controllerId=2] Controller > 2's connection to broker kafka-3:9093 (id: 3 rack: eu-central-1c) was > unsuccessful (kafka.controller.RequestSendThread) > org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake > failed > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem > at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) > at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) > at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214) > at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186) > at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) > at > org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439) > at > org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304) > at > org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258) > at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134) > at > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487) > at org.apache.kafka.common.network.Selector.poll(Selector.java:425) > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510) > at > org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73) > at > kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) > at > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) > at java.security.AccessController.doPrivileged(Native Method) > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) > at > org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393) > at > org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473) > at > org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331) > ... 9 more > Caused by: java.security.cert.CertificateException: No name matching kafka-3 > found > at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:231) > at sun.security.util.HostnameChecker.match(HostnameChecker.java:96) > at > sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455) > at > sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436) > at > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252) > at > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) > at > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) > ... 18 more > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005)