Maria Isabel Florez Rodriguez created KAFKA-13372: -----------------------------------------------------
Summary: failed authentication due to: SSL handshake failed Key: KAFKA-13372 URL: https://issues.apache.org/jira/browse/KAFKA-13372 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 2.2.2 Reporter: Maria Isabel Florez Rodriguez Hi everyone, I have the next issue about authentication SCRAM + SSL. I’m using the CLI and this is the version of my client (./kafka_2.13-2.8.1/bin/kafka-topics.sh). In this example I will talk about list topics, but another operations (consumer, producer) failed too. First, let me describe the current scenario: * I have 5 Kafka servers with * kafka-broker-0.mydomain.com * kafka-broker-1.mydomain.com * kafka-broker-2.mydomain.com * kafka-broker-3.mydomain.com * kafka-broker-4.mydomain.com * I have a DNS principal configured with Round Robin to IPs broker: * kafka-broker-princial.mydomain.com (Round Robin) I have configured for each broker the next listeners (I'm using 3 ports): {quote}advertised.listeners=SASL_SSL://kafka-broker-0.mydomain.com:9094,SASL_PLAINTEXT://kafka-broker-0.mydomain.com:9093,PLAINTEXT://kafka-broker-0.mydomain.com:9092{quote} * 9092 for PLAINTEXT * 9093 for SASL_PLAINTEXT * 9094 for SASL_SSL My Kafka broker servers have the next config server.properties: {quote}advertised.listeners=SASL_SSL://kafka-broker-X.mydomain.com:9094,SASL_PLAINTEXT://kafka-broker-X.mydomain.com:9093,PLAINTEXT://kafka-broker-X.mydomain.com:9092 authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer auto.create.topics.enable=false auto.leader.rebalance.enable=true background.threads=10 broker.id=X broker.rack=us-east-1c compression.type=producer connections.max.idle.ms=2700000 controlled.shutdown.enable=true delete.topic.enable=true host.name=localhost leader.imbalance.check.interval.seconds=300 leader.imbalance.per.broker.percentage=10 listeners=SASL_SSL://0.0.0.0:9094,SASL_PLAINTEXT://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092 log.cleaner.enable=true log.dirs=/var/lib/kafka/log/data1,/var/lib/kafka/log/data2,/var/lib/kafka/log/data3 log.retention.check.interval.ms=300000 log.retention.hours=336 log.segment.bytes=1073741824 message.max.bytes=1000012 min.insync.replicas=2 num.io.threads=8 num.network.threads=3 num.partitions=3 num.recovery.threads.per.data.dir=1 num.replica.fetchers=1 offset.metadata.max.bytes=4096 offsets.commit.timeout.ms=5000 offsets.retention.minutes=129600 offsets.topic.num.partitions=50 offsets.topic.replication.factor=3 port=9092 queued.max.requests=500 replica.fetch.min.bytes=1 replica.fetch.wait.max.ms=500 sasl.enabled.mechanisms=SCRAM-SHA-256,GSSAPI sasl.kerberos.service.name=xxxxx sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 security.inter.broker.protocol=SASL_SSL socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 socket.send.buffer.bytes=102400 ssl.client.auth=required {{ssl.endpoint.identification.algorithm=""}} ssl.enabled.protocols=TLSv1.2 ssl.key.password=xxxx ssl.keystore.location=/etc/ssl/default_keystore.jks ssl.keystore.password=xxxxxxxx ssl.truststore.location=/usr/lib/jvm/java-11-adoptopenjdk-hotspot/lib/security/cacerts ssl.truststore.password= xxxxxxxx ssl.truststore.type=JKS super.users=User:xxxxx zookeeper.connect=kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com:2181,kafka-zk-X.mydomain.com :2181,kafka-zk-X.mydomain.com:218/my-environment zookeeper.connection.timeout.ms=6000 zookeeper.sasl.client=false{quote} I was trying the next things: * (/)*PLAINTEXT:* I can consume directly to broker to broker with port *9092* (Using IP or dns broker) * (/)*PLAINTEXT:* I also can consume directly to DNS principal configured with Round Robin with port *9092* (Using DNS principal) * (/)*SASL_SSL:* I can consume directly to broker to broker with port *9094* (Using only dns broker due it needs to validate the certificate) * (x)*SASL_SSL:* I cannot consume directly to DNS principal configured with Round Robin with port *9094* The issue is: * *(x)SASL_SSL(x):* I cannot consume directly to DNS principal configured with Round Robin with port *9094*. Only I have the issue with I try to connect directly to DNS principal. My certificates contains permissions with all my subdomains under the domain. * I have the next _file.config_ when that I use when I try to connect to DNS principal. (Is the same file that I used for consume directly to broker to broker with port 9094) {quote}# Required connection configs for Kafka producer, consumer, and admin{quote} {quote}ssl.keystore.location=/My/Path/default_keystore.jks ssl.keystore.password=xxxxx ssl.truststore.location=/My/Path/cacerts ssl.truststore.password= xxxxx ssl.truststore.type=JKS ssl.enabled.protocols=TLSv1.2 security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=‘xxxxx' password=‘xxxxxx'; client.dns.lookup=use_all_dns_ips{quote} The command that I'm using to try consume directly principal kafka DNS: {quote}$ ./kafka_2.13-2.8.1/bin/kafka-topics.sh --bootstrap-server kafka-broker-princial.mydomain.com:9094 --command-config java9094.config --list [2021-10-13 01:04:58,206] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-broker-princial.mydomain.com/10.110.209.136:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2021-10-13 01:04:58,207] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found. at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found. at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212) at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103) at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:412) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632) ... 19 more Error while executing topic command : SSL handshake failed [2021-10-13 01:04:58,212] ERROR org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found. at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kafka-broker-princial.mydomain.com found. at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212) at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103) at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:412) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632) ... 19 more (kafka.admin.TopicCommand$){quote} Can you help me with this issue? Thanks for reading me! @maisfloro -- This message was sent by Atlassian Jira (v8.3.4#803005)