Just following up, I realized I forgot to add some information. This is using kafka 3.5.1,
I am in the process of setting up a kafka cluster which is configured to > use KRaft. There is a set of three controller nodes and a set of six > brokers. Both the controllers and the brokers are configured to use mTLS > (Mutual TLS). So the part of the controller config looks like: > > listeners=CONTROLLER://:9097 > listener.security.protocol.map=CONTROLLER:SSL > controller.listener.names=CONTROLLER > > Now the certificates initially were missing a SAN that corresponded to the > fqdn of the nodes. The fqdn was used in creating the controller quorum > voters config. > > When the controllers started up I did not see any errors or issues. When > the brokers started up I saw a couple of SSL connection errors when it > tried to connect to the controllers, giving the controller hostname was > missing from the SAN of the certificate. But the whole cluster seemed to > function normally. No other errors and everything was in sync. And the > kafka-metadata-quorum.sh ... describe --status showed the correct status of > the controllers and the brokers. > I saw some errors such as: 2024-04-16 20:05:57,924 [broker-4-to-controller-forwarding-channel-manager] ERROR (kafka.server.BrokerToControllerRequestThread) - [broker-4-to-controller-forwarding-channel-manager]: Request EnvelopeRequestData(requestData=java.nio.HeapByteBuffer[pos=0 lim=169 cap=169], requestPrincipal=[0, 0, 5, 85, 115, 101, 114, 23, 97, 106, 110, 97, 108, 111, 99, 97, 108, 116, 101, 115, 116, 49, 46, 97, 106, 110, 97, 107, 109, 102, 0, 0], clientHostAddress=[10, 16, -125, -115]) failed due to authentication error with controller 2024-04-16 20:05:57,924 [broker-4-to-controller-forwarding-channel-manager] ERROR (kafka.server.BrokerToControllerRequestThread) - [broker-4-to-controller-forwarding-channel-manager]: Failed to send the following request due to authentication error: ClientRequest(expectResponse=true, callback=kafka.server.BrokerToControllerRequestThread$$Lambda$695/0x00000007c0703440@35746167, destination=900, correlationId=7, clientId=4, createdTimeMs=1713297957917, requestBuilder=EnvelopeRequestData(requestData=java.nio.HeapByteBuffer[pos=0 lim=169 cap=169], requestPrincipal=[0, 0, 5, 85, 115, 101, 114, 23, 97, 106, 110, 97, 108, 111, 99, 97, 108, 116, 101, 115, 116, 49, 46, 97, 106, 110, 97, 107, 109, 102, 0, 0], clientHostAddress=[10, 16, -125, -115])) 2024-04-16 20:05:57,923 [broker-4-to-controller-forwarding-channel-manager] ERROR (org.apache.kafka.clients.NetworkClient) - [BrokerToControllerChannelManager id=4 name=forwarding] Connection to node 900 (kafka-controller-0.kafka-controller-internal-svc.ajnalocaltest1.svc.cluster.local/ 10.16.135.173:9097) failed authentication due to: SSL handshake failed I fixed the SAN in the cert and the errors went away on the brokers. > > My question is if the certs prevented the SSL connection from being > established between the brokers and the controllers, or even between the > controllers, is there some fallback that was used? PLAINTEXT or was some > of the validation skipped? > Thanks! ttyl Dima