Hello, I’ve been managing my team’s Kafka usage for the past few months. Everything is going fine, but I’ve noticed some strange behavior and would like some clarification. Some of our services use the Kafka Streams API in order to communicate with the cluster. When looking at the logs, it seems the auto-generated Command Configs for them do not indicate that it’s using the TLS settings I added to the Properties file when starting up the services in question.
For background, these are the settings used on the Kafka server to enforce TLS communication: allow.everyone.if.no.acl.found=false authorizer.class.name=kafka.security.authorizer.AclAuthorizer ssl.client.auth=required ssl.enabled.protocols=TLSv1.2 ssl.endpoint.identification.algorithm=https I have the Keystore and Truststore settings also setup with the passwords for those two and the private key set appropriately. Now for the client: props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SSL"); props.put(SslConfigs.SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG, "https"); props.put(SslConfigs.SSL_KEYSTORE_TYPE_CONFIG, "JKS"); props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG, KAFKA_KEYSTORE_LOCATION); props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, KAFKA_KEYSTORE_PASSWORD); props.put(SslConfigs.SSL_TRUSTSTORE_TYPE_CONFIG, "JKS"); props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, KAFKA_TRUSTSTORE_LOCATION); props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, KAFKA_TRUSTSTORE_PASSWORD); props.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, KAFKA_KEY_PASSWORD); Each of the values that isn’t hardcoded is an environment variable I inject from a Docker image. I’ve confirmed that all of these environment variables are set correctly and that the correct X.509 certificates are loaded in the Keystore and Truststore. However, looking at the logs of this service, I see: ConsumerConfig values: ... security.protocol = PLAINTEXT ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS So the automatically generated Command Configs for the Consumer and Producer (this is shown in the ProducerConfig section as well) have the Truststore and Keystore set to null along with their passwords, the security protocol being used is PLAINTEXT rather than SSL, etc. The odd thing is, this service is still able to communicate with the Kafka server even though the server requires TLS communication and uses client authentication in order to be identified (for later ACL authorizations). The Command Configs should not work from this generated state, but they are somehow communicating with the server. Any help regarding this generated Command Config for the Kafka Streams API would be much appreciated. Matthew Rabey Staff Software Engineer [Logo Description automatically generated] LinQuest Corporation Space Systems Engineering and Integration
