[ https://issues.apache.org/jira/browse/KAFKA-12534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345886#comment-17345886 ]
kaushik srinivas edited comment on KAFKA-12534 at 5/17/21, 5:53 AM: -------------------------------------------------------------------- Hi [~cricket007] , We have tried the exact steps. Captured the commands and logs in detail. The scenario to change the keystore password does not work still. sequence of steps to reproduce # install kafka broker by generating a CA, truststore and keystore. (password for stores: 123456) # re generate the keystore with a new password (1234567). Use the same old generated CA and trust store from step1. # issue the dynamic reconfig command after replacing the keystore file in the specified location. # dynamic config command issued: {code:java} ./kafka-configs --bootstrap-server kafkabroker:9092 --command-config ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config 'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore' {code} Note: listener name is ssl and is in the format specified in [https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener] # command fails with following trace {code:java} Error while executing config command with args '--bootstrap-server kafkabroker:9092 --command-config ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore' java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid dynamic configuration at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272) at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338) at kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308) at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85) at kafka.admin.ConfigCommand.main(ConfigCommand.scala) Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid dynamic configuration {code} Kafka broker logs the below output {code:java} { "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 - kafka.server.AdminManager - [Admin Manager on Broker 1001]: Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid dynamic configuration"}} {code} As per docs, the CA is not supposed to be changed and we have maintained that and the CA and trust stores are not changed. Also another observation is that, when for example the country name in the cert generation is changed and the certificate is regenerated, the dynamic config command works fine and we could see the ssl certs being reloaded in the kafka broker logs. But when the keystore password is changed, things have never worked for us even after so many attempts of retries. Can you please help in reproducing this issue and provide some detailed steps if possible for the case where the keystore's password is being changed ? It has clearly never worked for us, even after many attempts. was (Author: kaushik srinivas): Hi, We have tried the exact steps. Captured the commands and logs in detail. The scenario to change the keystore password does not work still. sequence of steps to reproduce # install kafka broker by generating a CA, truststore and keystore. (password for stores: 123456) # re generate the keystore with a new password (1234567). Use the same old generated CA and trust store from step1. # issue the dynamic reconfig command after replacing the keystore file in the specified location. # dynamic config command issued: {code:java} ./kafka-configs --bootstrap-server kafkabroker:9092 --command-config ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config 'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore' {code} Note: listener name is ssl and is in the format specified in [https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener] # command fails with following trace {code:java} Error while executing config command with args '--bootstrap-server kafkabroker:9092 --command-config ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore' java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid dynamic configuration at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272) at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338) at kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308) at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85) at kafka.admin.ConfigCommand.main(ConfigCommand.scala) Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid dynamic configuration {code} Kafka broker logs the below output {code:java} { "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 - kafka.server.AdminManager - [Admin Manager on Broker 1001]: Invalid config value for resource ConfigResource(type=BROKER, name='1001'): Invalid value org.apache.kafka.common.config.ConfigException: Validation of dynamic config update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for configuration Invalid dynamic configuration"}} {code} As per docs, the CA is not supposed to be changed and we have maintained that and the CA and trust stores are not changed. Also another observation is that, when for example the country name in the cert generation is changed and the certificate is regenerated, the dynamic config command works fine and we could see the ssl certs being reloaded in the kafka broker logs. But when the keystore password is changed, things have never worked for us even after so many attempts of retries. Can you please help in reproducing this issue and provide some detailed steps if possible for the case where the keystore's password is being changed ? It has clearly never worked for us, even after many attempts. > kafka-configs does not work with ssl enabled kafka broker. > ---------------------------------------------------------- > > Key: KAFKA-12534 > URL: https://issues.apache.org/jira/browse/KAFKA-12534 > Project: Kafka > Issue Type: Bug > Affects Versions: 2.6.1 > Reporter: kaushik srinivas > Priority: Critical > > We are trying to change the trust store password on the fly using the > kafka-configs script for a ssl enabled kafka broker. > Below is the command used: > kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers > --entity-name 1001 --alter --add-config 'ssl.truststore.password=xxx' > But we see below error in the broker logs when the command is run. > {"type":"log", "host":"kf-2-0", "level":"INFO", > "neid":"kafka-cfd5ccf2af7f47868e83473408", "system":"kafka", > "time":"2021-03-23T12:14:40.055", "timezone":"UTC", > "log":\{"message":"data-plane-kafka-network-thread-1002-ListenerName(SSL)-SSL-2 > - org.apache.kafka.common.network.Selector - [SocketServer brokerId=1002] > Failed authentication with /127.0.0.1 (SSL handshake failed)"}} > How can anyone configure ssl certs for the kafka-configs script and succeed > with the ssl handshake in this case ? > Note : > We are trying with a single listener i.e SSL: -- This message was sent by Atlassian Jira (v8.3.4#803005)