[ 
https://issues.apache.org/jira/browse/KAFKA-12534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345886#comment-17345886
 ] 

kaushik srinivas edited comment on KAFKA-12534 at 5/17/21, 5:52 AM:
--------------------------------------------------------------------

Hi,

We have tried the exact steps. Captured the commands and logs in detail. The 
scenario to change the keystore password does not work still. 

sequence of steps to reproduce
 # install kafka broker by generating a CA, truststore and keystore. (password 
for stores: 123456)
 # re generate the keystore with a new password (1234567). Use the same old 
generated CA and trust store from step1.
 # issue the dynamic reconfig command after replacing the keystore file in the 
specified location.
 # dynamic config command issued: 
{code:java}
./kafka-configs --bootstrap-server kafkabroker:9092 --command-config 
ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config 
'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
{code}
Note: listener name is ssl and is in the format specified in 
[https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener]

 # command fails with following trace 
{code:java}
Error while executing config command with args '--bootstrap-server 
kafkabroker:9092 --command-config ssl.properties --entity-type brokers 
--entity-name 1001 --alter --add-config 
listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value 
for resource ConfigResource(type=BROKER, name='1001'): Invalid value 
org.apache.kafka.common.config.ConfigException: Validation of dynamic config 
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to 
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for 
configuration Invalid dynamic configuration
        at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
        at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
        at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338)
        at 
kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid 
config value for resource ConfigResource(type=BROKER, name='1001'): Invalid 
value org.apache.kafka.common.config.ConfigException: Validation of dynamic 
config update of SSLFactory failed: org.apache.kafka.common.KafkaException: 
Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for 
configuration Invalid dynamic configuration

{code}
Kafka broker logs the below output 

 
{code:java}
{ "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 - 
kafka.server.AdminManager - [Admin Manager on Broker 1001]: Invalid config 
value for resource ConfigResource(type=BROKER, name='1001'): Invalid value 
org.apache.kafka.common.config.ConfigException: Validation of dynamic config 
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to 
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for 
configuration Invalid dynamic configuration"}} {code}

 

As per docs, the CA is not supposed to be changed and we have maintained that 
and the CA and trust stores are not changed. Also another observation is that, 
when for example the country name in the cert generation is changed and the 
certificate is regenerated, the dynamic config command works fine and we could 
see the ssl certs being reloaded in the kafka broker logs.

But when the keystore password is changed, things have never worked for us even 
after so many attempts of retries. Can you please help in reproducing this 
issue and provide some detailed steps if possible for the case where the 
keystore's password is being changed ? It has clearly never worked for us, even 
after many attempts.


was (Author: kaushik srinivas):
Hi,

We have tried the exact steps. Captured the commands and logs in detail. The 
scenario to change the keystore password does not work still. 

sequence of steps to reproduce
 # install kafka broker by generating a CA, truststore and keystore. (password 
for stores: 123456)
 # re generate the keystore with a new password (1234567). Use the same old 
generated CA and trust store from step1.
 # issue the dynamic reconfig command after replacing the keystore file in the 
specified location.
 # dynamic config command issued: 
{code:java}
./kafka-configs --bootstrap-server kafkabroker:9092 --command-config 
ssl.properties --entity-type brokers --entity-name 1001 --alter --add-config 
'listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
{code}
Note: listener name is ssl and is in the format specified in 
[https://docs.confluent.io/platform/current/kafka/dynamic-config.html#updating-ssl-keystore-of-an-existing-listener]
 # command fails with following trace 
{code:java}
Error while executing config command with args '--bootstrap-server 
kafkabroker:9092 --command-config ssl.properties --entity-type brokers 
--entity-name 1001 --alter --add-config 
listener.name.ssl.ssl.keystore.password=1234567,listener.name.ssl.ssl.keystore.location=/etc/kafka/secrets/ssl/keyStore'
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.InvalidRequestException: Invalid config value 
for resource ConfigResource(type=BROKER, name='1001'): Invalid value 
org.apache.kafka.common.config.ConfigException: Validation of dynamic config 
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to 
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for 
configuration Invalid dynamic configuration
        at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
        at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
        at kafka.admin.ConfigCommand$.alterBrokerConfig(ConfigCommand.scala:338)
        at 
kafka.admin.ConfigCommand$.processBrokerConfig(ConfigCommand.scala:308)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:85)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid 
config value for resource ConfigResource(type=BROKER, name='1001'): Invalid 
value org.apache.kafka.common.config.ConfigException: Validation of dynamic 
config update of SSLFactory failed: org.apache.kafka.common.KafkaException: 
Failed to load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for 
configuration Invalid dynamic configuration

{code}
Kafka broker logs the below output 
{code:java}
{ "timezone":"UTC", "log":{"message":"data-plane-kafka-request-handler-5 - 
kafka.server.AdminManager - [Admin Manager on Broker 1001]: Invalid config 
value for resource ConfigResource(type=BROKER, name='1001'): Invalid value 
org.apache.kafka.common.config.ConfigException: Validation of dynamic config 
update of SSLFactory failed: org.apache.kafka.common.KafkaException: Failed to 
load SSL keystore /etc/kafka/secrets//ssl/keyStore of type JKS for 
configuration Invalid dynamic configuration"}}
{code}
{code:java}
 {code}

> kafka-configs does not work with ssl enabled kafka broker.
> ----------------------------------------------------------
>
>                 Key: KAFKA-12534
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12534
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 2.6.1
>            Reporter: kaushik srinivas
>            Priority: Critical
>
> We are trying to change the trust store password on the fly using the 
> kafka-configs script for a ssl enabled kafka broker.
> Below is the command used:
> kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers 
> --entity-name 1001 --alter --add-config 'ssl.truststore.password=xxx'
> But we see below error in the broker logs when the command is run.
> {"type":"log", "host":"kf-2-0", "level":"INFO", 
> "neid":"kafka-cfd5ccf2af7f47868e83473408", "system":"kafka", 
> "time":"2021-03-23T12:14:40.055", "timezone":"UTC", 
> "log":\{"message":"data-plane-kafka-network-thread-1002-ListenerName(SSL)-SSL-2
>  - org.apache.kafka.common.network.Selector - [SocketServer brokerId=1002] 
> Failed authentication with /127.0.0.1 (SSL handshake failed)"}}
>  How can anyone configure ssl certs for the kafka-configs script and succeed 
> with the ssl handshake in this case ? 
> Note : 
> We are trying with a single listener i.e SSL: 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to