Status of KIP-687 and best practices for X.509 certificates refreshing

Kafka 4.0 here. I thought that KIP-687 was available but "ssl.keystore.location.refresh.interval.ms" seems to be silently ignored and not certificate reloading is done. KIP-687 is marked as "

How to change the IP address of a kafka controller

Hello All. I am running Kafka 3.9.0 in kRaft mode with three controllers. I need to relocate one of the three controllers to a new IP. I can move its data, nothings is lost. Only the IP will change. I am using a hardcoded "controller.quorum.voters". What would be the procedure to move one of

Kafka "kafka-metadata-quorum.sh" regression in 3.9.0

In Kafka 3.8.1 I see this: """ /home/kafka/bin/kafka-metadata-quorum.sh --command-config /home/kafka-broker-data/command.properties --bootstrap-server [HIDDEN]:9092 describe --status ClusterId: 8a31cmC7Tn-IHxEDnIfQoA LeaderId: 1001 LeaderEpoch:117580 High

Re: [ANNOUNCE] Apache Kafka 3.8.1

On 30/10/24 18:15, Jesus Cea wrote: On 29/10/24 15:23, Josep Prat wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.8.1 This is a bug fix release and it includes fixes and improvements. All of the changes in this release can be found in the release notes

Re: [ANNOUNCE] Apache Kafka 3.8.1

On 30/10/24 18:18, Jesus Cea wrote: On 30/10/24 18:15, Jesus Cea wrote: On 29/10/24 15:23, Josep Prat wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.8.1 This is a bug fix release and it includes fixes and improvements. All of the changes in this

Re: [ANNOUNCE] Apache Kafka 3.8.1

On 29/10/24 15:23, Josep Prat wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.8.1 This is a bug fix release and it includes fixes and improvements. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.8

Re: How to get a X509 broker certificate with "openssl s_client"?

On 9/11/23 3:37, Jesus Cea wrote: I am trying to remotely access to the brokers certificates (for audit purposes, expiration alarms, etc) using this command: """ openssl s_client -showcerts -connect localhost:9092 """ The connection is correctly established, b

How do you know you are at the end of the partition?

I wonder how I can know I have processed all the messages in a partition?. What I currently do is: 1. Publish a "ping" message with a "nonce", 2. read the partition (for the last know processed offset) replacing the application state every time I get a new state message in the topic, 3. when I

Re: PANIC: Unable to recover the cluster after all the controllers in KRaft mode were dead at the same time

On 1/12/23 20:42, Jesus Cea wrote: I use SASL_SSL. The controller credentials are "wired" in the configuration, so no "metadata recovery watermark" knowledge should be necessary: """ listener.name.controller.sasl.ena

PANIC: Unable to recover the cluster after all the controllers in KRaft mode were dead at the same time

Kafka 3.6.0. I have a KRaft cluster with three quorum servers. A power failure killed all the controllers at the same time. After rebooting, the controllers can not connect to each other. So, the cluster is down. Log: """ [...] [2023-12-01 20:29:24,931] INFO [MetadataLoader id=1000] initial

Re: How to dynamically change configurations in the controllers

On 16/11/23 8:41, Luke Chen wrote: Hi Jesus, KIP-919 is what you're looking for: https://cwiki.apache.org/confluence/display/KAFKA/KIP-919%3A+Allow+AdminClient+to+Talk+Directly+with+the+KRaft+Controller+Quorum+and+add+Controller+Registration This feature will be included in next release (i.e. K

How to dynamically change configurations in the controllers

Kafka 3.6.0. The tool "kafka-configs.sh" is able to read and change configuration in the brokers, but I am unable to read/change configurations in the Kraft controllers. How is that done? I am interested, for instance, in being able to update the TLS certificates. Help! Thanks. -- Jesús Ce

Re: Example dynamic TLS certificates reconfiguration

On 11/11/23 2:30, Jesus Cea wrote: I am trying to use "ssl.keystore.certificate.chain" and "ssl.keystore.key" in my brokers' configuration in order to be able to use dynamic recompilation for short TLS certificates expiration. No luck so far. I have found my mis

Example dynamic TLS certificates reconfiguration

I am trying to use "ssl.keystore.certificate.chain" and "ssl.keystore.key" in my brokers' configuration in order to be able to use dynamic recompilation for short TLS certificates expiration. No luck so far. I have been unable to find a complete example anywhere. My current configuration is t

How to get a X509 broker certificate with "openssl s_client"?

I am trying to remotely access to the brokers certificates (for audit purposes, expiration alarms, etc) using this command: """ openssl s_client -showcerts -connect localhost:9092 """ The connection is correctly established, but something is wrong. The TLS session is has some errors at the beg

Re: NO SCRAM-SASL AUTHENTICATION BETWEEN KRAFT CONTROL NODES

On 3/11/23 16:45, Richard Bosch wrote: I haven't worked with KRaft controllers in SASL mode yet, but could I think that the early.start.listeners might help here. ( https://kafka.apache.org/documentation/#brokerconfigs_early.start.listeners) It was meant to indicate that these listeners are depen

Re: NO SCRAM-SASL AUTHENTICATION BETWEEN KRAFT CONTROL NODES

What version of Kafka are you using? I am testing 3.6.0. Do you have an account to open an issue at https://issues.apache.org/jira/projects/KAFKA/issues/? Thanks -- Jesús Cea Avión _/_/ _/_/_/_/_/_/ j...@jcea.es - https://www.jcea.es/_/_/_/_/ _/_/

Re: NO SCRAM-SASL AUTHENTICATION BETWEEN KRAFT CONTROL NODES

On 3/11/23 11:25, Daniele Carminati wrote: But, when i try to enable SCRAM/SASL on controller -> controller i get this error [2023-11-03 09:05:02,134] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Failed authentication with /192.168.1.34 (channelId=192.168.1.33:9093-192.168.1.34:56006-130

Example configuration for kraft controllers with SASL_PLAINTEXT

Hi, there. I have a working 3-node kafka kraft mode network. Everything works fine with no authentication. I am using new Kafka 3.6. The node_id for the kraft controllers are "1000", "1001" and "1002". There is a regular kafka broker with node_id "1". I am trying to move that controller conf

About X.509 CRL behaviour

I am trying to deploy a brand new Kafka cluster and trying to do it "the right way". I am thinking about TLS everywhere, with my own private CA. I wonder about the CRL management. I know I can enable CRL in Kafka via KAFKA_OPTS: "-Dcom.sun.security.enableCRLDP=true -Dcom.sun.net.ssl.checkRevo