[
https://issues.apache.org/jira/browse/KAFKA-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096321#comment-18096321
]
Aditya Mohanty edited comment on KAFKA-15513 at 7/14/26 10:23 PM:
------------------------------------------------------------------
Hello Team,
A bit late to the party, but good news - we were able to achieve both broker
and controller communication over the SASL/SCRAM mechanism using a hybrid
approach.
{*}The core idea (tl;dr) is{*}: Do not provide the SCRAM admin principal during
storage formatting. Start with multiple listeners for broker and controller
communication(both PLAINTEXT and SCRAM). Add the SCRAM admin principal via
`kafka-configs.sh`. Retire the PLAINTEXT listener.
----
For the example, I will be using SASL_PLAINTEXT as the communication protocol
with the SCRAM mechanism, though the same would work with SASL_SSL as well.
*Step 1:*
Do not provide any credentials while formatting the storage using
kafka-storage.sh.
*Step 2:*
Start the servers (brokers and controllers) with multiple listeners at the same
time, for example, PLAINTEXT and SASL_PLAINTEXT.
Use PLAINTEXT as the protocol for inter-broker and controller communication,
which can be set via the value of the config `inter.broker.listener.name` and
the order of values in the config `controller.listener.names`.
Keeping a few of the configs of interest:
{code:java}
listeners=PLAINTEXT://broker0.example.com:9092,CONTROLLER://broker0.example.com:9093,BROKER_SECURE://broker0.example.com:9094,CONTROLLER_SECURE://broker0.example.com:9095
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER,CONTROLLER_SECURE
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER_SECURE:SASL_PLAINTEXT,BROKER_SECURE:SASL_PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-256
allow.everyone.if.no.acl.found=true
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:<place your scram admin user>; {code}
*Step 3:*
Now, by making a request to the PLAINTEXT port, add the SCRAM admin principal
via `kafka-configs.sh`, exactly similar to how we add other SCRAM user
principals.
* Here is the catch: Kafka allows adding SCRAM credentials via the PLAINTEXT
port as well, probably due to the config `sasl.enabled.mechanisms` needed in
the previous step.
* And the issue between quorum formation and admin credential in cluster
metadata , as explained by [~pprovenzano] in the previous comments, will not
occur here, as the controller communication is already established and running
via the PLAINTEXT protocol. The addition of this SCRAM admin principal will get
replicated across other controllers.
*Step 4:*
Perform a rolling restart of the servers to switch `inter.broker.listener.name`
and `controller.listener.names` to use the SASL_PLAINTEXT protocol.
Keeping a few of the configs of interest:
{code:java}
inter.broker.listener.name=BROKER_SECURE
controller.listener.names=CONTROLLER_SECURE,CONTROLLER
sasl.mechanism.controller.protocol=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN {code}
*Step 5:*
Now, the PLAINTEXT ports can be retired. Perform a rolling restart of the
servers to remove the PLAINTEXT listeners. And this config calue can be set as
`allow.everyone.if.no.acl.found=false`
Keeping a few of the configs of interest:
{code:java}
listeners=BROKER_SECURE://broker0.example.com:9094,CONTROLLER_SECURE://broker0.example.com:9095
listener.security.protocol.map=CONTROLLER_SECURE:SASL_PLAINTEXT,BROKER_SECURE:SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false {code}
was (Author: JIRAUSER313791):
Hello Team,
A bit late to the party, but good news - we were able to achieve both broker
and controller communication over the SASL/SCRAM mechanism using a hybrid
approach.
{*}The core idea (tl;dr) is{*}: Do not provide the SCRAM admin principal during
storage formatting. Start with multiple listeners for broker and controller
communication(both PLAINTEXT and SCRAM). Add the SCRAM admin principal via
`kafka-configs.sh`. Retire the PLAINTEXT listener.
----
For the example, I will be using SASL_PLAINTEXT as the communication protocol
with the SCRAM mechanism, though the same would work with SASL_SSL as well.
*Step 1:*
- Do not provide any credentials while formatting the storage using
kafka-storage.sh.
*Step 2:*
Start the servers (brokers and controllers) with multiple listeners at the same
time, for example, PLAINTEXT and SASL_PLAINTEXT.
Use PLAINTEXT as the protocol for inter-broker and controller communication,
which can be set via the value of the config `inter.broker.listener.name` and
the order of values in the config `controller.listener.names`.
Keeping a few of the configs of interest:
{code:java}
listeners=PLAINTEXT://broker0.example.com:9092,CONTROLLER://broker0.example.com:9093,BROKER_SECURE://broker0.example.com:9094,CONTROLLER_SECURE://broker0.example.com:9095
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER,CONTROLLER_SECURE
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,CONTROLLER_SECURE:SASL_PLAINTEXT,BROKER_SECURE:SASL_PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-256
allow.everyone.if.no.acl.found=true
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:<place your scram admin user>; {code}
*Step 3:*
Now, by making a request to the PLAINTEXT port, add the SCRAM admin principal
via `kafka-configs.sh`, exactly similar to how we add other SCRAM user
principals. * Here is the catch: Kafka allows adding SCRAM credentials via the
PLAINTEXT port as well, probably due to the config `sasl.enabled.mechanisms`
needed in the previous step.
* And the issue between quorum formation and admin credential in cluster
metadata , as explained by [~pprovenzano] in the previous comments, will not
occur here, as the controller communication is already established and running
via the PLAINTEXT protocol. The addition of this SCRAM admin principal will get
replicated across other controllers.
*Step 4:*
Perform a rolling restart of the servers to switch `inter.broker.listener.name`
and `controller.listener.names` to use the SASL_PLAINTEXT protocol.
Keeping a few of the configs of interest:
{code:java}
inter.broker.listener.name=BROKER_SECURE
controller.listener.names=CONTROLLER_SECURE,CONTROLLER
sasl.mechanism.controller.protocol=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN {code}
*Step 5:*
Now, the PLAINTEXT ports can be retired. Perform a rolling restart of the
servers to remove the PLAINTEXT listeners. And this config calue can be set as
`allow.everyone.if.no.acl.found=false`
Keeping a few of the configs of interest:
{code:java}
listeners=BROKER_SECURE://broker0.example.com:9094,CONTROLLER_SECURE://broker0.example.com:9095
listener.security.protocol.map=CONTROLLER_SECURE:SASL_PLAINTEXT,BROKER_SECURE:SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false {code}
> KRaft cluster fails with SCRAM authentication enabled for control-plane
> -----------------------------------------------------------------------
>
> Key: KAFKA-15513
> URL: https://issues.apache.org/jira/browse/KAFKA-15513
> Project: Kafka
> Issue Type: Bug
> Components: kraft
> Affects Versions: 3.5.1, 3.6.0
> Reporter: migruiz4
> Priority: Major
>
> We have observed a scenario where a KRaft cluster fails to bootstrap when
> using SCRAM authentication for controller-to-controller communications.
> The steps to reproduce are simple:
> * Deploy (at least) 2 Kafka servers using latest version 3.5.1.
> * Configure a KRaft cluster, where the controller listener uses
> SASL_PLAINTEXT + SCRAM-SHA-256 or SCRAM-SHA-512. In my case, I'm using the
> recommended in-line jaas config
> '{{{}listener.name.<controller-name>.scram-sha-512.sasl.jaas.config{}}}'
> * Run 'kafka-storage.sh' in both nodes using option '--add-scram' to create
> the SCRAM user.
> When initialized, Controllers will fail to connect to each other with an
> authentication error:
>
> {code:java}
> [2023-08-01 11:12:45,295] ERROR [kafka-1-raft-outbound-request-thread]:
> Failed to send the following request due to authentication error:
> ClientRequest(expectResponse=true,
> callback=kafka.raft.KafkaNetworkChannel$$Lambda$687/0x00007f27d443fc60@2aba6075,
> destination=0, correlationId=129, clientId=raft-client-1,
> createdTimeMs=1690888364960,
> requestBuilder=VoteRequestData(clusterId='abcdefghijklmnopqrstug',
> topics=[TopicData(topicName='__cluster_metadata',
> partitions=[PartitionData(partitionIndex=0, candidateEpoch=4, candidateId=1,
> lastOffsetEpoch=0, lastOffset=0)])])) (kafka.raft.RaftSendThread) {code}
> Some additional details about the scenario that we tested out:
> * Controller listener does work when configured with SASL+PLAIN
> * The issue only affects the Controller listener, SCRAM users created using
> the same method work for data-plane listeners and inter-broker listeners.
>
> Below you can find the exact configuration and command used to deploy:
> * server.properties
> {code:java}
> listeners=INTERNAL://:9092,CLIENT://:9091,CONTROLLER://:9093
> advertised.listeners=INTERNAL://kafka-0:9092,CLIENT://:9091
> listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/bitnami/kafka/data
> num.partitions=1
> num.recovery.threads.per.data.dir=1
> offsets.topic.replication.factor=1
> transaction.state.log.replication.factor=1
> transaction.state.log.min.isr=1
> log.retention.hours=168
> log.retention.check.interval.ms=300000
> controller.listener.names=CONTROLLER
> controller.quorum.voters=0@kafka-0:9093,1@kafka-1:9093
> inter.broker.listener.name=INTERNAL
> node.id=0
> process.roles=controller,broker
> sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
> sasl.mechanism.controller.protocol=SCRAM-SHA-512
> listener.name.controller.sasl.enabled.mechanisms=SCRAM-SHA-512
> listener.name.controller.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
> required username="controller_user" password="controller_password";{code}
> * kafka-storage.sh command
> {code:java}
> kafka-storage.sh format --config /path/to/server.properties
> --ignore-formatted --cluster-id abcdefghijklmnopqrstuv --add-scram
> SCRAM-SHA-512=[name=controller_user,password=controller_password] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
