jsancio commented on code in PR #12682: URL: https://github.com/apache/kafka/pull/12682#discussion_r980449119
########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> Review Comment: ```suggestion <p>Kafka servers support listening for connections on multiple ports. This is configured through the <code>listeners</code> property in the server configuration, which accepts a comma-separated list of the listeners to enable. At least one listener must be defined on each server. The format of each listener defined in <code>listeners</code> is given below:</p> ``` ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> + + <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre> + + <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of + the listener. For example, many configurations use a separate listener for client traffic, + so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p + + <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre> + + <p>The security protocol of each listener is defined in a separate configuration: + <code>listener.security.protocol.map</code>. The value is a comma-separated list + of each listener mapped to its security protocol. For example, the follow value + configuration specifies that the <code>CLIENT</code> listener will use SSL while the + <code>BROKER</code> listener will use plaintext.</p> + + <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre> + + <p>Possible options for the security protocol are given below:</p> + <ol> + <li>PLAINTEXT</li> + <li>SSL</li> + <li>SASL_PLAINTEXT</li> + <li>SASL_SSL</li> + </ol> + + <p>The plaintext protocol provides no security and does not require any additional configuration. + In the following sections, this document covers how to configure the remaining protocols.</p> + + <p>If each required listener uses a separate security protocol, it is also possible to use the + security protocol name as the listener name in <code>listeners</code>. Using the example above, + we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners + using the following definition:</p> + + <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre> + + <p>However, we recommend users to provide explicit names for the listeners since it + makes the intended usage clearer.</p> + + <p>Among the listeners in this list, it is possible to declare the listener to be used for + inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration + to the name of the listener. If not defined, then the inter-broker listener is determined + by the security protocol defined by <code>security.inter.broker.protocol</code>, which + defaults to <code>PLAINTEXT</code>.</p> + + <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to + declare a separate listener to be used for metadata propagation from the active controller + to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller + will use this listener when it needs to push metadata updates to the brokers in the cluster. + The benefit of using a control plane listener is that it uses a separate processing thread, + which makes it less likely for application traffic to impede timely propagation of metadata changes + (such as partition leader and ISR updates). Note that the default value is null, which + means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p> + + <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used + exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>. + For legacy clusters, the active controller will connect to all brokers in the cluster + to send metadata updates. In KRaft, it is the other way around. The broker is responsible + for finding the current controller and connecting to it in order to receive metadata updates. + Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p> + + <p>Instead, nodes with the "controller" role must define separate listeners through the + <code>controller.listener.names</code> configuration. Any node running with both the + "broker" and "controller" roles must use separate listeners for the broker and controller. + For example, a minimal configuration for a combined node might look like the following:</p> + + <pre class="line-numbers"><code class="language-text">process.roles=broker,controller +listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093 +inter.broker.listener.name=BROKER +controller.listener.names=CONTROLLER +listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre> + + <p>This defines two listeners, one for the controller and one for the broker. It is conventional + to use a separate listener for client traffic as well.</p> + + <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>. + For outbound requests to other controller nodes, the first listener listed in <code>controller.listener.names</code> + will be used. The list of listeners provides a way to change the active listener from one port + to another through a roll of the cluster. Basically do one roll to expose the new listener, + and one roll to remove the old listener.</p> + + <p>In the following section, this document covers how to enable SSL on a listener for encryption + as well as authentication. The subsequent section will then cover additional authentication + mechanism using SASL.</p> Review Comment: Should we have these relative references "In the following section" and "The subsequent section"? Should we instead use the anchor id to provide a direct reference to them. Feel free to ignore this comment if it is not possible with the current HTML documentation implementation. ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> + + <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre> + + <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of + the listener. For example, many configurations use a separate listener for client traffic, + so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p + + <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre> + + <p>The security protocol of each listener is defined in a separate configuration: + <code>listener.security.protocol.map</code>. The value is a comma-separated list + of each listener mapped to its security protocol. For example, the follow value + configuration specifies that the <code>CLIENT</code> listener will use SSL while the + <code>BROKER</code> listener will use plaintext.</p> + + <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre> + + <p>Possible options for the security protocol are given below:</p> + <ol> + <li>PLAINTEXT</li> + <li>SSL</li> + <li>SASL_PLAINTEXT</li> + <li>SASL_SSL</li> + </ol> + + <p>The plaintext protocol provides no security and does not require any additional configuration. + In the following sections, this document covers how to configure the remaining protocols.</p> + + <p>If each required listener uses a separate security protocol, it is also possible to use the + security protocol name as the listener name in <code>listeners</code>. Using the example above, + we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners + using the following definition:</p> + + <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre> + + <p>However, we recommend users to provide explicit names for the listeners since it + makes the intended usage clearer.</p> + + <p>Among the listeners in this list, it is possible to declare the listener to be used for + inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration + to the name of the listener. If not defined, then the inter-broker listener is determined + by the security protocol defined by <code>security.inter.broker.protocol</code>, which + defaults to <code>PLAINTEXT</code>.</p> + + <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to + declare a separate listener to be used for metadata propagation from the active controller + to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller + will use this listener when it needs to push metadata updates to the brokers in the cluster. + The benefit of using a control plane listener is that it uses a separate processing thread, + which makes it less likely for application traffic to impede timely propagation of metadata changes + (such as partition leader and ISR updates). Note that the default value is null, which + means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p> + + <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used + exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>. + For legacy clusters, the active controller will connect to all brokers in the cluster + to send metadata updates. In KRaft, it is the other way around. The broker is responsible + for finding the current controller and connecting to it in order to receive metadata updates. + Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p> Review Comment: ```suggestion <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used exclusively for requests sent by brokers to brokers. A Kafka server is a broker if the <code>broker</code> role enabled in <code>process.roles</code>. In KRaft, the broker is responsible for finding the current controller and connecting to it in order to receive metadata updates. The broker determines the listener and security protocol based the configuration in <code>controller.listener.names</code> and <code>listener.security.protocol.map</code></p> ``` I think it is easier to read if we document how legacy cluster are configure in one paragraph/section and how kraft clusters are configured in another paragraph/section. I think that comparing the two makes it harder to read and understand. I would argue that at this point the user it trying to configure their cluster correctly. They are less interest in the reason of why the configuration are different. I think they can read the KIPs and discussion for that kind of detail. What do you think? ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to Review Comment: ```suggestion communicate with the servers. Each server must define the set of listeners that are used to ``` ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> + + <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre> + + <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of + the listener. For example, many configurations use a separate listener for client traffic, + so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p + + <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre> + + <p>The security protocol of each listener is defined in a separate configuration: + <code>listener.security.protocol.map</code>. The value is a comma-separated list + of each listener mapped to its security protocol. For example, the follow value + configuration specifies that the <code>CLIENT</code> listener will use SSL while the + <code>BROKER</code> listener will use plaintext.</p> + + <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre> + + <p>Possible options for the security protocol are given below:</p> + <ol> + <li>PLAINTEXT</li> + <li>SSL</li> + <li>SASL_PLAINTEXT</li> + <li>SASL_SSL</li> + </ol> + + <p>The plaintext protocol provides no security and does not require any additional configuration. + In the following sections, this document covers how to configure the remaining protocols.</p> + + <p>If each required listener uses a separate security protocol, it is also possible to use the + security protocol name as the listener name in <code>listeners</code>. Using the example above, + we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners + using the following definition:</p> + + <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre> + + <p>However, we recommend users to provide explicit names for the listeners since it + makes the intended usage clearer.</p> + + <p>Among the listeners in this list, it is possible to declare the listener to be used for + inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration + to the name of the listener. If not defined, then the inter-broker listener is determined + by the security protocol defined by <code>security.inter.broker.protocol</code>, which + defaults to <code>PLAINTEXT</code>.</p> Review Comment: Should we mentioned that inter-broker communication is mainly used for the replication of topic partitions? ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> + + <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre> + + <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of + the listener. For example, many configurations use a separate listener for client traffic, + so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p + + <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre> + + <p>The security protocol of each listener is defined in a separate configuration: + <code>listener.security.protocol.map</code>. The value is a comma-separated list + of each listener mapped to its security protocol. For example, the follow value + configuration specifies that the <code>CLIENT</code> listener will use SSL while the + <code>BROKER</code> listener will use plaintext.</p> + + <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre> + + <p>Possible options for the security protocol are given below:</p> + <ol> + <li>PLAINTEXT</li> + <li>SSL</li> + <li>SASL_PLAINTEXT</li> + <li>SASL_SSL</li> + </ol> + + <p>The plaintext protocol provides no security and does not require any additional configuration. + In the following sections, this document covers how to configure the remaining protocols.</p> + + <p>If each required listener uses a separate security protocol, it is also possible to use the + security protocol name as the listener name in <code>listeners</code>. Using the example above, + we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners + using the following definition:</p> + + <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre> + + <p>However, we recommend users to provide explicit names for the listeners since it + makes the intended usage clearer.</p> Review Comment: I say we remove this if we don't want to recommend this type of configuration. If a user is ready this section to understand an existing configuration maybe we can move this to the end of the section as a note. ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> + + <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre> + + <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of + the listener. For example, many configurations use a separate listener for client traffic, + so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p + + <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre> + + <p>The security protocol of each listener is defined in a separate configuration: + <code>listener.security.protocol.map</code>. The value is a comma-separated list + of each listener mapped to its security protocol. For example, the follow value + configuration specifies that the <code>CLIENT</code> listener will use SSL while the + <code>BROKER</code> listener will use plaintext.</p> + + <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre> + + <p>Possible options for the security protocol are given below:</p> + <ol> + <li>PLAINTEXT</li> + <li>SSL</li> + <li>SASL_PLAINTEXT</li> + <li>SASL_SSL</li> + </ol> + + <p>The plaintext protocol provides no security and does not require any additional configuration. + In the following sections, this document covers how to configure the remaining protocols.</p> + + <p>If each required listener uses a separate security protocol, it is also possible to use the + security protocol name as the listener name in <code>listeners</code>. Using the example above, + we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners + using the following definition:</p> + + <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre> + + <p>However, we recommend users to provide explicit names for the listeners since it + makes the intended usage clearer.</p> + + <p>Among the listeners in this list, it is possible to declare the listener to be used for + inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration + to the name of the listener. If not defined, then the inter-broker listener is determined + by the security protocol defined by <code>security.inter.broker.protocol</code>, which + defaults to <code>PLAINTEXT</code>.</p> + + <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to + declare a separate listener to be used for metadata propagation from the active controller + to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller + will use this listener when it needs to push metadata updates to the brokers in the cluster. + The benefit of using a control plane listener is that it uses a separate processing thread, + which makes it less likely for application traffic to impede timely propagation of metadata changes + (such as partition leader and ISR updates). Note that the default value is null, which + means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p> + + <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used + exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>. + For legacy clusters, the active controller will connect to all brokers in the cluster + to send metadata updates. In KRaft, it is the other way around. The broker is responsible + for finding the current controller and connecting to it in order to receive metadata updates. + Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p> + + <p>Instead, nodes with the "controller" role must define separate listeners through the + <code>controller.listener.names</code> configuration. Any node running with both the + "broker" and "controller" roles must use separate listeners for the broker and controller. + For example, a minimal configuration for a combined node might look like the following:</p> Review Comment: ```suggestion <p>Instead, servers with the <code>controlle</code> role must define separate listeners through the <code>controller.listener.names</code> configuration. Any server running with both the <code>broker</code> and <code>controller</code> roles must use separate listeners for the broker and controller. For example, a minimal configuration for a combined server might look like the following:</p> ``` The broker also needs to the controller listener name so that it knows how which security protocol to use. ########## docs/security.html: ########## @@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a The guides below explain how to configure and use the security features in both clients and brokers. - <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3> + <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3> + + <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to + communicate with the servers. Each node must define the set of listeners that are used to + receive requests from clients as well as other servers. Each listener may be configured + to authenticate clients using various mechanisms and to ensure traffic between the + server and the client is encrypted. This section provides a primer for the configuration + of listeners.</p> + + <p>Kafka brokers support listening for connections on multiple ports. This is configured through + the <code>listeners</code> property in the server configuration, which accepts a comma-separated + list of the listeners to enable. At least one listener must be defined on each node. The format + of each listener defined in <code>listeners</code> is given below:</p> + + <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre> + + <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of + the listener. For example, many configurations use a separate listener for client traffic, + so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p + + <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre> + + <p>The security protocol of each listener is defined in a separate configuration: + <code>listener.security.protocol.map</code>. The value is a comma-separated list + of each listener mapped to its security protocol. For example, the follow value + configuration specifies that the <code>CLIENT</code> listener will use SSL while the + <code>BROKER</code> listener will use plaintext.</p> + + <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre> + + <p>Possible options for the security protocol are given below:</p> + <ol> + <li>PLAINTEXT</li> + <li>SSL</li> + <li>SASL_PLAINTEXT</li> + <li>SASL_SSL</li> + </ol> + + <p>The plaintext protocol provides no security and does not require any additional configuration. + In the following sections, this document covers how to configure the remaining protocols.</p> + + <p>If each required listener uses a separate security protocol, it is also possible to use the + security protocol name as the listener name in <code>listeners</code>. Using the example above, + we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners + using the following definition:</p> + + <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre> + + <p>However, we recommend users to provide explicit names for the listeners since it + makes the intended usage clearer.</p> + + <p>Among the listeners in this list, it is possible to declare the listener to be used for + inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration + to the name of the listener. If not defined, then the inter-broker listener is determined + by the security protocol defined by <code>security.inter.broker.protocol</code>, which + defaults to <code>PLAINTEXT</code>.</p> + + <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to + declare a separate listener to be used for metadata propagation from the active controller + to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller + will use this listener when it needs to push metadata updates to the brokers in the cluster. + The benefit of using a control plane listener is that it uses a separate processing thread, + which makes it less likely for application traffic to impede timely propagation of metadata changes + (such as partition leader and ISR updates). Note that the default value is null, which + means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p> + + <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used + exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>. + For legacy clusters, the active controller will connect to all brokers in the cluster + to send metadata updates. In KRaft, it is the other way around. The broker is responsible + for finding the current controller and connecting to it in order to receive metadata updates. + Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p> + + <p>Instead, nodes with the "controller" role must define separate listeners through the + <code>controller.listener.names</code> configuration. Any node running with both the + "broker" and "controller" roles must use separate listeners for the broker and controller. + For example, a minimal configuration for a combined node might look like the following:</p> + + <pre class="line-numbers"><code class="language-text">process.roles=broker,controller +listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093 +inter.broker.listener.name=BROKER +controller.listener.names=CONTROLLER +listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre> + + <p>This defines two listeners, one for the controller and one for the broker. It is conventional + to use a separate listener for client traffic as well.</p> + + <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>. + For outbound requests to other controller nodes, the first listener listed in <code>controller.listener.names</code> + will be used. The list of listeners provides a way to change the active listener from one port + to another through a roll of the cluster. Basically do one roll to expose the new listener, + and one roll to remove the old listener.</p> Review Comment: ```suggestion <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>. For outbound requests to other controller servers, the first listener listed in <code>controller.listener.names</code> will be used. The list of listeners provides a way to change the active listener from one port to another through a roll of the cluster. Basically do one roll to expose the new listener, and one roll to remove the old listener.</p> ``` Hmm. Regarding the last sentence, maybe we can add a section at the end describing the process of changing the controller listener. What do you think? I think that last sentence is hiding a lot of details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org