Re: Unable to start broker only config in Kraft mode (Kafka 3.0.0)
Hi Vamsee, > To summarize, if a minimal broker only and controller only config examples can be added to the Kraft README they would be awesome 👍 Thanks for your suggestion. Actually, there are 2 example configuration files in Kraft config folder: https://github.com/apache/kafka/blob/trunk/config/kraft/broker.properties https://github.com/apache/kafka/blob/trunk/config/kraft/controller.properties FYR. Also, I found the description for the `controller.listener.names` in the example properties file is wrong and misleading. (thanks to your question :) ) I've submitted a PR to update it. Welcome to provide feedback. PR: https://github.com/apache/kafka/pull/11616 Thank you. Luke On Sun, Dec 19, 2021 at 2:10 AM Vamsee Lakamsani wrote: > Hi Luke, > > Thanks for your response but I don't understand. > As this node is not a controller I first tried commenting out the > 'controller.listener.names' setting altogether. It doesn't like it. So I > tried setting it to PLAINTEXT in order to match the PLANTEXT:PLAINTEXT that > I set for `listener.security.protocol.map' in the config I posted in my > initial email. Sorry the formatting didn't come through well in that one. > So here they are again better formatted. > > > ### > > process.roles=broker > > node.id=6 > > controller.quorum.voters=1@1.2.3.1:9093,2@ 1.2.3.2:9093,3@ 1.2.3.3:9093,4@ > 1.2.3.4:9093,5@ 1.2.3.5:9093 > > controller.listener.names=PLAINTEXT > > listeners=PLAINTEXT://1.2.3.6:9092 > > inter.broker.listener.name=PLAINTEXT > > > listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL > > > > Given that this node's role is only a broker I am wondering why we need to > set > 'controller.listener.names' at all in it's config file? Isn't > 'controller.quorum.voters' pointing to the other controller nodes not > sufficient? I tried commenting out 'controller.listener.names' altogether > on this node but it fails to start with another exception. > > To summarize, if a minimal broker only and controller only config examples > can be added to the Kraft README they would be awesome 👍 > > > On Sat, Dec 18, 2021 at 4:59, Luke Chen > wrote: > Hi Vamsee, > > From the error stack trace, it is complaining the listener name: set in > `controller.listener.names` cannot find the mapping security protocol. Ex: > In > https://github.com/apache/kafka/blob/trunk/config/kraft/broker.properties, > we set the `controller.listener.names=CONTROLLER`, and in the ` > listener.security.protocol.map=CONTROLLER:PLAINTEXT`, we know it is > mapped to `PLAINTEXT`. So it can find the security protocol successfully. > > Please try to check and update the 2 settings in the config file. > > Hope it helps. > > Thank you. > Luke > > > > > On Tue, Dec 14, 2021 at 12:48 PM Vamsee Lakamsani > wrote: > > Hi, I have a 6 node cluster with Kafka 3.0.0 with 5 nodes running in dual > mode (controller and broker) using instructions from here: > https://github.com/apache/kafka/blob/3.0/config/kraft/README.md. They are > running fine. > I set up the 6th one as just a broker to experiment. It fails to startup > with the error. > > [2021-12-14 04:38:16,085] ERROR Exiting Kafka due to fatal exception > (kafka.Kafka$)java.lang.IllegalArgumentException: No enum constant > org.apache.kafka.common.security.auth.SecurityProtocol. at > java.base/java.lang.Enum.valueOf(Enum.java:240) at > org.apache.kafka.common.security.auth.SecurityProtocol.valueOf(SecurityProtocol.java:26) > at > org.apache.kafka.common.security.auth.SecurityProtocol.forName(SecurityProtocol.java:72) > at > kafka.raft.KafkaRaftManager.$anonfun$buildNetworkClient$1(RaftManager.scala:230) > at scala.collection.immutable.Map$Map4.getOrElse(Map.scala:530) at > kafka.raft.KafkaRaftManager.buildNetworkClient(RaftManager.scala:230) at > kafka.raft.KafkaRaftManager.buildNetworkChannel(RaftManager.scala:208) at > kafka.raft.KafkaRaftManager.(RaftManager.scala:124) at > kafka.server.KafkaRaftServer.(KafkaRaftServer.scala:73) at > kafka.Kafka$.buildServer(Kafka.scala:79) at > kafka.Kafka$.main(Kafka.scala:87) at kafka.Kafka.main(Kafka.scala) > Its config/kraft/server.properties has relevant config like this: > === > # The role of this server. Setting this puts us in KRaft > modeprocess.roles=broker > # The node id associated with this instance's rolesnode.id=6 > # The connect string for the controller quorum (these IPs are annymized > for privacy)controller.quorum.voters=1@1.2.3.1:9093,2@ 1.2.3.2:9093,3@ > 1.2.3.3:9093,4@ 1.2.3.4:9093,5@ 1.2.3.5:9093 > # Socket Server Settings > # > # The address the socket server listens on. It will get the value returned > from# java.net.InetAddress.getCanonicalHostName() if not configured.# > FORMAT:# listeners = listener_name://host_name:port# EXAMPLE:# > listeners = PLAINTEXT://your.host.name > :9092listeners=PLAINTEXT://1.2.3.6:9092inter.broker.listener.name > =PLAINTEXT > # H
Re: [ANNOUNCE] New Kafka PMC member: David Jacot
Congrats! On 12/17/21 15:46, Bill Bejeck wrote: Congratulations David! Well deserved. -Bill On Fri, Dec 17, 2021 at 6:43 PM José Armando García Sancio wrote: Congrats David! On Fri, Dec 17, 2021 at 3:09 PM Gwen Shapira wrote: Hi everyone, David Jacot has been an Apache Kafka committer since Oct 2020 and has been contributing to the community consistently this entire time - especially notable the fact that he reviewed around 150 PRs in the last year. It is my pleasure to announce that David agreed to join the Kafka PMC. Congratulations, David! Gwen Shapira, on behalf of Apache Kafka PMC -- -Jose
Re: Unable to start broker only config in Kraft mode (Kafka 3.0.0)
Hi Luke, Thanks for your response but I don't understand.As this node is not a controller I first tried commenting out the 'controller.listener.names' setting altogether. It doesn't like it. So I tried setting it to PLAINTEXT in order to match the PLANTEXT:PLAINTEXT that I set for `listener.security.protocol.map' in the config I posted in my initial email. Sorry the formatting didn't come through well in that one. So here they are again better formatted. ### process.roles=broker node.id=6 controller.quorum.voters=1@1.2.3.1:9093,2@ 1.2.3.2:9093,3@ 1.2.3.3:9093,4@ 1.2.3.4:9093,5@ 1.2.3.5:9093 controller.listener.names=PLAINTEXT listeners=PLAINTEXT://1.2.3.6:9092 inter.broker.listener.name=PLAINTEXT listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL Given that this node's role is only a broker I am wondering why we need to set 'controller.listener.names' at all in it's config file? Isn't 'controller.quorum.voters' pointing to the other controller nodes not sufficient? I tried commenting out 'controller.listener.names' altogether on this node but it fails to start with another exception. To summarize, if a minimal broker only and controller only config examples can be added to the Kraft README they would be awesome 👍 On Sat, Dec 18, 2021 at 4:59, Luke Chen wrote: Hi Vamsee, >From the error stack trace, it is complaining the listener name: set in >`controller.listener.names` cannot find the mapping security protocol. Ex: In https://github.com/apache/kafka/blob/trunk/config/kraft/broker.properties, we set the `controller.listener.names=CONTROLLER`, and in the `listener.security.protocol.map=CONTROLLER:PLAINTEXT`, we know it is mapped to `PLAINTEXT`. So it can find the security protocol successfully. Please try to check and update the 2 settings in the config file. Hope it helps. Thank you.Luke On Tue, Dec 14, 2021 at 12:48 PM Vamsee Lakamsani wrote: Hi, I have a 6 node cluster with Kafka 3.0.0 with 5 nodes running in dual mode (controller and broker) using instructions from here: https://github.com/apache/kafka/blob/3.0/config/kraft/README.md. They are running fine. I set up the 6th one as just a broker to experiment. It fails to startup with the error. [2021-12-14 04:38:16,085] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)java.lang.IllegalArgumentException: No enum constant org.apache.kafka.common.security.auth.SecurityProtocol. at java.base/java.lang.Enum.valueOf(Enum.java:240) at org.apache.kafka.common.security.auth.SecurityProtocol.valueOf(SecurityProtocol.java:26) at org.apache.kafka.common.security.auth.SecurityProtocol.forName(SecurityProtocol.java:72) at kafka.raft.KafkaRaftManager.$anonfun$buildNetworkClient$1(RaftManager.scala:230) at scala.collection.immutable.Map$Map4.getOrElse(Map.scala:530) at kafka.raft.KafkaRaftManager.buildNetworkClient(RaftManager.scala:230) at kafka.raft.KafkaRaftManager.buildNetworkChannel(RaftManager.scala:208) at kafka.raft.KafkaRaftManager.(RaftManager.scala:124) at kafka.server.KafkaRaftServer.(KafkaRaftServer.scala:73) at kafka.Kafka$.buildServer(Kafka.scala:79) at kafka.Kafka$.main(Kafka.scala:87) at kafka.Kafka.main(Kafka.scala) Its config/kraft/server.properties has relevant config like this: === # The role of this server. Setting this puts us in KRaft modeprocess.roles=broker # The node id associated with this instance's rolesnode.id=6 # The connect string for the controller quorum (these IPs are annymized for privacy)controller.quorum.voters=1@1.2.3.1:9093,2@ 1.2.3.2:9093,3@ 1.2.3.3:9093,4@ 1.2.3.4:9093,5@ 1.2.3.5:9093 # Socket Server Settings # # The address the socket server listens on. It will get the value returned from# java.net.InetAddress.getCanonicalHostName() if not configured.# FORMAT:# listeners = listener_name://host_name:port# EXAMPLE:# listeners = PLAINTEXT://your.host.name:9092listeners=PLAINTEXT://1.2.3.6:9092inter.broker.listener.name=PLAINTEXT # Hostname and port the broker will advertise to producers and consumers. If not set,# it uses the value for "listeners" if configured. Otherwise, it will use the value# returned from java.net.InetAddress.getCanonicalHostName().#advertised.listeners=PLAINTEXT://1.2.3.6:9092 # Listener, host name, and port for the controller to advertise to the brokers. If# this server is a controller, this listener must be configured.#controller.listener.names=PLAINTEXT # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more detailslistener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL ==Thoughts?
Re: Unable to start broker only config in Kraft mode (Kafka 3.0.0)
Hi Vamsee, >From the error stack trace, it is complaining the listener name: set in `controller.listener.names` cannot find the mapping security protocol. Ex: In https://github.com/apache/kafka/blob/trunk/config/kraft/broker.properties, we set the `controller.listener.names=CONTROLLER`, and in the ` listener.security.protocol.map=CONTROLLER:PLAINTEXT`, we know it is mapped to `PLAINTEXT`. So it can find the security protocol successfully. Please try to check and update the 2 settings in the config file. Hope it helps. Thank you. Luke On Tue, Dec 14, 2021 at 12:48 PM Vamsee Lakamsani wrote: > Hi, I have a 6 node cluster with Kafka 3.0.0 with 5 nodes running in dual > mode (controller and broker) using instructions from here: > https://github.com/apache/kafka/blob/3.0/config/kraft/README.md. They are > running fine. > I set up the 6th one as just a broker to experiment. It fails to startup > with the error. > > [2021-12-14 04:38:16,085] ERROR Exiting Kafka due to fatal exception > (kafka.Kafka$)java.lang.IllegalArgumentException: No enum constant > org.apache.kafka.common.security.auth.SecurityProtocol. at > java.base/java.lang.Enum.valueOf(Enum.java:240) at > org.apache.kafka.common.security.auth.SecurityProtocol.valueOf(SecurityProtocol.java:26) > at > org.apache.kafka.common.security.auth.SecurityProtocol.forName(SecurityProtocol.java:72) > at > kafka.raft.KafkaRaftManager.$anonfun$buildNetworkClient$1(RaftManager.scala:230) > at scala.collection.immutable.Map$Map4.getOrElse(Map.scala:530) at > kafka.raft.KafkaRaftManager.buildNetworkClient(RaftManager.scala:230) at > kafka.raft.KafkaRaftManager.buildNetworkChannel(RaftManager.scala:208) at > kafka.raft.KafkaRaftManager.(RaftManager.scala:124) at > kafka.server.KafkaRaftServer.(KafkaRaftServer.scala:73) at > kafka.Kafka$.buildServer(Kafka.scala:79) at > kafka.Kafka$.main(Kafka.scala:87) at kafka.Kafka.main(Kafka.scala) > Its config/kraft/server.properties has relevant config like this: > === > # The role of this server. Setting this puts us in KRaft > modeprocess.roles=broker > # The node id associated with this instance's rolesnode.id=6 > # The connect string for the controller quorum (these IPs are annymized > for privacy)controller.quorum.voters=1@1.2.3.1:9093,2@ 1.2.3.2:9093,3@ > 1.2.3.3:9093,4@ 1.2.3.4:9093,5@ 1.2.3.5:9093 > # Socket Server Settings > # > # The address the socket server listens on. It will get the value returned > from# java.net.InetAddress.getCanonicalHostName() if not configured.# > FORMAT:# listeners = listener_name://host_name:port# EXAMPLE:# > listeners = PLAINTEXT://your.host.name > :9092listeners=PLAINTEXT://1.2.3.6:9092inter.broker.listener.name > =PLAINTEXT > # Hostname and port the broker will advertise to producers and consumers. > If not set,# it uses the value for "listeners" if configured. Otherwise, > it will use the value# returned from > java.net.InetAddress.getCanonicalHostName().#advertised.listeners=PLAINTEXT:// > 1.2.3.6:9092 > # Listener, host name, and port for the controller to advertise to the > brokers. If# this server is a controller, this listener must be > configured.#controller.listener.names=PLAINTEXT > # Maps listener names to security protocols, the default is for them to be > the same. See the config documentation for more > detailslistener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL > ==Thoughts?
