Hi, using Kafka 11.0.3 I set up a 2 node Kafka cluster on CentOS 7 with broker id 100 and 101. There is no firewall enabled. Both nodes can ping each other and establish the SSH connection successfully. When I start the cluster both broker successfully register their id into the zookeeper cluster (I checked the path /brokers/ids) and the command to list the topics can be run successfully.
But monitoring brokers logs I see it continuously logs in the file server.log on the broker with id 100 : INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral) WARN Connection to node 101 could not be established. Broker may not be available. (org.apache.kafka.clients.Net$ INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral) Also in the controller.log file on the broker id 100, it logs continuously: INFO [Controller 100]: Broker 100 resigned as the controller (kafka.controller.KafkaController) DEBUG [Controller 100]: Controller resigning, broker id 100 (kafka.controller.KafkaController) DEBUG [Controller 100]: De-registering IsrChangeNotificationListener (kafka.controller.KafkaController) INFO [Partition state machine on Controller 100]: Stopped partition state machine (kafka.controller.PartitionStateMachine) INFO [Replica state machine on controller 100]: Stopped replica state machine (kafka.controller.ReplicaStateMachine) INFO [Controller 100]: Broker 100 resigned as the controller (kafka.controller.KafkaController) INFO [Controller 100]: 100 successfully elected as the controller (kafka.controller.KafkaController) INFO [Controller 100]: Broker 100 starting become controller state transition (kafka.controller.KafkaController) INFO [Controller 100]: Initialized controller epoch to 96100 and zk version 96099 (kafka.controller.KafkaController) INFO [Controller 100]: Controller 100 incremented epoch to 96101 (kafka.controller.KafkaController) DEBUG [Controller 100]: Registering IsrChangeNotificationListener (kafka.controller.KafkaController) DEBUG [Channel manager on controller 100]: Controller 100 trying to connect to broker 100 DEBUG [Channel manager on controller 100]: Controller 100 trying to connect to broker 101 I also check the network packets using tcpdump and I observed Zookeeper nodes could get packets from both brokers successfully, but there isn't any packet has been sent from each broker to the other. I checked the established connections on both brokers using netstat and I observed there are much TIME_WAIT established connections between two brokers. What could be the reason?
