Hi Chuck,
Are you running zookeepers in the same containers as Kafka brokers?
Kafka brokers should be able to communicate with any of the zookeepers
and, more importantly, zookeepers need to be able to talk to each-other.
Therefore, the zookeeper port should be exposed too (2181 by default),
otherwise you're not going to have a valid cluster.
Docs on the controller are here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals
Let me know if that helped.
Cheers,
Michał
On 27/04/17 19:08, Chuck Musser wrote:
I'm running into a problem with a 3 broker cluster where,
intermittently, one of the broker's controller begins to report that
it cannot connect to the other brokers and repeatedly logs the
failure.
Each broker is running in its own Docker container on separate
machines. These Docker containers have exposed 9092, which I think is
sufficient for operation, but not sure.
The log message are these:
[2017-04-27 17:16:28,985] WARN [Controller-3-to-broker-2-send-thread],
Controller 3's connection to broker 64174aa85d04:9092 (id: 2 rack: null) was
unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 64174aa85d04:9092 (id: 2 rack: null) failed
at
kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at
kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
at
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-04-27 17:16:28,986] WARN [Controller-3-to-broker-1-send-thread],
Controller 3's connection to broker d4b8943ad4b5:9092 (id: 1 rack: null) was
unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to d4b8943ad4b5:9092 (id: 1 rack: null) failed
at
kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:84)
at
kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:94)
at
kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:232)
at
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:185)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:184)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
This is Kafka 2.12-0.10.2.0. I'm wondering:
1. How do we figure out the cause of the connect failures?
2. What's the controller anyway?
3. Are there some command-line diagnostic tools for inspecting the health of
the system?
Thanks for any help,
Chuck
--
Signature
<http://www.openbet.com/> Michal Borowiecki
Senior Software Engineer L4
T: +44 208 742 1600
+44 203 249 8448
E: michal.borowie...@openbet.com
W: www.openbet.com <http://www.openbet.com/>
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
UK
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmas...@openbet.com <mailto:postmas...@openbet.com> and delete it
from your system as well as any copies. The content of e-mails as well
as traffic data may be monitored by OpenBet for employment and security
purposes. To protect the environment please do not print this e-mail
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company
registered in England and Wales. Registered no. 3134634. VAT no.
GB927523612
