Hello everyone,
I am trying to setup a MirrorMaker between my company's local cluster and
another cluster in AWS to have replication over clusters. We have setup a
VPN between these two clusters, and as far as I can see, everything works
correctly, meaning that I can ping the nodes and telnet into them without
any issues.
Now, when I run the following command in the local cluster to use a
Zookeeper instance located in AWS (10.1.83.6:2181), in order to read a
topic "test"
~/kafka_2.11-0.8.2.2$ ./bin/kafka-console-consumer.sh --zookeeper
10.1.83.6:2181 --topic test --from-beginning
A bunch of errors comes up:
WARN Fetching topic metadata with correlation id 1 for topics [Set(test)]
from broker [id:2,host:ip-10-1-83-5.ec2.internal,port:9092] failed
(kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
WARN Fetching topic metadata with correlation id 1 for topics [Set(test)]
from broker [id:3,host:ip-10-1-83-6.ec2.internal,port:9092] failed
(kafka.client.ClientUtils$)
As far as I know, this is due to the fact that Zookeeper has registered an
IP/Port for each Kafka instance and these need to be consistent with the
producer configuration, as described here (
https://cwiki.apache.org/confluence/display/KAFKA/FAQ).
I tried to search on the web, and some people were recommending to change
the setting "advertised.host.name" to be either the public IP address
coming from AWS (we cannot) or a specific hostname. Now, considering that
we have a VPN between the clusters, the only choice left seems to be the
one setting the hostname.
What should this value be? Is there anything else I need to know for this
kind of setup? Any suggestions?
Thanks in advance.
Kind regards,
Marco
