Hi Ben, Thanks for your answer. What if the instance does not have a public DNS hostname? These are all private nodes without public/elastic IP, therefore I don't know what to set.
Marco 2016-06-01 15:09 GMT+02:00 Ben Davison <[email protected]>: > Hi Marco, > > We use the public DNS hostname that you can get from the AWS metadata > service. > > Thanks, > > Ben > > On Wed, Jun 1, 2016 at 1:54 PM, Marco B. <[email protected]> wrote: > > > Hello everyone, > > > > I am trying to setup a MirrorMaker between my company's local cluster and > > another cluster in AWS to have replication over clusters. We have setup a > > VPN between these two clusters, and as far as I can see, everything works > > correctly, meaning that I can ping the nodes and telnet into them without > > any issues. > > > > Now, when I run the following command in the local cluster to use a > > Zookeeper instance located in AWS (10.1.83.6:2181), in order to read a > > topic "test" > > > > ~/kafka_2.11-0.8.2.2$ ./bin/kafka-console-consumer.sh --zookeeper > > 10.1.83.6:2181 --topic test --from-beginning > > > > A bunch of errors comes up: > > > > WARN Fetching topic metadata with correlation id 1 for topics [Set(test)] > > from broker [id:2,host:ip-10-1-83-5.ec2.internal,port:9092] failed > > (kafka.client.ClientUtils$) > > java.nio.channels.ClosedChannelException > > at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) > > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) > > at > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) > > at kafka.producer.SyncProducer.send(SyncProducer.scala:113) > > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) > > at > > > > > kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) > > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) > > WARN Fetching topic metadata with correlation id 1 for topics [Set(test)] > > from broker [id:3,host:ip-10-1-83-6.ec2.internal,port:9092] failed > > (kafka.client.ClientUtils$) > > > > As far as I know, this is due to the fact that Zookeeper has registered > an > > IP/Port for each Kafka instance and these need to be consistent with the > > producer configuration, as described here ( > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ). > > > > I tried to search on the web, and some people were recommending to change > > the setting "advertised.host.name" to be either the public IP address > > coming from AWS (we cannot) or a specific hostname. Now, considering that > > we have a VPN between the clusters, the only choice left seems to be the > > one setting the hostname. > > > > What should this value be? Is there anything else I need to know for this > > kind of setup? Any suggestions? > > > > Thanks in advance. > > > > Kind regards, > > Marco > > > > -- > > > This email, including attachments, is private and confidential. If you have > received this email in error please notify the sender and delete it from > your system. Emails are not secure and may contain viruses. No liability > can be accepted for viruses that might be transferred by this email or any > attachment. Any unauthorised copying of this message or unauthorised > distribution and publication of the information contained herein are > prohibited. > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. > Registered in England and Wales. Registered No. 04843573. >
