<< My goal is to be able to hide the underlying hosts from the producer and be able to use a virtual endpoint (like the vip).
That is a typical approach and good to be done if you can for the reasons you brought up. The first call of a producer is to "bootstrap" itself with the list of kafka brokers (through a metadatarequest) and their current ownership. More on that here https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Partitioningandbootstrapping if you are interested. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Wed, Jan 21, 2015 at 1:44 PM, Sekhar2, Sinu <sinu_sekh...@intuit.com> wrote: > Hello Kafka Users > > I am new to Kafka and am evaluating it for using it for our high > throughput messaging needs. > > I tested my producers using two different approaches. > (a) metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 > And > (b) metadata.broker.list=vip:9092 > Where vip is an ELB (elastic load balancer in AWS) pointing to kafka1, > kafka2 and kafka3 > > partitions=3,replication=3 for the topic that I am using. > > Both the approaches seem to work from a producer standpoint and I am able > to consume these messages fine as well. > > My goal is to be able to hide the underlying hosts from the producer and > be able to use a virtual endpoint (like the vip). > But I am not sure how the clients will know which node is hosting a > specific partition. If my number of partitions are less than the number of > nodes, then there is a chance that the request can end up in a node that is > not hosting the partition as master….what happens in that case? > > Has anyone implemented Kafka with a virtual endpoint using ELB, or some > other load balancer like Nginx in AWS? > What are the recommendations to hide the underlying node information from > the kafka clients when it comes to producers OR even consumers? > > Any guidance will be extremely helpful! > > Cheers! > Sinu >
