Hello Kafka Users I am new to Kafka and am evaluating it for using it for our high throughput messaging needs.
I tested my producers using two different approaches. (a) metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 And (b) metadata.broker.list=vip:9092 Where vip is an ELB (elastic load balancer in AWS) pointing to kafka1, kafka2 and kafka3 partitions=3,replication=3 for the topic that I am using. Both the approaches seem to work from a producer standpoint and I am able to consume these messages fine as well. My goal is to be able to hide the underlying hosts from the producer and be able to use a virtual endpoint (like the vip). But I am not sure how the clients will know which node is hosting a specific partition. If my number of partitions are less than the number of nodes, then there is a chance that the request can end up in a node that is not hosting the partition as master….what happens in that case? Has anyone implemented Kafka with a virtual endpoint using ELB, or some other load balancer like Nginx in AWS? What are the recommendations to hide the underlying node information from the kafka clients when it comes to producers OR even consumers? Any guidance will be extremely helpful! Cheers! Sinu
