I apologize, you are correct. I forgot that you can pass in a list of hosts. Using a balancer is the way we handle it here, mostly because we have many producers, and they are not all under our direct control.
To answer the question of what load balancer, we have hardware load balancers in place in our environment. I'm not certain of the model, as I generally don't interface with them directly (that's the purview of the network operations team). -Todd On 3/12/14 1:47 PM, "Cassa L" <[email protected]> wrote: >Hi Todd, > I am not very clear about you mentioned - "Both the producer and >consumer >clients will take care of rebalancing if >there is a broker failure." " >For the producer side, you do need some sort of front end to handle the >metadata requests, or else you would constantly have to change your >producer configuration if brokers fail." > > Isn't broker client API needs to take care of collecting metadata based >on >the list of brokers that I pass in producer API? Why would I need load >balancer just to get metadata? > >Regards, >LCass > > >On Wed, Mar 12, 2014 at 2:39 AM, Atig <[email protected]> wrote: > >> Hello >> >> Can you elaborate on what is your producer setup in terms of fetching >>the >> metadata (in conjunction with a load balancer) and some sort of code >> skeleton? New to Kafka here. >> >> Also, if you don't mind sharing what do you use as load balancer in this >> scenario? >> >> Thanks >> >> --- Original Message --- >> >> From: "Todd Palino" <[email protected]> >> Sent: 12 March 2014 07:11 >> To: [email protected] >> Subject: Re: Kafka Cluster management >> >> Both the producer and consumer clients will take care of rebalancing if >> there is a broker failure. The consumer handles this entirely in >>Zookeeper. >> >> For the producer side, you do need some sort of front end to handle the >> metadata requests, or else you would constantly have to change your >> producer configuration if brokers fail. In our case, we use a load >> balancer for this because it's the most robust solution. Only the >>metadata >> request itself will go through the load balancer. Once the producer has >> the metadata for the topic and partition, it reconnects directly to the >> leader for the partition for producing. >> >> -Todd >> >> On 3/11/14 10:07 PM, "Cassa L" <[email protected]> wrote: >> >> >Hi, >> > Is there documentation on how connection management, connection >>pooling >> >works in Kafka for multiple brokers? Do I need to take care of my own >> >connection management for cluster? e.g. while publishing messages, if >>one >> >broker stops responding, I need to switch to the other. Does Kafka >>client >> >takes care of this or do I need load balancer between brokers and >>client? >> > >> >Thanks, >> >LCassa >> >>
