Re: Is there a way to pull out kafka metadata from zookeeper?

What I suggested was *not* to get the metadata from the zookeeper, but simply the broker list, to avoid having a list brokers on the configuration. This seems reasonable for installations that don't have access to a VIP. One option is to support broker list and zookeeper and get the broker list fr

Re: Is there a way to pull out kafka metadata from zookeeper?

What I understood from Neha is that querying the metadata from one (any) kafka will return everything in one go. Querying the data indirectly via zookeeper would be more complicated and would involve more requests between the zookeeper and the brokers before being able to answer back. What I su

Re: Is there a way to pull out kafka metadata from zookeeper?

Hello Siyuan, Pre 0.8 Kafka does have a zookeeper based producer api, and this has been removed in 0.8. You can find some of the reasons in this JIRA: https://issues.apache.org/jira/browse/KAFKA-369 Guozhang On Fri, Oct 11, 2013 at 5:16 PM, hsy...@gmail.com wrote: > That's why I'm asking, I

Re: Is there a way to pull out kafka metadata from zookeeper?

That's why I'm asking, I would like to see a kafka zookeeper client api to get TopicMetadata instead of my own hacky way to query the zookeeper Thanks! Best, Siyuan On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues < bruno.rodrig...@litux.org> wrote: > Why not ask zookeeper for the list of br

Re: Is there a way to pull out kafka metadata from zookeeper?

Why not ask zookeeper for the list of brokers and then ask a random broker for the metadata (and repeat if the broker is down), even if it's two calls. Heck it already does unnecessary connections. It connects to a broker, gets the metadata, disconnects, and then connects again for the data. If it

Re: Is there a way to pull out kafka metadata from zookeeper?

What if the broker it tries to get the TopicMetadata from is down? I assume a broker is more like to be down than a zookeeper. Thanks, Siyuan On Fri, Oct 11, 2013 at 2:39 PM, Neha Narkhede wrote: > >> For each consumer consumes different > topic/replica I have to specify those 20 brokers and go

Re: Is there a way to pull out kafka metadata from zookeeper?

>> For each consumer consumes different topic/replica I have to specify those 20 brokers and go over all of them to know which broker is alive. And even worse how about I dynamically add new broker into the cluster and remove the old one TopicMetadataRequest is a batch API and you can get metadata

Re: Is there a way to pull out kafka metadata from zookeeper?

Thanks guys! But I feel weird. Assume I have 20 brokers for 10 different topics with 2 partitions and 2 replicas for each. For each consumer consumes different topic/replica I have to specify those 20 brokers and go over all of them to know which broker is alive. And even worse how about I dynamic

Re: Is there a way to pull out kafka metadata from zookeeper?

If, for some reason, you don't have access to a virtual IP or load balancer, you need to round robin once through all the brokers before failing a TopicMetadataRequest. So unless all the brokers in your cluster are down, this should not be a problem. Thanks, Neha On Thu, Oct 10, 2013 at 10:50 PM

Re: Is there a way to pull out kafka metadata from zookeeper?

You could use a broker list or a VIP to deal with broker failures. Thanks, Jun On Thu, Oct 10, 2013 at 10:50 PM, hsy...@gmail.com wrote: > Hi guys, > > I'm trying to maintain a bunch of simple kafka consumer to consume messages > from brokers. I know there is a way to send TopicMetadataReques

Is there a way to pull out kafka metadata from zookeeper?

Hi guys, I'm trying to maintain a bunch of simple kafka consumer to consume messages from brokers. I know there is a way to send TopicMetadataRequest to broker and get the response from the broker. But you have to specify the broker list to query the information. But broker might not be available