Re: Producer load balancing

The behavior that you described is explained here - https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? Thanks, Neha On Mon, Mar 17, 2014 at 6:26 PM, Abhinav Anand wrote: > Hi, > I am using the kafka produc

Producer load balancing

Hi, I am using the kafka producer 0.8. Each producer seem to be sending messages only to a specific broker until metadata refresh. Also I find each producer thread connected to only one broker at once. I had read that producer send messages in round robin fashion. Is there some specific configura

Re: Hardware planning

OK, I understand. So for the Zookeeper cluster, can I go with something like: 3 x Dell R320: Single hexcore 2.5GHz Xeon, 32GB RAM, 4x10K 300GB SAS drives, 10GbE and if I do, can I drop the CPU specs on the broker machines to say, dual 6 cores? Or are we looking at something that is core bound

Re: about consumer group name

Thanks Guozhang! Cheers, Churu On Mar 17, 2014, at 3:48 PM, Guozhang Wang wrote: > The short answer to both questions is yes. > > For the second question, the long answer will be: on startup of the > high-level consumer, it will register itself under its ZK group path, so it > knows other memb

Re: Documentation for metadata.broker.list

Currently metadata.broker.list is a separate list, unrelated to the cluster, which is used for all metadata requests. We are actually just now discussing whether this makes sense or not with respect to the new producer and consumer we are working on. I actually misunderstood how the existing produ

Re: New Consumer API discussion

I'm not quite sure if I fully understood your question. The consumer API exposes a close() method that will shutdown the consumer's connections to all brokers and frees up resources that the consumer uses. I've updated the javadoc for the new consumer API to include a few examples of different way

Re: about consumer group name

The short answer to both questions is yes. For the second question, the long answer will be: on startup of the high-level consumer, it will register itself under its ZK group path, so it knows other members in the group also; and upon rebalance triggered, it will read its peer number, and also the

queuing support and javadocs (newbie)

We are trying to adapt kafka for our project and would appreciate comments on my following questions-thank you 1)where is javadoc for kafka 0.81? This helps me look at availaible API support. Googling (nor kafka wiki) didnt take me to javadoc 2)One of the use cases we are looking at is queuing s

Re: Request Kafka / ZooKeeper JMS details

Hi Arun, Kafka doesn't provide JMS services. It has some architectural similarities with JMS, though Kafka is designed to scale to much larger systems than JMS, but if your app uses JMS, it will need to be modified to use Kafka Producers and Consumers in order to work. Steve On Mon, Mar 17, 20

Re: Request Kafka / ZooKeeper JMS details

Hi  I am new to Kafka and ZooKeeper, could you please help with answering the below questions..  We have both Kafka and Zookeeper in installed in DEV and we are successfully able to publish and consume messages, the question I have is,  1. Can you tell me how I can make sure JMS config is enab

Documentation for metadata.broker.list

I recently moved my 0.8.0 cluster to a set of entirely new brokers, and was surprised to find that the producers did not update their list of brokers to remove the brokers that were no longer in the cluster. That is, I had brokers 1,2,3,4,5, added brokers 6,7,8,910, waited a day, and stopped broker

Wirbelsturm released, 1-click deployments of Kafka clusters

Hi everyone, I have released a tool called Wirbelsturm (https://github.com/miguno/wirbelsturm) that allows you to perform local and remote deployments of Kafka. It's also a small way of saying a big "thank you" to the Kafka community. Wirbelsturm uses Vagrant for creating and managing machines,

Re: about consumer group name

Hi Guozhang, I really appreciate the detailed explanation! Does this mean that the rule “1 partition will be consumed by exactly one consumer in the consumer group” will only be checked and ensured at the consumer side? Also, does the “consumer group registration in Zookeeper” need to be taken

Re: Kafka Cluster management

Thanks. I checked in my sample code that requests have been distributed to brokers evenly without load balancers. ..Leena On Wed, Mar 12, 2014 at 3:34 PM, Todd Palino wrote: > I apologize, you are correct. I forgot that you can pass in a list of > hosts. Using a balancer is the way we handle