This is a total shot in the dark here so please ignore this if it fails to make sense, but I remember that on some previous implementation of the producer prior to when round-robin was enabled, producers would send messages to only one of the partitions for a set period of time (configurable, I believe) before moving onto the next one. This caused me a similar grievance as I would notice only a few of my consumers would get data while others were completely idle.
Sounds similar, so check if that's a possibility at all? On Wed, Jul 15, 2015 at 3:04 PM, Jagbir Hooda <jho...@gmail.com> wrote: > Hi Stefan, > > Have you looked at the following output for message distribution > across the topic-partitions and which topic-partition is consumed by > which consumer thread? > > kafaka-server/bin>./kafka-run-class.sh > kafka.tools.ConsumerOffsetChecker --zkconnect localhost:2181 --group > <consumer_group_name> > > Jagbir > > On Wed, Jul 15, 2015 at 12:50 PM, Stefan Miklosovic > <mikloso...@gmail.com> wrote: > > I have following problem, I tried almost everything I could but without > any luck > > > > All I want to do is to have 1 producer, 1 topic, 10 partitions and 10 > consumers. > > > > All I want is to send 1M of messages via producer to these 10 consumers. > > > > I am using built Kafka 0.8.3 from current upstream so I have bleeding > > edge stuff. It does not work on 0.8.1.1 nor 0.8.2 stream. > > > > The problem I have is that I expect that when I send 1 milion of > > messages via that producer, I will have all consumers busy. In other > > words, if a message to be sent via producer is sent to partition > > randomly (roundrobin / range), I expect that all 10 consumers will > > process about 100k of messages each because producer sends it to > > random partition of these 10. > > > > But I have never achieved such outcome. > > > > I was trying these combinations: > > > > 1) old scala producer vs old scala consumer > > > > Consumer was created by Consumers.createJavaConsumer() ten times. > > Every consumer is running in the separate thread. > > > > 2) old scala producer vs new java consumer > > > > new consumer was used like I have 10 consumers listening for a topic > > and 10 consumers subscribed to 1 partition. (consumer 1 - partition 1, > > consumer 2 - paritition 2 and so on) > > > > 3) old scala producer with custom partitioner > > > > I even tried to use my own partitioner, I just generated a random > > number from 0 to 9 so I expected that the messages will be sent > > randomly to the partition of that number. > > > > All I see is that there are only couple of consumers from these 10 > > utilized, even I am sending 1M of messages, all I got from the > > debugging output is some preselected set of consumers which appear to > > be selected randomly. > > > > Do you have ANY hint why all consumers are not utilized even > > partitions are selected randomly? > > > > My initial suspicion was that rebalancing was done badly. The think > > was I was generating old consumers in a loop quicky one after another > > and I can imaging that rebalancing algorithm got mad. > > > > So I abandon this solution and I was thinking that let's just > > subscribe these consumers one by one to some partition so I will have > > 1 consumer subscribed just to 1 partition and there will not be any > > rebalancing at all. > > > > Oh my how wrong was I ... nothing changed. > > > > So I was thinking that if I have 10 consumers, each one subscribed to > > 1 paritition, maybe producer is just sending messages to some set of > > partitions and that's it. I was not sure how this can be possible so > > to be super sure about the even spreading of message to partitions, I > > used custom partitioner class in old consumer so I will be sure that > > the partition the message will be sent to is super random. > > > > But that does not seems to work either. > > > > Please people, help me. > > > > -- > > Stefan Miklosovic > -- Jiefu Gong University of California, Berkeley | Class of 2017 B.A Computer Science | College of Letters and Sciences jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427
