Thank you for your reply. I am running tests using a simple application with one broker. That is why I am running everything on a single machine. For the scalability, my application's throughput scales by more than 2x going from 1 consumer/partition to 2 consumers/partitions which is great. However, going from 2 to 4 consumers/partitions the scalability diminishes to 1.3x. I am monitoring the resources of the machine (memory,disk,cpu) and didn't see any bottleneck forming. My application is very simple. I assign each consumer to a specific partition. I have a single producer that produces logs to all my partitions evenly before I start the consumers. My consumers retrieves logs one by one and print them to terminal. After processing each record, I seek the offset of the partition to the offset of the processed record+1 for the following poll request. I am aware that I don't need a poll request for each record. but I am doing that as part of my tests.
On Thu, May 19, 2016 at 11:45 AM, Tom Crayford <tcrayf...@heroku.com> wrote: > Hi there, > > Firstly, I'd recommend not running the consumers and the brokers on the > same machine. Are you running multiple brokers? If not, that'd be my first > recommendation (it sounds like you might not be). > > Secondly, yes, consumers scale up with partitions. At most you can have the > same number of consumers as partitions (but each consumer can have multiple > partitions). > > That error message seems pretty bad - it sounds like the broker is falling > over repeatedly. I'd check the broker logs and metrics (see > http://docs.confluent.io/2.0.1/kafka/monitoring.html). > > Thanks > > Tom Crayford > Heroku Kafka > > On Thu, May 19, 2016 at 7:14 PM, Yazeed Alabdulkarim < > y.alabdulka...@gmail.com> wrote: > > > Hi, > > > > For Kafka consumers, is it expected that the throughput will scale > linearly > > as I increase the number of consumers/partitions? > > > > Also, I keep getting this info message: "Kafka Consumer Marking the > > coordinator 2147483647 dead." What is the problem? How can I fix it? My > > program continues without any problem but I am worried that it maybe > > impacting the performance. I tried to set all the config timeout > parameters > > to a large number. Both Kafka consumers and server are running on the > same > > machine. > > > > -- > > Best Regards, > > Yazeed Alabdulkarim > > > -- Best Regards, Yazeed Alabdulkarim
