Consumer stops consuming after 5 gb

2013-07-23 Thread Anurup Raveendran
After consuming about 5 gb of messages , it stopped consuming and it got stuck at a particular offset. After running the following command I got an error regarding the broker. $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group air-DummyProductionConsumerGroup-After --zkconnect

kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
I have 2 kafka brokers running on two systems with the same configuration CPU - Dual Core RAM - 4 GB I'm trying to benchmark my kafka setup Number of messages - 1 for 2 partitions 2 threads configuration - consumer consumes in 1.175 seconds for 1 partition 1 thread configuration -

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
305 bytes On Mon, Jul 8, 2013 at 8:41 PM, Tom Brown tombrow...@gmail.com wrote: What is the size of each message? --Tom On Mon, Jul 8, 2013 at 9:04 AM, Anurup Raveendran anurup.raveend...@fluturasolutions.com wrote: I have 2 kafka brokers running on two systems with the same

Re: kafka multi threaded consumer with very low throughput

2013-07-08 Thread Anurup Raveendran
..!!\n\n); } System.out.println(main thread woke up); example.shutdown(); } } On Mon, Jul 8, 2013 at 8:43 PM, Anurup Raveendran anurup.raveend...@fluturasolutions.com wrote: 305 bytes On Mon, Jul 8, 2013 at 8:41 PM, Tom Brown tombrow...@gmail.com wrote: What is the size