Hi Prabhjot, Do you intent to use the old producer performance microbenchmark?
Thanks, -Tao On Fri, Aug 21, 2015 at 2:41 AM, Prabhjot Bharaj <prabhbha...@gmail.com> wrote: > Hi, > > I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along > the bundle > > I have setup a 5 machine cluster and on the same 5 machines, I'm also > running zookeeper as well > > I am trying to see what is the maximum produce throughput I can get on this > 5 node cluster > > I have created only 1 topic - tops1 in the cluster:- > > root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper > localhost:2182 > > Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: > > Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 > > Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 > > Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 > > Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 > > > This is the output of the kafka-producer-perf-test.sh for request-num-acks > 1 and request-num-acks -1:- > > root@x.x.x.x:~# date;time kafka-producer-perf-test.sh --broker-list > x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages > 1000000 --message-size 500 --topics tops1 --show-detailed-stats --threads > 30 --request-num-acks 1 --batch-size 1000 --request-timeout-ms 10000 > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > > 2015-08-21 09:36:01:021, 2015-08-21 09:36:49:351, 0, 500, 1000, 476.83, > 9.8662, 999990, 20690.8752 > > > real 0m49.375s > > user 0m38.777s > > sys 0m5.951s > > > > root@x.x.x.x:~# date;time kafka-producer-perf-test.sh > --broker-list > x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 > --messages > 1000000 --message-size 500 --topics tops1 --show-detailed-stats --threads > 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms 10000 > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > > 2015-08-21 09:31:19:329, 2015-08-21 09:32:08:627, 0, 500, 1000, 476.83, > 9.6724, 999990, 20284.5957 > > > real 0m50.341s > > user 0m39.029s > > sys 0m6.051s > > > I'll continue to test with more number of messages, to see if I get any > difference. > > But, I just want to be sure whatever I'm doing is right. > > > Please share your thoughts on the results. > > > Regards, > > prabcs >
