I have run the end to end latency test and the producerPerformance test on my kafka cluster according to https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
In end to end latency test, the latency was around 2ms. In producerperformance test, if use batch size 8196 to send 50,000,000 records: >bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance speedx1 50000000 100 -1 acks=1 bootstrap.servers=192.168.1.1:9092 buffer.memory=67108864 batch.size=8196 The results show that max latency is 3617ms, avg latency 626.7ms. I wanna know why the latency in producerperformance test is significantly larger than end to end test? Is it because of batching? Are the definitons of these two latencies different? I looked at the source code and I believe the latency is measure for the producer.send() function to complete. So does this latency includes transmission delay, transferring delay, and what other components? Thanks. best, Yuheng On Wed, Jul 15, 2015 at 3:51 AM, Yuheng Du <yuheng.du.h...@gmail.com> wrote: > Tao, > > If I am running on the command line the following command > >bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092 > 192.168.1.1:2181 speedx3 50000000 100 1 -Xmx1024m > > It promped that it is not correct. So where should I put the -Xmx1024m > option? Thanks. > > > > On Wed, Jul 15, 2015 at 3:44 AM, Tao Feng <fengta...@gmail.com> wrote: > >> (Please correct me if I am wrong.) Based on TestEndToEndLatency( >> >> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/kafka/tools/TestEndToEndLatency.scala >> ), >> consumer_fetch_max_wait corresponds to fetch.wait.max.ms in consumer >> config( >> http://kafka.apache.org/documentation.html#consumerconfigs). The default >> value listed at document is 100(ms). >> >> To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm >> option. >> >> On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du <yuheng.du.h...@gmail.com> >> wrote: >> >> > Tao, >> > >> > Thanks. The example on >> https://gist.github.com/jkreps/c7ddb4041ef62a900e6c >> > is outdated already. The error message shows: >> > >> > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list >> zookeeper_connect >> > topic num_messages consumer_fetch_max_wait producer_acks >> > >> > >> > Can anyone helps me what should be put in consumer_fetch_max_wait? >> Thanks. >> > >> > On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng <fengta...@gmail.com> wrote: >> > >> > > I think ProducerPerformance microbenchmark only measure between >> client >> > to >> > > brokers(producer to brokers) and provide latency information. >> > > >> > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du <yuheng.du.h...@gmail.com >> > >> > > wrote: >> > > >> > > > Currently, the latency test from kafka test the end to end latency >> > > between >> > > > producers and consumers. >> > > > >> > > > Is there a way to test the producer to broker and broker to >> consumer >> > > > delay seperately? >> > > > >> > > > Thanks. >> > > > >> > > >> > >> > >
