Also, When I set the target throughput to be 10000 records/s, The actual test results show I got an average of 579.86 records per second among all my producers. How did that happen? Why this number is not 10000 then? Thanks.
On Tue, Aug 18, 2015 at 10:03 AM, Yuheng Du <yuheng.du.h...@gmail.com> wrote: > Thank you Jay, that really helps! > > Kishore, Where you can monitor whether the network is busy on IO in visual > vm? Thanks. I am running 90 producer process on 90 physical machines in the > experiment. > > On Tue, Aug 18, 2015 at 1:19 AM, Jay Kreps <j...@confluent.io> wrote: > >> Yuheng, >> >> From the command you gave it looks like you are configuring the perf test >> to send data as fast as possible (the -1 for target throughput). This >> means >> it will always queue up a bunch of unsent data until the buffer is >> exhausted and then block. The larger the buffer, the bigger the queue. >> This >> is where the latency comes from. This is exactly what you would expect and >> what the buffering is supposed to do. >> >> If you want to measure latency this test doesn't really make sense, you >> need to measure with some fixed throughput. Instead of -1 enter the target >> throughput you want to measure latency at (e.g. 100000 records/sec). >> >> -Jay >> >> On Thu, Aug 13, 2015 at 12:18 PM, Yuheng Du <yuheng.du.h...@gmail.com> >> wrote: >> >> > Thank you Alvaro, >> > >> > How to use sync producers? I am running the standard ProducerPerformance >> > test from kafka to measure the latency of each message to send from >> > producer to broker only. >> > The command is like "bin/kafka-run-class.sh >> > org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 >> > acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 >> > buffer.memory=67108864 batch.size=8196" >> > >> > For running producers, where should I put the producer.type=sync >> > configuration into? The config/server.properties? Also Does this mean we >> > are using batch size of 1? Which version of Kafka are you using? >> > thanks. >> > >> > On Thu, Aug 13, 2015 at 3:01 PM, Alvaro Gareppe <agare...@gmail.com> >> > wrote: >> > >> > > Are you measuring latency as time between producer and consumer ? >> > > >> > > In that case, the ack shouldn't affect the latency, cause even tough >> your >> > > producer is not going to wait for the ack, the consumer will only get >> the >> > > message after its commited in the server. >> > > >> > > About latency my best result occur with sync producers, but the >> > throughput >> > > is much lower in that case. >> > > >> > > About not flushing to disk I'm pretty sure that it's not an option in >> > kafka >> > > (correct me if I'm wrong) >> > > >> > > Regards, >> > > Alvaro Gareppe >> > > >> > > On Thu, Aug 13, 2015 at 12:59 PM, Yuheng Du <yuheng.du.h...@gmail.com >> > >> > > wrote: >> > > >> > > > Also, the latency results show no major difference when using ack=0 >> or >> > > > ack=1. Why is that? >> > > > >> > > > On Thu, Aug 13, 2015 at 11:51 AM, Yuheng Du < >> yuheng.du.h...@gmail.com> >> > > > wrote: >> > > > >> > > > > I am running an experiment where 92 producers is publishing data >> > into 6 >> > > > > brokers and 10 consumer are reading online data simultaneously. >> > > > > >> > > > > How should I do to reduce the latency? Currently when I run the >> > > producer >> > > > > performance test the average latency is around 10s. >> > > > > >> > > > > Should I disable log.flush? How to do that? Thanks. >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > Ing. Alvaro Gareppe >> > > agare...@gmail.com >> > > >> > >> > >
