Hi Yuheng, The 10000 record/s is just a param for producerperformance for your producer target tput. It only takes effect to do the throttling if you tries to send more than 10000 record/s. The actual tput of the test depends on your producer config and your setup.
-Tao On Tue, Aug 18, 2015 at 11:34 AM, Yuheng Du <yuheng.du.h...@gmail.com> wrote: > Also, When I set the target throughput to be 10000 records/s, The actual > test results show I got an average of 579.86 records per second among all > my producers. How did that happen? Why this number is not 10000 then? > Thanks. > > On Tue, Aug 18, 2015 at 10:03 AM, Yuheng Du <yuheng.du.h...@gmail.com> > wrote: > > > Thank you Jay, that really helps! > > > > Kishore, Where you can monitor whether the network is busy on IO in > visual > > vm? Thanks. I am running 90 producer process on 90 physical machines in > the > > experiment. > > > > On Tue, Aug 18, 2015 at 1:19 AM, Jay Kreps <j...@confluent.io> wrote: > > > >> Yuheng, > >> > >> From the command you gave it looks like you are configuring the perf > test > >> to send data as fast as possible (the -1 for target throughput). This > >> means > >> it will always queue up a bunch of unsent data until the buffer is > >> exhausted and then block. The larger the buffer, the bigger the queue. > >> This > >> is where the latency comes from. This is exactly what you would expect > and > >> what the buffering is supposed to do. > >> > >> If you want to measure latency this test doesn't really make sense, you > >> need to measure with some fixed throughput. Instead of -1 enter the > target > >> throughput you want to measure latency at (e.g. 100000 records/sec). > >> > >> -Jay > >> > >> On Thu, Aug 13, 2015 at 12:18 PM, Yuheng Du <yuheng.du.h...@gmail.com> > >> wrote: > >> > >> > Thank you Alvaro, > >> > > >> > How to use sync producers? I am running the standard > ProducerPerformance > >> > test from kafka to measure the latency of each message to send from > >> > producer to broker only. > >> > The command is like "bin/kafka-run-class.sh > >> > org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 > -1 > >> > acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > >> > buffer.memory=67108864 batch.size=8196" > >> > > >> > For running producers, where should I put the producer.type=sync > >> > configuration into? The config/server.properties? Also Does this mean > we > >> > are using batch size of 1? Which version of Kafka are you using? > >> > thanks. > >> > > >> > On Thu, Aug 13, 2015 at 3:01 PM, Alvaro Gareppe <agare...@gmail.com> > >> > wrote: > >> > > >> > > Are you measuring latency as time between producer and consumer ? > >> > > > >> > > In that case, the ack shouldn't affect the latency, cause even tough > >> your > >> > > producer is not going to wait for the ack, the consumer will only > get > >> the > >> > > message after its commited in the server. > >> > > > >> > > About latency my best result occur with sync producers, but the > >> > throughput > >> > > is much lower in that case. > >> > > > >> > > About not flushing to disk I'm pretty sure that it's not an option > in > >> > kafka > >> > > (correct me if I'm wrong) > >> > > > >> > > Regards, > >> > > Alvaro Gareppe > >> > > > >> > > On Thu, Aug 13, 2015 at 12:59 PM, Yuheng Du < > yuheng.du.h...@gmail.com > >> > > >> > > wrote: > >> > > > >> > > > Also, the latency results show no major difference when using > ack=0 > >> or > >> > > > ack=1. Why is that? > >> > > > > >> > > > On Thu, Aug 13, 2015 at 11:51 AM, Yuheng Du < > >> yuheng.du.h...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > I am running an experiment where 92 producers is publishing data > >> > into 6 > >> > > > > brokers and 10 consumer are reading online data simultaneously. > >> > > > > > >> > > > > How should I do to reduce the latency? Currently when I run the > >> > > producer > >> > > > > performance test the average latency is around 10s. > >> > > > > > >> > > > > Should I disable log.flush? How to do that? Thanks. > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Ing. Alvaro Gareppe > >> > > agare...@gmail.com > >> > > > >> > > >> > > > > >
