>From what I understand, if you set the throughput to -1, the producerperformance will push records as much as possible to an internal per topic per partition queue. In the background there is a sender IO thread handling the actual record sending process. If you push record to the queue faster than the send rate, your queue will become longer and longer, eventually record latency will become meaningless for a latency-purpose test.
On Tue, Aug 18, 2015 at 11:48 AM, Yuheng Du <yuheng.du.h...@gmail.com> wrote: > I see. Thank you Tao. But now I don't get it what Jay said that my latency > test only makes sense if I set a fixed throughput. Why do I need to set a > fixed throughput for my test instead of just set the expected throughput to > be -1 (as much as possible)? > > Thanks. > > On Tue, Aug 18, 2015 at 2:43 PM, Tao Feng <fengta...@gmail.com> wrote: > > > Hi Yuheng, > > > > The 10000 record/s is just a param for producerperformance for your > > producer target tput. It only takes effect to do the throttling if you > > tries to send more than 10000 record/s. The actual tput of the test > > depends on your producer config and your setup. > > > > -Tao > > > > On Tue, Aug 18, 2015 at 11:34 AM, Yuheng Du <yuheng.du.h...@gmail.com> > > wrote: > > > > > Also, When I set the target throughput to be 10000 records/s, The > actual > > > test results show I got an average of 579.86 records per second among > all > > > my producers. How did that happen? Why this number is not 10000 then? > > > Thanks. > > > > > > On Tue, Aug 18, 2015 at 10:03 AM, Yuheng Du <yuheng.du.h...@gmail.com> > > > wrote: > > > > > > > Thank you Jay, that really helps! > > > > > > > > Kishore, Where you can monitor whether the network is busy on IO in > > > visual > > > > vm? Thanks. I am running 90 producer process on 90 physical machines > in > > > the > > > > experiment. > > > > > > > > On Tue, Aug 18, 2015 at 1:19 AM, Jay Kreps <j...@confluent.io> wrote: > > > > > > > >> Yuheng, > > > >> > > > >> From the command you gave it looks like you are configuring the perf > > > test > > > >> to send data as fast as possible (the -1 for target throughput). > This > > > >> means > > > >> it will always queue up a bunch of unsent data until the buffer is > > > >> exhausted and then block. The larger the buffer, the bigger the > queue. > > > >> This > > > >> is where the latency comes from. This is exactly what you would > expect > > > and > > > >> what the buffering is supposed to do. > > > >> > > > >> If you want to measure latency this test doesn't really make sense, > > you > > > >> need to measure with some fixed throughput. Instead of -1 enter the > > > target > > > >> throughput you want to measure latency at (e.g. 100000 records/sec). > > > >> > > > >> -Jay > > > >> > > > >> On Thu, Aug 13, 2015 at 12:18 PM, Yuheng Du < > yuheng.du.h...@gmail.com > > > > > > >> wrote: > > > >> > > > >> > Thank you Alvaro, > > > >> > > > > >> > How to use sync producers? I am running the standard > > > ProducerPerformance > > > >> > test from kafka to measure the latency of each message to send > from > > > >> > producer to broker only. > > > >> > The command is like "bin/kafka-run-class.sh > > > >> > org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 > > 100 > > > -1 > > > >> > acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > >> > buffer.memory=67108864 batch.size=8196" > > > >> > > > > >> > For running producers, where should I put the producer.type=sync > > > >> > configuration into? The config/server.properties? Also Does this > > mean > > > we > > > >> > are using batch size of 1? Which version of Kafka are you using? > > > >> > thanks. > > > >> > > > > >> > On Thu, Aug 13, 2015 at 3:01 PM, Alvaro Gareppe < > agare...@gmail.com > > > > > > >> > wrote: > > > >> > > > > >> > > Are you measuring latency as time between producer and consumer > ? > > > >> > > > > > >> > > In that case, the ack shouldn't affect the latency, cause even > > tough > > > >> your > > > >> > > producer is not going to wait for the ack, the consumer will > only > > > get > > > >> the > > > >> > > message after its commited in the server. > > > >> > > > > > >> > > About latency my best result occur with sync producers, but the > > > >> > throughput > > > >> > > is much lower in that case. > > > >> > > > > > >> > > About not flushing to disk I'm pretty sure that it's not an > option > > > in > > > >> > kafka > > > >> > > (correct me if I'm wrong) > > > >> > > > > > >> > > Regards, > > > >> > > Alvaro Gareppe > > > >> > > > > > >> > > On Thu, Aug 13, 2015 at 12:59 PM, Yuheng Du < > > > yuheng.du.h...@gmail.com > > > >> > > > > >> > > wrote: > > > >> > > > > > >> > > > Also, the latency results show no major difference when using > > > ack=0 > > > >> or > > > >> > > > ack=1. Why is that? > > > >> > > > > > > >> > > > On Thu, Aug 13, 2015 at 11:51 AM, Yuheng Du < > > > >> yuheng.du.h...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > I am running an experiment where 92 producers is publishing > > data > > > >> > into 6 > > > >> > > > > brokers and 10 consumer are reading online data > > simultaneously. > > > >> > > > > > > > >> > > > > How should I do to reduce the latency? Currently when I run > > the > > > >> > > producer > > > >> > > > > performance test the average latency is around 10s. > > > >> > > > > > > > >> > > > > Should I disable log.flush? How to do that? Thanks. > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > Ing. Alvaro Gareppe > > > >> > > agare...@gmail.com > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > >
