I would like to test locally first as it is easier than setting up a test
cluster to model the production, yet the script kafka-producer-perf-test is
not available for windows.
Jun, what kind of "basic I/O testing on the local FS" did you have in mind?
Thanks,
Shlomi
On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar <j_thak...@yahoo.com.invalid>
wrote:
> Have you tried using the built-in stress test scripts?
> bin/kafka-producer-perf-test.sh
> bin/kafka-consumer-perf-test.sh
>
> Here's how I stress tested them -
> nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
>
> nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> 1>kafka-consumer-perf-test.sh.log 2>&1 &
>
> And I used screen scrapping of the jmx ui screens to push metrics into
> TSDB to get the following.The rate below is per second - so I could push
> the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> little utilization (<30% utilization).
>
>
> From: Shlomi Hazan <shl...@viber.com>
> To: users@kafka.apache.org
> Sent: Tuesday, January 6, 2015 1:06 AM
> Subject: Re: latency - how to reduce?
>
> Will do. What did you have in mind? just write a big file to disk and
> measure the time it took to write? maybe also read back? using specific
> API's?
> Apart from the local Win machine case, are you aware of any issues with
> Amazon EC2 instances that may be causing that same latency in production?
> Thanks,
> Shlomi
>
>
>
> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <j...@confluent.io> wrote:
>
> > Not setting "log.flush.interval.messages" is good since the default gives
> > the best latency. Could you do some basic I/O testing on the local FS in
> > your windows machine to make sure the I/O latency is ok?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <shl...@viber.com> wrote:
> >
> > > Happy new year!
> > > I did not set "log.flush.interval.messages".
> > > I also could not find a default value in the docs.
> > > Could you explain about that?
> > > Thanks,
> > > Shlomi
> > >
> > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <j...@confluent.io> wrote:
> > >
> > > > What's your setting of log.flush.interval.messages on the broker?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <shl...@viber.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> > > even
> > > > > seconds at worst.
> > > > > I have this latency both on production, (with peak load of 30K
> > msg/sec,
> > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > and on the local windows machine using just one process for each of
> > > > > producer, zookeeper, kafka, consumer.
> > > > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > > > machine
> > > > > but saw no improvement.
> > > > > How can I push latency down to several millis, at least when
> running
> > > > local?
> > > > > Thanks,
> > > > > Shlomi
> > > > >
> > > >
> > >
> >
>
>
>
>
