Thank you for the guidance, yet I want to know I am doing the test Jun had
in mind
On Thu, Jan 8, 2015 at 5:36 PM, Jayesh Thakrar <j_thak...@yahoo.com.invalid>
wrote:
> I do see the Windows based scripts in the tar file - but haven't them
> though.You should find them under bin/windows.
> Also you can always use other Windows stress testing tools/suites to check
> your local I/O performance..
> From: Shlomi Hazan <shl...@viber.com>
> To: users@kafka.apache.org; Jayesh Thakrar <j_thak...@yahoo.com>
> Sent: Thursday, January 8, 2015 6:25 AM
> Subject: Re: latency - how to reduce?
>
> I would like to test locally first as it is easier than setting up a test
> cluster to model the production, yet the script kafka-producer-perf-test is
> not available for windows.
> Jun, what kind of "basic I/O testing on the local FS" did you have in mind?
>
> Thanks,
> Shlomi
>
>
>
>
> On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar <j_thak...@yahoo.com.invalid
> >
> wrote:
>
> > Have you tried using the built-in stress test scripts?
> > bin/kafka-producer-perf-test.sh
> > bin/kafka-consumer-perf-test.sh
> >
> > Here's how I stress tested them -
> > nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> > ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> > --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
> >
> > nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> > ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> > 1>kafka-consumer-perf-test.sh.log 2>&1 &
> >
> > And I used screen scrapping of the jmx ui screens to push metrics into
> > TSDB to get the following.The rate below is per second - so I could push
> > the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> > little utilization (<30% utilization).
> >
> >
> > From: Shlomi Hazan <shl...@viber.com>
> > To: users@kafka.apache.org
> > Sent: Tuesday, January 6, 2015 1:06 AM
> > Subject: Re: latency - how to reduce?
> >
> > Will do. What did you have in mind? just write a big file to disk and
> > measure the time it took to write? maybe also read back? using specific
> > API's?
> > Apart from the local Win machine case, are you aware of any issues with
> > Amazon EC2 instances that may be causing that same latency in production?
> > Thanks,
> > Shlomi
> >
> >
> >
> > On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <j...@confluent.io> wrote:
> >
> > > Not setting "log.flush.interval.messages" is good since the default
> gives
> > > the best latency. Could you do some basic I/O testing on the local FS
> in
> > > your windows machine to make sure the I/O latency is ok?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <shl...@viber.com> wrote:
> > >
> > > > Happy new year!
> > > > I did not set "log.flush.interval.messages".
> > > > I also could not find a default value in the docs.
> > > > Could you explain about that?
> > > > Thanks,
> > > > Shlomi
> > > >
> > > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <j...@confluent.io> wrote:
> > > >
> > > > > What's your setting of log.flush.interval.messages on the broker?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <shl...@viber.com>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best
> and
> > > > even
> > > > > > seconds at worst.
> > > > > > I have this latency both on production, (with peak load of 30K
> > > msg/sec,
> > > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > > and on the local windows machine using just one process for each
> of
> > > > > > producer, zookeeper, kafka, consumer.
> > > > > > Also tried batch.num.messages=1 and producer.type=sync on the
> local
> > > > > machine
> > > > > > but saw no improvement.
> > > > > > How can I push latency down to several millis, at least when
> > running
> > > > > local?
> > > > > > Thanks,
> > > > > > Shlomi
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
>
>
>
