etc. But it effected that one broker more than
> others. And the reasons for my cluster displaying this behavior could be
> very different than the reason for any other cluster.
>
> Its worth noting that this was mostly a latency test over a stress test.
> There was a single kafka
reason for any other cluster.
Its worth noting that this was mostly a latency test over a stress test.
There was a single kafka producer object, very small message sizes (100
bytes), and it was only pushing through around 5MB/s worth of data. And
the client was configured to minimize the amount of
So are you suggesting that the long delays happened in %1 percentile
happens in the slower partitions that are further away? Thanks.
On Wed, Sep 9, 2015 at 3:15 PM, Helleren, Erik
wrote:
> So, I did my own latency test on a cluster of 3 nodes, and there is a
> significant difference arou
So, I did my own latency test on a cluster of 3 nodes, and there is a
significant difference around the 99%’ile and higher for partitions when
measuring the the ack time when configured for a single ack. The graph
that I wish I could attach or post clearly shows that around 1/3 of the
partitions
As for latency tests, there isn’t a lot of code to do a latency test. If
> you just want to measure ack time its around 100 lines. I will try to
> push out some good latency testing code to github, but my company is
> scared of open sourcing code… so it might be a while…
> -Erik
>
&g
I thing the suggestion is to have partitions/brokers >=1, so 32 should be
enough.
As for latency tests, there isn’t a lot of code to do a latency test. If
you just want to measure ack time its around 100 lines. I will try to
push out some good latency testing code to github, but my company
brokers always results in more throughput. But
> >> throughput and latency are two different creatures. Its worth noting
> >>that
> >> kafka is designed to be high throughput first and low latency second.
> >>And
> >> it does a really good job at both.
job at both.
>>
>> Disclaimer: I might not like linear algebra, but I do like statistics.
>> Let me know if there are topics that need more explanation above that
>> aren¹t covered by Gil¹s lecture.
>> -Erik
>>
>> On 9/4/15, 9:03 AM, "Yuheng Du&q
Du" wrote:
>
> >When I using 32 partitions, the 4 brokers latency becomes larger than the
> >8
> >brokers latency.
> >
> >So is it always true that using more brokers can give less latency when
> >the
> >number of partitions is at least the size of
the 4 brokers latency becomes larger than the
>8
>brokers latency.
>
>So is it always true that using more brokers can give less latency when
>the
>number of partitions is at least the size of the brokers?
>
>Thanks.
>
>On Thu, Sep 3, 2015 at 10:45 PM, Yuheng Du
>wr
am running a producer latency test. When using 92 producers in 92
> physical node publishing to 4 brokers, the latency is slightly lower than
> using 8 brokers, I am using 8 partitions for the topic.
>
> I have rerun the test and it gives me the same result, the 4 brokers
> scenar
I am running a producer latency test. When using 92 producers in 92
physical node publishing to 4 brokers, the latency is slightly lower than
using 8 brokers, I am using 8 partitions for the topic.
I have rerun the test and it gives me the same result, the 4 brokers
scenario still has lower
t; >> > > USAGE: java kafka.tools.TestEndToEndLatency$ broker_list
> >> > zookeeper_connect
> >> > > topic num_messages consumer_fetch_max_wait producer_acks
> >> > >
> >> > >
> >> > > Can anyone
EndToEndLatency$ broker_list
>> > zookeeper_connect
>> > > topic num_messages consumer_fetch_max_wait producer_acks
>> > >
>> > >
>> > > Can anyone helps me what should be put in consumer_fetch_max_wait?
>> > Thanks.
>> > >
2015 at 5:21 PM, Tao Feng wrote:
> > >
> > > > I think ProducerPerformance microbenchmark only measure between
> client
> > > to
> > > > brokers(producer to brokers) and provide latency information.
> > > >
> > > > On Tue, Jul 14,
I have run the end to end latency test and the producerPerformance test on
my kafka cluster according to
https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
In end to end latency test, the latency was around 2ms. In
producerperformance test, if use batch size 8196 to send 50,000,000 records
etween client
> > to
> > > brokers(producer to brokers) and provide latency information.
> > >
> > > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du
> > > wrote:
> > >
> > > > Currently, the latency test from kafka test the end to end latency
> > > between
> > > > producers and consumers.
> > > >
> > > > Is there a way to test the producer to broker and broker to
> consumer
> > > > delay seperately?
> > > >
> > > > Thanks.
> > > >
> > >
> >
>
14, 2015 at 5:21 PM, Tao Feng wrote:
>
> > I think ProducerPerformance microbenchmark only measure between client
> to
> > brokers(producer to brokers) and provide latency information.
> >
> > On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du
> > wrote:
>
I got java out of heap error when running end to end latency test:
yuhengdu@consumer0:/packages/kafka_2.10-0.8.2.1$ bin/kafka-run-class.sh
kafka.tools.TestEndToEndLatency 192.168.1.3:9092 192.168.1.1:2181 speedx3
5000 100 1
Exception in thread "main" java.lang.OutOfMemoryError:
Du
> wrote:
>
> > Currently, the latency test from kafka test the end to end latency
> between
> > producers and consumers.
> >
> > Is there a way to test the producer to broker and broker to consumer
> > delay seperately?
> >
> > Thanks.
> >
>
I think ProducerPerformance microbenchmark only measure between client to
brokers(producer to brokers) and provide latency information.
On Tue, Jul 14, 2015 at 11:05 AM, Yuheng Du
wrote:
> Currently, the latency test from kafka test the end to end latency between
> producers and con
Currently, the latency test from kafka test the end to end latency between
producers and consumers.
Is there a way to test the producer to broker and broker to consumer
delay seperately?
Thanks.
22 matches
Mail list logo
