Also, I can see the topic "speedx2" being created in the broker, but not
message data is coming through.
On Sun, Nov 29, 2015 at 7:00 PM, Yuheng Du wrote:
> Hi guys,
>
> I was running a single node broker in a cluster. And when I run the
> producer in another cluster, I g
Hi guys,
I was running a single node broker in a cluster. And when I run the
producer in another cluster, I got connection time out error.
I can ping into port 9092 and other ports on the broker machine from the
producer. I just can't publish any messages. The command I used to run the
producer i
ll the client about all the other brokers. But best practices state
> including all of them is best.
> -Erik
>
> On 9/14/15, 2:46 PM, "Yuheng Du" wrote:
>
> >I am writing a kafka producer application in java. I want the producer to
> >publish data to a cluster o
I am writing a kafka producer application in java. I want the producer to
publish data to a cluster of 6 brokers. Is there a way to specify only the
load balancing node but not all the brokers list?
For example, like in the benchmarking kafka commandssdg:
bin/kafka-run-class.sh org.apache.kafka.c
ssages which caused this behavior, or if it was a consistent
> issue with that node.
> -Erik
>
>
> On 9/9/15, 2:24 PM, "Yuheng Du" wrote:
>
> >So are you suggesting that the long delays happened in %1 percentile
> >happens in the slower partitions that are furt
brokers is further than the others.
> -Erik
>
> On 9/4/15, 1:06 PM, "Yuheng Du" wrote:
>
> >No problem. Thanks for your advice. I think it would be fun to explore. I
> >only know how to program in java though. Hope it will work.
> >
> >On Fri, Sep 4, 20
ecially of writing to a file in the callback that
> >>kfaka
> >> provides.
> >>
> >> Also, I have noticed that most of the very slow observations are at
> >> startup. But don’t trust me, trust the data and share your findings.
> >> Also, having
r”, or for values that are high like 99.9%’ile, “0.1% of
> all events will be slower than this”.
> -Erik
>
> On 9/4/15, 12:05 PM, "Yuheng Du" wrote:
>
> >Thank you Erik! That's is helpful!
> >
> >But also I see jitters of the maximum latencies when runni
atency second. And
> it does a really good job at both.
>
> Disclaimer: I might not like linear algebra, but I do like statistics.
> Let me know if there are topics that need more explanation above that
> aren¹t covered by Gil¹s lecture.
> -Erik
>
> On 9/4/15, 9:03 AM, "Yuheng
Can't read it. Sorry
On Fri, Sep 4, 2015 at 12:08 PM, Roman Shramkov
wrote:
> Её ай н Анны уйг
>
> sent from a mobile device, please excuse brevity and typos
>
>
> ----Пользователь Yuheng Du написал
>
> According to the section 3.1 of the paper "Kafka: a
When I using 32 partitions, the 4 brokers latency becomes larger than the 8
brokers latency.
So is it always true that using more brokers can give less latency when the
number of partitions is at least the size of the brokers?
Thanks.
On Thu, Sep 3, 2015 at 10:45 PM, Yuheng Du wrote:
> I
According to the section 3.1 of the paper "Kafka: a Distributed Messaging
System for Log Processing":
"a message is only exposed to the consumers after it is flushed"?
Is it still true in the current kafka? like the message can only be
available after it is flushed to disk?
Thanks.
I am running a producer latency test. When using 92 producers in 92
physical node publishing to 4 brokers, the latency is slightly lower than
using 8 brokers, I am using 8 partitions for the topic.
I have rerun the test and it gives me the same result, the 4 brokers
scenario still has lower latenc
atency will become meaningless for a
> latency-purpose test.
>
>
> On Tue, Aug 18, 2015 at 11:48 AM, Yuheng Du
> wrote:
>
> > I see. Thank you Tao. But now I don't get it what Jay said that my
> latency
> > test only makes sense if I set a fixed throughput.
r config and your setup.
>
> -Tao
>
> On Tue, Aug 18, 2015 at 11:34 AM, Yuheng Du
> wrote:
>
> > Also, When I set the target throughput to be 1 records/s, The actual
> > test results show I got an average of 579.86 records per second among all
> > my produc
Also, When I set the target throughput to be 1 records/s, The actual
test results show I got an average of 579.86 records per second among all
my producers. How did that happen? Why this number is not 1 then?
Thanks.
On Tue, Aug 18, 2015 at 10:03 AM, Yuheng Du
wrote:
> Thank you
o measure latency at (e.g. 10 records/sec).
>
> -Jay
>
> On Thu, Aug 13, 2015 at 12:18 PM, Yuheng Du
> wrote:
>
> > Thank you Alvaro,
> >
> > How to use sync producers? I am running the standard ProducerPerformance
> > test from kafka to measure the l
> the batches are not waiting in the queue unnecessarily) . Also may be you
> want to increase the batch.size further more, you will get even better
> throughput with more or less same latency (as there is no shortage of
> events in the test program).
>
> On Thu, Aug 13, 2015 at 1:13 PM
gt; flush may not influence the end-to-end latency from the Producer to
> Consumer, however it can influence the throughput of the broker.
>
> On Fri, Aug 14, 2015 at 9:20 AM Yuheng Du
> wrote:
>
> > So if I understand correctly, even if I delay flushing, the consumer will
>
ived it (this
> > much is documented). This doesn't need to include writing to disk, which
> > will happen asynchronously.
> >
> >
> > >
> > > Even though you flush periodically based on log.flush.interval.messages
> > and
> > > log.flush.in
Hi,
As I understand it, kafka brokers will store the incoming messages into
pagecache as much as possible and then flush them into disk, right?
But in my experiment where 90 producers is publishing data into 6 brokers,
I see that the log directory on disk where broker stores the data is
constantl
erPerformance, I created my
> own. but I think there is a producer.properties file in config folder in
> kafka.. is that configuration not for this tester ?
>
> On Thu, Aug 13, 2015 at 4:18 PM, Yuheng Du
> wrote:
>
> > Thank you Alvaro,
> >
> > How
me if I'm wrong)
>
> Regards,
> Alvaro Gareppe
>
> On Thu, Aug 13, 2015 at 12:59 PM, Yuheng Du
> wrote:
>
> > Also, the latency results show no major difference when using ack=0 or
> > ack=1. Why is that?
> >
> > On Thu, Aug 13, 2015 at 11:51
I am running an experiment where 92 producers is publishing data into 6
brokers and 10 consumer are reading online data simultaneously.
How should I do to reduce the latency? Currently when I run the producer
performance test the average latency is around 10s.
Should I disable log.flush? How to d
Also, the latency results show no major difference when using ack=0 or
ack=1. Why is that?
On Thu, Aug 13, 2015 at 11:51 AM, Yuheng Du
wrote:
> I am running an experiment where 92 producers is publishing data into 6
> brokers and 10 consumer are reading online data simultaneously.
&
Hi,
I am running a test which 92 producers each publish 53000 records of size
254 bytes to 2 brokers.
The average latency shown in each producer has high variations. For some
producer, the average latency is as low as 38ms to send the 53000 records;
but for some producer, the average latency is a
Hi guys,
I was reading a paper today in which the latency of kafka and rabbitmq is
compared:
http://downloads.hindawi.com/journals/js/2015/468047.pdf
To my surprise, kafka has shown some large variations of latency as the
number of records per second increases.
So I am curious about why is that.
15 PM, Prabhjot Bharaj
wrote:
> Hi,
>
> Have you tried with acks=1 and -1 as well?
> Please share the numbers and the message size
>
> Regards,
> Prabcs
> On Jul 27, 2015 10:24 PM, "Yuheng Du" wrote:
>
> > Hi,
> >
> > I am running 40 producers
t looks
> like the default setting for log.segment.bytes is 1GB, so given fast enough
> cleanup of old logs, you may not need to adjust that setting -- assuming
> you have a reasonable amount of storage, you'll easily fit many dozen log
> files of that size.
>
> -Ewen
>
>
eck.
>
> -Ewen
>
> On Mon, Jul 27, 2015 at 10:03 AM, Yuheng Du
> wrote:
>
> > If I want to get higher throughput, should I increase the
> > log.segment.bytes?
> >
> > I don't see log.retention.check.interval.ms, but there is
> > log.cleanup.interva
ht also need to adjust log.segment.bytes/log.roll.{ms,hours} and
> log.retention.check.interval.ms.
>
> On Fri, Jul 24, 2015 at 12:49 PM, Yuheng Du
> wrote:
>
> > Hi,
> >
> > I am testing the kafka producer performance. So I created a queue and
> > writes a large amoun
Hi,
I am running 40 producers on 40 nodes cluster. The messages are sent to 6
brokers in another cluster. The producers are running ProducerPerformance
test.
When 20 nodes are running, the throughput is around 13MB/s and when running
40 nodes, the throughput is around 9MB/s.
I have set log.reten
Hi,
I am testing the kafka producer performance. So I created a queue and
writes a large amount of data to that queue.
Is there a way to delete the data automatically after some time, say
whenever the data size reaches 50GB or the retention time exceeds 10
seconds, it will be deleted so my disk w
I deleted the queue and recreated it before I run the test. Things are
working after restart the broker cluster, thanks!
On Fri, Jul 24, 2015 at 12:06 PM, Gwen Shapira
wrote:
> Does topic "speedx1" exist?
>
> On Fri, Jul 24, 2015 at 7:09 AM, Yuheng Du
> wrote:
> >
Hi,
I am trying to run 20 performance test on 10 nodes using pbsdsh.
The messages will send to a 6 brokers cluster. It seems to work for a
while. When I delete the test queue and rerun the test, the broker does not
seem to process incoming messages:
[yuhengd@node1739 kafka_2.10-0.8.2.1]$ bin/kaf
Thank you, Nicolas!
On Tue, Jul 21, 2015 at 10:46 AM, Nicolas Phung
wrote:
> Yes indeed.
>
> # A comma seperated list of directories under which to store log files
> log.dirs=/var/lib/kafka
>
> You can put several disk/partitions too.
>
> Regards,
>
> On Tue, Jul
Just wanna make sure, in server.properties, the configuration
log.dirs=/tmp/kafka-logs
specifies the directory of where the log (data) stores, right?
If I want the data to be saved elsewhere, this is the configuration I need
to change, right?
Thanks for answering.
best,
sistently full (which it should be for
> ProducerPerformance since it's sending as fast as it can), that means
> *every* record waits that long.
>
> Of course, these numbers are estimates, depend on my having used 1ms, but
> hopefully should make it clear why you can see relatively l
thub.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh
> > KAFKA_JVM_PERFORMANCE_OPTS.
> >
> >
> >
> > On Wed, Jul 15, 2015 at 12:51 AM, Yuheng Du
> > wrote:
> >
> >> Tao,
> >>
> >> If I am running on the command line the following co
t; https://github.com/confluentinc/kafka/tree/KAFKA-2276/tests).
>
> The tool we're using to bring up the slave virtual machines is called
> vagrant, so the "vagrant" steps in the quickstart are really telling you
> how to install the virtual machines.
>
> Hope that
record the transferring of a message from producer
> to broker, then to consumer.
>
> I cannot remember the details not but I think the EndtoEndLatency test
> record the latency as average, hence it is small.
>
> Guozhang
>
> On Wed, Jul 15, 2015 at 12:28 PM, Yuheng Du
> wr
reases.
>
> Guozhang
>
> On Wed, Jul 15, 2015 at 11:36 AM, Yuheng Du
> wrote:
>
> > In kafka performance tests https://gist.github.com/jkreps
> > /c7ddb4041ef62a900e6c
> >
> > The TestEndtoEndLatency results are typically around 2ms, while the
> > Prod
n the test
> you're interested in:
> cd kafka/tests
> ducktape kafkatest/tests/benchmark_test.py
>
> Definitely keep us posted about which parts are difficult, annoying, or
> confusing about this process and we'll do our best to help.
>
> Thanks,
> Geoff
>
In kafka performance tests https://gist.github.com/jkreps
/c7ddb4041ef62a900e6c
The TestEndtoEndLatency results are typically around 2ms, while the
ProducerPerformance normally has "average latency"around several hundres ms
when using batch size 8196.
Are both results talking about end to end lat
lay, transferring delay, and what
other components?
Thanks.
best,
Yuheng
On Wed, Jul 15, 2015 at 3:51 AM, Yuheng Du wrote:
> Tao,
>
> If I am running on the command line the following command
> >bin/kafka-run-class.sh kafka.tools.TestEndToEndLatency 192.168.1.3:9092
> 192.168.1.1:2181 s
Hi,
I have run the end to end latency test and the producerPerformance test on
my kafka cluster according to
https://gist.github.com/jkreps/c7ddb4041ef62a900e6c
In end to end latency test, the latency was around 2ms. In
producerperformance test, if use batch size 8196 to send 50,000,000 records:
s.
> >
> > Ewen,
> > Thank you immensely for your thoughts, they shed a lot of insight into
> the
> > issue. Though it is understandable that your specific results need to be
> > verified, it seems that the KIP-25 patch is functional and I can use it
> for
r
> config(
> http://kafka.apache.org/documentation.html#consumerconfigs). The default
> value listed at document is 100(ms).
>
> To add java heap space to jvm, put -Xmx$Size(max heap size) for your jvm
> option.
>
> On Wed, Jul 15, 2015 at 12:29 AM, Yuheng Du
> wrote:
>
Java heap space
at kafka.tools.TestEndToEndLatency$.main(TestEndToEndLatency.scala:69)
at kafka.tools.TestEndToEndLatency.main(TestEndToEndLatency.scala)
What command should I do to add java heap space to jvm? Thanks!
Yuheng
On Wed, Jul 15, 2015 at 3:29 AM, Yuheng Du wrote:
> Tao,
&g
put in consumer_fetch_max_wait? Thanks.
On Tue, Jul 14, 2015 at 5:21 PM, Tao Feng wrote:
> I think ProducerPerformance microbenchmark only measure between client to
> brokers(producer to brokers) and provide latency information.
>
> On Tue, Jul 14, 2015 at 11:05 AM, Yuheng
] INFO [Kafka Server 1], shutting down
(kafka.server.KafkaServer)
I have checked that the zookeeper is running fine. Can anyone help why I
got the error? Thanks.
On Tue, Jul 14, 2015 at 10:24 PM, Yuheng Du
wrote:
> But is there a way to let kafka override the old data if the disk is
> filled?
But is there a way to let kafka override the old data if the disk is
filled? Or is it not necessary to use this figure? Thanks.
On Tue, Jul 14, 2015 at 10:14 PM, Yuheng Du
wrote:
> Jiefu,
>
> I agree with you. I checked the hardware specs of my machines, each one of
> them
ect any more attempts to write data?
>
> On Tue, Jul 14, 2015 at 2:27 PM, Yuheng Du
> wrote:
>
> > Also, the log in another broker (not the bootstrap) says:
> >
> > [2015-07-14 15:18:41,220] FATAL [Replica Manager on Broker 1]: Error
> > writing to highwatermark fi
)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffe
(END)
On Tue, Jul 14, 2015 at 5:24 PM, Yuheng Du wrote:
> Hi Jiefu, Gwen,
>
> I am running the Throughput versus stored data test:
> bin/kafka-
$foreach$1.apply(HashMap.scala:98)
at scala.coll
Can you help me with this problem? Thanks.
On Tue, Jul 14, 2015 at 5:12 PM, Yuheng Du wrote:
> I checked the logs on the brokers, it seems that the zookeeper or the
> kafka server process is not running on this broker...Thank you
any of
> your brokers fall out of the ISR when sending messages? It seems like your
> setup should be fine, so I'm not entirely sure.
>
> On Tue, Jul 14, 2015 at 1:31 PM, Yuheng Du
> wrote:
>
> > Jiefu,
> >
> > I am performing these tests on a 6 nodes cluster
pendently.
>
> On Tue, Jul 14, 2015 at 1:17 PM, Gwen Shapira
> wrote:
>
> > Are there any errors on the broker logs?
> >
> > On Tue, Jul 14, 2015 at 11:57 AM, Yuheng Du
> > wrote:
> > > Jiefu,
> > >
> > > Thank you. The three producers
;
> Yes, if you read the blog post it specifies that he's using three separate
> machines. There's no reason the producers cannot be started at the same
> time, I believe.
>
> On Tue, Jul 14, 2015 at 11:42 AM, Yuheng Du
> wrote:
>
> > Hi,
> >
>
Hi,
I am running the performance test for kafka. https://gist.github.com/jkreps
/c7ddb4041ef62a900e6c
For the "Three Producers, 3x async replication" scenario, the command is
the same as one producer:
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test 5000 100 -1
Currently, the latency test from kafka test the end to end latency between
producers and consumers.
Is there a way to test the producer to broker and broker to consumer
delay seperately?
Thanks.
Also, I guess setting the target throughput to -1 means let it be as high
as possible?
On Tue, Jul 14, 2015 at 10:36 AM, Yuheng Du
wrote:
> Thanks. If I set the acks=1 in the producer config options in
> bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
> test7
Kafka cluster
>
> https://kafka.apache.org/documentation.html#newproducerconfigs
>
> On Tue, Jul 14, 2015 at 7:29 PM, Yuheng Du
> wrote:
>
> > Does anyone know what is bootstrap.servers=
> > esv4-hcl198.grid.linkedin.com:9092 means in the following test com
Does anyone know what is bootstrap.servers=
esv4-hcl198.grid.linkedin.com:9092 means in the following test command:
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test7 5000 100 -1 acks=1 bootstrap.servers=
esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 ba
mance
>
> USAGE: java org.apache.kafka.clients.tools.ProducerPerformance topic_name
> num_records record_size target_records_sec [prop_name=prop_value]*
>
>
>
> On Tue, 14 Jul 2015 at 05:08 Yuheng Du wrote:
>
> > I am using the binaries of kafka_2.10-0.8.2.1. Could that be the problem?
> > Sh
ext editor and modify the
> classpath -- I believe I had a similar error before.
>
> On Mon, Jul 13, 2015 at 1:16 PM, Yuheng Du
> wrote:
>
> > Hi guys,
> >
> > I am trying to replicate the test of benchmarking kafka at
> >
> >
> http://engineering.lin
directory is the ProducerPerformance class resides?
Thanks.
On Mon, Jul 13, 2015 at 4:37 PM, JIEFU GONG wrote:
> You may need to open up your run-class.sh in a text editor and modify the
> classpath -- I believe I had a similar error before.
>
> On Mon, Jul 13, 2015 at 1:16 PM, Yuheng
Hi guys,
I am trying to replicate the test of benchmarking kafka at
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
.
When I run
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test7 5000 100 -1 acks=1 boo
Hi Wan,
I tried to install this DCMonitor, but when I try to clone the project, but
it gives me "Permission denied, the remote end hung up unexpectedly". Can
you provide any suggestions to this issue?
Thanks.
best,
Yuheng
On Mon, Mar 23, 2015 at 8:54 AM, Wan Wei wrote:
> We have make a simple
p://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html
>
>
> --
> Harsha
>
> On March 9, 2015 at 8:39:00 AM, Yuheng Du (yuheng.du.h...@gmail.com)
> wrote:
>
> Harsha,
>
> Thanks for reply. So what if the zookeeper cluster fails? Will the topics
> information
topic
> metadata as well. You can use zookeeper-shell.sh or zkCli.sh to check zk
> nodes, /brokers/topics will give you the list of topics .
>
> --
> Harsha
>
>
> On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.h...@gmail.com)
> wrote:
>
> I am wondering where doe
I am wondering where does kafka cluster keep the topic metadata (name,
partition, replication, etc)? How does a server recover the topic's
metadata and messages after restart and what data will be lost?
Thanks for anyone to answer my questions.
best,
Yuheng
uster.
>
> Good luck!
>
> On Thu, Mar 5, 2015 at 12:30 PM, Yuheng Du
> wrote:
> > Thank you Gwen,
> >
> > I also need the kafka cluster continue to provide message brokering
> service
> > to a Storm cluster after the benchmarking. I am fairly new to cluster
ults:
>
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
>
> Gwen
>
> On Thu, Mar 5, 2015 at 12:16 PM, Yuheng Du
> wrote:
> > Hi everyone,
> >
> > I am trying to set up a kafka cluster consisti
Hi everyone,
I am trying to set up a kafka cluster consisting of three machines. I wanna
run a benchmarking program in them. Can anyone recommend a step by step
tutorial/instruction of how I can do it?
Thanks.
best,
Yuheng
74 matches
Mail list logo
