Thanks, Jaikiran
I was trying to duplicate the same issue by running the same performance
test on master node of cluster , say exemplary-birds.master, and I did see
such error again
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server
is not the leader for that topic
, 19995.0 records/sec (57.21 MB/sec), 774.8 ms avg
latency, 3858.0 max latency.
On Wed, Jan 7, 2015 at 12:07 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I am doing performance test by
bin/kafka-run-class.sh org.apache.kafka.clients.
tools.ProducerPerformance
test-rep-three 5 100 -1
.
100315 records sent, 19995.0 records/sec (57.21 MB/sec), 774.8 ms avg
latency, 3858.0 max latency.
On Wed, Jan 7, 2015 at 12:07 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I am doing performance test by
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three
at 12:07 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I am doing performance test by
bin/kafka-run-class.sh org.apache.kafka.clients.
tools.ProducerPerformance
test-rep-three 5 100 -1 acks=1 bootstrap.servers=
10.100.98.100:9092,10.100.98.101:9092,10.100.98.102:9092
buffer.memory=67108864
Hi, All
I am doing performance test by
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three 5 100 -1 acks=1 bootstrap.servers=10.100.98.100:9092,
10.100.98.101:9092,10.100.98.102:9092 buffer.memory=67108864 batch.size=8196
where the topic test-rep
avg
latency, 3858.0 max latency.
On Wed, Jan 7, 2015 at 12:07 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I am doing performance test by
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three 5 100 -1 acks=1 bootstrap.servers=
10.100.98.100
I checked topic config, isr changes dynamically.
root@voluminous-mass:/srv/kafka# bin/kafka-topics.sh --describe --zookeeper
10.100.98.101:2181 --topic test-rep-three
Topic:test-rep-threePartitionCount:8ReplicationFactor:3
Configs:
Topic: test-rep-three Partition: 0
Hi, All
I am running performance test on kafka, the command
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three 500 100 -1 acks=1 bootstrap.servers=
10.100.10.101:9092 buffer.memory=67108864 batch.size=8196
Since we send 50 billions to brokers
Thanks Joe,
that does the trick with sbt and the 0.8.2.-beta-test jar.
Regards,
Markus
Joe Crobak joec...@gmail.com schrieb am 23:28 Sonntag, 9.November 2014:
For sbt, you need to use something like:
org.apache.kafka %% kafka %0.8.2-beta % test classifier test
That tells sbt to pull
For sbt, you need to use something like:
org.apache.kafka %% kafka %0.8.2-beta % test classifier test
That tells sbt to pull in the kafka artifact with the test classifier
only when running tests. The %% tells sbt to fill in the scala version (so
it'll map to kafka_2.10 like in your example
Hello,
I want to use the kafka_2.10-0.8.2-beta-test.jar in my Scala project.
It can be found here:
http://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.10/0.8.1.1/
In my build.sbt I write the following definition:
org.apache.kafka % kafka_2.10 % 0.8.2-beta-test
But sbt cannot find
The following is how samza references the kafka test jar in gradle.
testCompile org.apache.kafka:kafka_$scalaVersion:$kafkaVersion:test
Thanks,
Jun
On Thu, Nov 6, 2014 at 6:38 AM, Markus Jais markus.j...@yahoo.de wrote:
Hello,
I want to use the kafka_2.10-0.8.2-beta-test.jar in my
and it should be there.
-Jay
On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I found
someone post following thread, that is exactly the problem I have:
-
While testing kafka producer
wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I
found
someone post following thread, that is exactly the problem I have:
-
While testing kafka producer performance, I found 2 testing scripts.
1) performance testing script in kafka
was developed with the new producer that is included on
trunk. Checkout trunk and build and it should be there.
-Jay
On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I
found
someone
On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I
found
someone post following thread, that is exactly the problem I have:
-
While testing kafka producer performance, I
on
trunk. Checkout trunk and build and it should be there.
-Jay
On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I
found
someone post following thread, that is exactly
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I found
someone post following thread, that is exactly the problem I have:
-
While testing kafka producer performance, I found 2 testing scripts.
1) performance testing script in kafka distribution
bin/kafka
It is available with Kafka package containing the source code. Download
the package, build it and run the above command.
Regards,
Ravi
On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I found
someone post
PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I found
someone post following thread, that is exactly the problem I have:
-
While testing kafka producer performance, I found 2 testing scripts.
1) performance testing
Hi Sa,
That script was developed with the new producer that is included on
trunk. Checkout trunk and build and it should be there.
-Jay
On Wed, Oct 1, 2014 at 7:55 PM, Sa Li sal...@gmail.com wrote:
Hi, All
I built a 3-node kafka cluster, I want to make performance test, I found
someone
. So to run Kafka, you need a scala jar.
Thanks,
Jun
On Wed, Aug 27, 2014 at 4:05 PM, Parin Jogani parin.jog...@gmail.com
wrote:
As far as I understand it shouldnt have such a dependency. JVM should be
able to compile this. I am using JDK1.6. Also this is a unit test.
Correct me if I
to run Kafka, you need a scala jar.
Thanks,
Jun
On Wed, Aug 27, 2014 at 4:05 PM, Parin Jogani parin.jog...@gmail.com
wrote:
As far as I understand it shouldnt have such a dependency. JVM should be
able to compile this. I am using JDK1.6. Also this is a unit test.
Correct me if I am
As far as I understand it shouldnt have such a dependency. JVM should be
able to compile this. I am using JDK1.6. Also this is a unit test.
Correct me if I am wrong. I have not downloaded any scala before even till
this far.
-Parin
On Tue, Aug 26, 2014 at 10:12 PM, Jun Rao jun...@gmail.com
test.
Correct me if I am wrong. I have not downloaded any scala before even till
this far.
-Parin
On Tue, Aug 26, 2014 at 10:12 PM, Jun Rao jun...@gmail.com wrote:
Could you make sure that you are using the scala 2.9.2 jar?
Thanks,
Jun
On Tue, Aug 26, 2014 at 9:28 PM, Parin
to create unit test case for Kafka in Java with a simple
call
Properties props =TestUtils.createBrokerConfig(1,
TestUtils.choosePort(), true);
It fails on
java.lang.NoSuchMethodError:
scala.Predef$.intWrapper(I)Lscala/runtime/RichInt;
at kafka.utils.TestUtils
I am trying to create unit test case for Kafka in Java with a simple call
Properties props =TestUtils.createBrokerConfig(1,
TestUtils.choosePort(), true);
It fails on
java.lang.NoSuchMethodError:
scala.Predef$.intWrapper(I)Lscala/runtime/RichInt;
at kafka.utils.TestUtils
Parin,
Which scala version are you using? And which kafka version are you working
on?
Guozhang
On Sat, Aug 23, 2014 at 7:16 PM, Parin Jogani parin.jog...@gmail.com
wrote:
I am trying to create unit test case for Kafka in Java with a simple call
Properties props
:
Parin,
Which scala version are you using? And which kafka version are you working
on?
Guozhang
On Sat, Aug 23, 2014 at 7:16 PM, Parin Jogani parin.jog...@gmail.com
wrote:
I am trying to create unit test case for Kafka in Java with a simple call
Properties props
Quick ? - I built kafka-0.8.1.1-src ( and then 0.8.1 for good measure) - when I
run the first test single thread, no replication I see the following:
USAGE: java org.apache.kafka.clients.tools.ProducerPerformance url num_records
record_size
I am using this syntax:
/var/tmp/kafka-0.8.1.1-src
is the
overhead required from decompression on the broker to assign Ids. Here
is
a good article that describes this
http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/
But again we haven’t decided just yet. Would like to test and evaluate.
Bert
?
We're going to test this against our production workload so it's not a big deal
for us but I wondered if this could give others skewed results?
---
Daniel
://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/
But again we haven’t decided just yet. Would like to test and evaluate.
Bert
On Mon, Jun 30, 2014 at 2:24 AM, Daniel Compton d...@danielcompton.net
wrote:
Hi folks
I was doing some performance testing using
?
We're going to test this against our production workload so it's not a big
deal for us but I wondered if this could give others skewed results?
---
Daniel
compression is the
overhead required from decompression on the broker to assign Ids. Here is
a good article that describes this
http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/
But again we haven’t decided just yet. Would like to test and evaluate.
Bert
I've got a very strange bug doing a long poll using SimpleConsumer on 0.8.1.1.
The test app I've got (attached) uses a thread to do a long poll of a topic,
and then sends single messages to that queue, one at a time, waiting for the
poll to return the message before sending the next one
SimpleConsumer on
0.8.1.1. The test app I've got (attached) uses a thread to do a long poll
of a topic, and then sends single messages to that queue, one at a time,
waiting for the poll to return the message before sending the next one.
The problem is that when a message is sent to the queue
Some emails don't seem to be delivered to the mailing list. This is just a
test to see if the mailing list still works.
Jun
I am using the latest version of kafka.
package kafka.test;
import kafka.consumer.Consumer;import kafka.consumer.ConsumerConfig;import
kafka.consumer.ConsumerIterator;import kafka.consumer.KafkaStream;import
kafka.javaapi.consumer.ConsumerConnector;import
kafka.javaapi.producer.Producer;import
Could you re-format your code in the email? They can hard be read from my
browser.
Guozhang
On Thu, Apr 10, 2014 at 6:58 AM, Khalef Bessaih
bessaihkha...@hotmail.comwrote:
I am using the latest version of kafka.
package kafka.test;
import kafka.consumer.Consumer;import
different error
responses from kafka to make sure my code handles it properly. How do you
guys advised to do this within my unit test? Do you have some kind of mock
kafka that capable of doing this? What would be your advise?
Thanks so much,
Vadim.
integration/unit tests that simulate different error
responses from kafka to make sure my code handles it properly. How do you
guys advised to do this within my unit test? Do you have some kind of
mock
kafka that capable of doing this? What would be your advise?
Thanks so much,
Vadim.
Neha. That is correct SimpleConsumer does not do much, but my code that is
calling it need to react properly at error response and I wanted to test
that.
On Thu, Mar 13, 2014 at 1:21 PM, Neha Narkhede neha.narkh...@gmail.comwrote:
Currently, we don't have a mock class for the server
I see. In that case, it might be easier to write a MockSimpleConsumer and
have it return different FetchResponse objects with the error codes you
want to test. We don't have a MockSimpleConsumer as well. If you happen to
write one, you could use it with EasyMock to get the fetch() API to return
Good evening. I am writing consumer that is based on SimpleConsumer. I
would like to write integration/unit tests that simulate different error
responses from kafka to make sure my code handles it properly. How do you
guys advised to do this within my unit test? Do you have some kind of mock
Why do we need to create that gradle file? Can't we ship a default for that?
Also we should really comment out the @Test annotation on the failing test.
I think we checked that in so I had a case to fix against but I think it is
a bit disruptive.
-Jay
On Mon, Feb 17, 2014 at 7:41 PM, Guozhang
comment out the @Test annotation on the failing test.
I think we checked that in so I had a case to fix against but I think it is
a bit disruptive.
-Jay
On Mon, Feb 17, 2014 at 7:41 PM, Guozhang Wang wangg...@gmail.com wrote:
testSendToPartition is suppose to fail now since it detects some
Jay.
I think the plan is to ship a default in the final release, details in the
RB:
https://reviews.apache.org/r/17879/
For now just commenting out the @Test tag does not help since the junit
test will still search for any functions with prefix test... So we need to
change the function name
in
the new producer code. We are working on it.
For other test failures, I just redid the unit tests from trunk but does
not see the failures. My test process:
1. Check out a new repository of trunk.
2. Create ~/.gradle/gradle.properties file according to README.md
3. ./gradlew test
Only
on it.
For other test failures, I just redid the unit tests from trunk but does
not see the failures. My test process:
1. Check out a new repository of trunk.
2. Create ~/.gradle/gradle.properties file according to README.md
3. ./gradlew test
Only testSendToPartition failed.
Guozhang
it detects some bugs in
the new producer code. We are working on it.
For other test failures, I just redid the unit tests from trunk but does
not see the failures. My test process:
1. Check out a new repository of trunk.
2. Create ~/.gradle/gradle.properties file according
to maven.
Thanks,
Jun
On Mon, Feb 17, 2014 at 7:41 PM, Guozhang Wang wangg...@gmail.com
wrote:
testSendToPartition is suppose to fail now since it detects some bugs
in
the new producer code. We are working on it.
For other test failures, I just redid the unit tests from trunk
testSendToPartition is suppose to fail now since it detects some bugs in
the new producer code. We are working on it.
For other test failures, I just redid the unit tests from trunk but does
not see the failures. My test process:
1. Check out a new repository of trunk.
2. Create ~/.gradle
Is this error stable? could you check out a clean trunk and retry unit
tests?
Guozhang
2014-02-14 22:56 GMT-08:00 Bae, Jae Hyeon metac...@gmail.com:
- LogOffsetTest kafka.server.LogOffsetTest.html.
testEmptyLogsGetOffsetskafka.server.LogOffsetTest.html#testEmptyLogsGetOffsets
-
Test failures are stable with the latest trunk, I am building Mac OSX with
Java version 1.7.0_45
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
The followings are log snippet from gradle build:
kafka.server.LogOffsetTest
- LogOffsetTest kafka.server.LogOffsetTest.html.
testEmptyLogsGetOffsetskafka.server.LogOffsetTest.html#testEmptyLogsGetOffsets
- LogOffsetTest kafka.server.LogOffsetTest.html.
testGetOffsetsBeforeEarliestTimekafka.server.LogOffsetTest.html#testGetOffsetsBeforeEarliestTime
-
Subject: Re: Kafka performance test: --request-num-acks -1 kills throughput
Michael,
Your understanding is mostly correct. For 2, the follower will issue another
fetch request as soon as it finishes processing the response of the previous
fetch (by adding data, if any, to its local log
What's replica.fetch.min.bytes? Is it the default value of 1? If not, could
you try to set it to 1? If yes, could you try using a smaller value of
replica.fetch.wait.max.ms? You can then re-run the test and report back the
results.
Thanks,
Jun
On Mon, Feb 3, 2014 at 9:26 AM, Michael Popov mpo
To: users@kafka.apache.org
Subject: Re: Kafka performance test: --request-num-acks -1 kills throughput
What's replica.fetch.min.bytes? Is it the default value of 1? If not, could you
try to set it to 1? If yes, could you try using a smaller value of
replica.fetch.wait.max.ms? You can then re-run
...@gmail.com]
Sent: Monday, February 3, 2014 10:09 AM
To: users@kafka.apache.org
Subject: Re: Kafka performance test: --request-num-acks -1 kills
throughput
What's replica.fetch.min.bytes? Is it the default value of 1? If not,
could you try to set it to 1? If yes, could you try using
a new topic with 1 partition for each client process.
Commands to create a topic looked like this one:
bin/kafka-create-topic.sh --partition 1 --replica 2 --zookeeper
10.0.0.4:2181 --topic d1
Commands to run a test producer looked like this:
bin/kafka-producer-perf-test.sh --broker
-perf-test.sh shows
--request-timeout-ms Integer The produce request timeout in ms
(default: 3000)
In my test runs I used the following command-line arguments:
bin/kafka-producer-perf-test.sh --broker-list 10.0.0.8:9092,
10.0.0.10:9092 --compression-codec 0 --message-size 1024
@kafka.apache.org
Subject: Re: Kafka performance test: --request-num-acks -1 kills throughput
Michael,
The producer client in 0.8 has a single thread that does blocking data sends
one broker at a time. However, we are working on a rewrite of the producer with
an improved design that can support
Hi,
We need a reliable low-latency message queue that can scale. Kafka looks like a
right system for this role.
I am running performance tests on multiple platforms: Linux and Windows. For
test purposes I create topics with 2 replicas and multiple partitions. In all
deployments running test
that can scale. Kafka looks
like a right system for this role.
I am running performance tests on multiple platforms: Linux and Windows.
For test purposes I create topics with 2 replicas and multiple partitions.
In all deployments running test producers that wait for both replicas' acks
I was just reading
this:https://cwiki.apache.org/confluence/display/KAFKA/Performance+testingHas
this been implemented yet? I can't seem tofind it in the source.
Best Regards
Jerry
The results there are obtained using our ProducerPerformance tool.
Thanks,
Jun
On Wed, Jan 15, 2014 at 1:14 AM, 陈小军 chenxiao...@nhn.com wrote:
I was just reading this:
https://cwiki.apache.org/confluence/display/KAFKA/Performance+testingHasthis
been implemented yet? I can't seem tofind
Sorry no clue about that - anyone else know?
On Mon, Oct 28, 2013 at 10:41 AM, Roger Hoover roger.hoo...@gmail.com wrote:
Joel,
Thank you! This is very helpful.
What I notice now is that it works for Test classes that
extend org.scalatest.junit.JUnit3Suite. There are other tests
Joel,
Thank you! This is very helpful.
What I notice now is that it works for Test classes that
extend org.scalatest.junit.JUnit3Suite. There are other tests in the
codebase that use a @Test annotation as in the example below. Any idea how
to run those?
import org.junit.{Test, After, Before
Hi,
I'm new to Scala but working on a simple patch for a configuration change
and want to run just my unit tests. When I run ./sbt test-only, it
executes all sorts of other tests but not the one I want. Is there an easy
way to run a single test? Any help is appreciated.
$ ./sbt test-only
In the sbt shell:
projects (to see the available projects)
project core
test-only unit test
Although lately if there is a test failure it isn't very helpful in
saying exactly where the test failed; my environment is probably
messed up but I know of one or two others who are having similar
Hi,
We observed below correlation between kafka configuration and performance
data:
1. producer insertion rate drops as compression enabled, especially gzip;
2. when the producer batch size is below 200, we got high CPU/network IO on
brokers, got high network IO on producers/consumers;
What's
Are these for Kafka 08?
For #2 above, when you say high network I/O, do you mean number of packets
transferred or size in bytes transferred?
Thanks,
Neha
On Mon, Oct 21, 2013 at 12:01 PM, Lu Xuechao lux...@gmail.com wrote:
Hi,
We observed below correlation between kafka configuration and
Yes, I am using 0.8.
Network IO means bytes transferred.
Thanks for reply.
On Mon, Oct 21, 2013 at 2:24 PM, Neha Narkhede neha.narkh...@gmail.comwrote:
Are these for Kafka 08?
For #2 above, when you say high network I/O, do you mean number of packets
transferred or size in bytes
If the data is compressed, the broker has to recompress the messages in
order to assign offsets. So, there is some CPU overhead. However, it
shouldn't be too high. How high of the CPU load did you observe?
Thanks,
Jun
On Mon, Oct 21, 2013 at 12:01 PM, Lu Xuechao lux...@gmail.com wrote:
Hi,
Hello,
I wrote a simple junit test to test a Kafka producer.
public class KafkaProducerTest {
private int brokerId = 0;
private String topic = test;
@Test
public void producerTest() throws InterruptedException {
// setup Zookeeper
String zkConnect
wrote:
Hello,
I wrote a simple junit test to test a Kafka producer.
public class KafkaProducerTest {
private int brokerId = 0;
private String topic = test;
@Test
public void producerTest() throws InterruptedException {
// setup Zookeeper
String zkConnect
I tried that, but I still got
junit.framework.AssertionFailedError: Partition [test,0] metadata not
propagated after timeout
at
kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:506)
at kafka.utils.TestUtils.waitUntilMetadataIsPropagated(TestUtils.scala
FYI in regards to https://cwiki.apache.org/confluence/display/KAFKA/Index
-- Forwarded message --
From: gmcdonald gmcdon...@apache.org
Date: Tue, Jun 18, 2013 at 7:56 AM
Subject: IMPORTANT: Major Confluence Upgrade Coming Soon. Please review
test instance now.
To: committ
I am using 0.8 in eclipse. I used the approach suggested to use the sbt plugin
and that works great (thank you to whomever recommended that). It pulled Scala
2.9.3. However, I am only able to run 2 common scala tests: TopicTest and
ConfigTest. How can I find and run the other scala tests?
for performance test.
I notice this line from the log:
INFO Property request.required.acks is overridden to -1
So what does -1 mean in this case? Is acknowledgement enabled?
In producer.properties, I set request.required.acks to 1 and started
the job. I still saw:
INFO Property request.required.acks
For our long running testing at LinkedIn we actually just use the mirror
maker tool and fork off a copy of the full production streams and throw it
at the test cluster. This is, of course, only useful if you have a
high-load cluster to replay...
-Jay
On Wed, Feb 6, 2013 at 11:27 AM, S Ahmed
201 - 282 of 282 matches
Mail list logo
