Re: kafka producer performance test
Yes, I did either way, 1: follow the instruction by http://www.gradle.org/installation, 2. apt-get install gradle. Thanks On Thu, Oct 2, 2014 at 2:21 PM, Guozhang Wang wrote: > Did you installed gradle as the README stated? > > "You need to have [gradle](http://www.gradle.org/installation) installed." > > Guozhang > > On Thu, Oct 2, 2014 at 1:55 PM, Sa Li wrote: > > > Thanks Guozhang > > > > I tried this as in KAFKA-1490: > > > > git clone https://git-wip-us.apache.org/repos/asf/kafka.git > > > > cd kafka > > > > gradle > > > > > > but fails to build: > > > > FAILURE: Build failed with an exception. > > > > * Where: > > > > Script '/home/stuser/trunk/gradle/license.gradle' line: 2 > > > > * What went wrong: > > > > A problem occurred evaluating script. > > > > > Could not find method create() for arguments [downloadLicenses, class > > nl.javadude.gradle.plugins.license.DownloadLicenses] on task set. > > > > * Try: > > > > Run with --stacktrace option to get the stack trace. Run with --info > > or --debug option to get more log output. > > > > BUILD FAILED > > > > > > > > > > Seems it is really not that straightforward to build > > > > > > thanks > > > > > > > > > > On Thu, Oct 2, 2014 at 12:56 PM, Guozhang Wang > wrote: > > > > > Hello Sa, > > > > > > KAFKA-1490 introduces a new step of downloading the wrapper, details > are > > > included in the latest README file. > > > > > > Guozhang > > > > > > On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: > > > > > > > Thanks, Jay, > > > > > > > > Here is what I did this morning, I git clone the latest version of > > kafka > > > > from git, (I am currently using kafka 8.0) now it is 8.1.1, and it > use > > > > gradle to build project. I am having trouble to build it. I installed > > > > gradle, and run ./gradlew jar in kafka root directory, it comes out: > > > > Error: Could not find or load main class > > > > org.gradle.wrapper.GradleWrapperMain > > > > > > > > Any idea about this. > > > > > > > > Thanks > > > > > > > > Alec > > > > > > > > On Wed, Oct 1, 2014 at 9:21 PM, Jay Kreps > wrote: > > > > > > > > > Hi Sa, > > > > > > > > > > That script was developed with the new producer that is included on > > > > > trunk. Checkout trunk and build and it should be there. > > > > > > > > > > -Jay > > > > > > > > > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > > > > > > Hi, All > > > > > > > > > > > > I built a 3-node kafka cluster, I want to make performance test, > I > > > > found > > > > > someone post following thread, that is exactly the problem I have: > > > > > > - > > > > > > While testing kafka producer performance, I found 2 testing > > scripts. > > > > > > > > > > > > 1) performance testing script in kafka distribution > > > > > > > > > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 > > > --messages > > > > > > 1000 --topic test --threads 10 --message-size 100 > --batch-size > > > > 1 > > > > > > --compression-codec 1 > > > > > > > > > > > > 2) performance testing script mentioned in > > > > > > > > > > > > > > > > > > > > > > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > > > > > > > > > bin/kafka-run-class.sh > > > > > > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 > > 100 > > > > > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > > > > buffer.memory=67108864 batch.size=8196 > > > > > > > > > > > > based on org.apache.kafka.clients.producer.Producer. > > > > > > > > > > > > —— > > > > > > > > > > > > > > > > > > I was unable to duplicate either of above method, I figure the > > > commands > > > > > are outdated, anyone point me how to do such test with new command? > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > Alec > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Alec Li > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > -- > > > > Alec Li > > > > > > -- > -- Guozhang > -- Alec Li
Re: kafka producer performance test
Did you installed gradle as the README stated? "You need to have [gradle](http://www.gradle.org/installation) installed." Guozhang On Thu, Oct 2, 2014 at 1:55 PM, Sa Li wrote: > Thanks Guozhang > > I tried this as in KAFKA-1490: > > git clone https://git-wip-us.apache.org/repos/asf/kafka.git > > cd kafka > > gradle > > > but fails to build: > > FAILURE: Build failed with an exception. > > * Where: > > Script '/home/stuser/trunk/gradle/license.gradle' line: 2 > > * What went wrong: > > A problem occurred evaluating script. > > > Could not find method create() for arguments [downloadLicenses, class > nl.javadude.gradle.plugins.license.DownloadLicenses] on task set. > > * Try: > > Run with --stacktrace option to get the stack trace. Run with --info > or --debug option to get more log output. > > BUILD FAILED > > > > > Seems it is really not that straightforward to build > > > thanks > > > > > On Thu, Oct 2, 2014 at 12:56 PM, Guozhang Wang wrote: > > > Hello Sa, > > > > KAFKA-1490 introduces a new step of downloading the wrapper, details are > > included in the latest README file. > > > > Guozhang > > > > On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: > > > > > Thanks, Jay, > > > > > > Here is what I did this morning, I git clone the latest version of > kafka > > > from git, (I am currently using kafka 8.0) now it is 8.1.1, and it use > > > gradle to build project. I am having trouble to build it. I installed > > > gradle, and run ./gradlew jar in kafka root directory, it comes out: > > > Error: Could not find or load main class > > > org.gradle.wrapper.GradleWrapperMain > > > > > > Any idea about this. > > > > > > Thanks > > > > > > Alec > > > > > > On Wed, Oct 1, 2014 at 9:21 PM, Jay Kreps wrote: > > > > > > > Hi Sa, > > > > > > > > That script was developed with the new producer that is included on > > > > trunk. Checkout trunk and build and it should be there. > > > > > > > > -Jay > > > > > > > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > > > > > Hi, All > > > > > > > > > > I built a 3-node kafka cluster, I want to make performance test, I > > > found > > > > someone post following thread, that is exactly the problem I have: > > > > > - > > > > > While testing kafka producer performance, I found 2 testing > scripts. > > > > > > > > > > 1) performance testing script in kafka distribution > > > > > > > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 > > --messages > > > > > 1000 --topic test --threads 10 --message-size 100 --batch-size > > > 1 > > > > > --compression-codec 1 > > > > > > > > > > 2) performance testing script mentioned in > > > > > > > > > > > > > > > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > > > > > > > bin/kafka-run-class.sh > > > > > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 > 100 > > > > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > > > buffer.memory=67108864 batch.size=8196 > > > > > > > > > > based on org.apache.kafka.clients.producer.Producer. > > > > > > > > > > —— > > > > > > > > > > > > > > > I was unable to duplicate either of above method, I figure the > > commands > > > > are outdated, anyone point me how to do such test with new command? > > > > > > > > > > > > > > > thanks > > > > > > > > > > Alec > > > > > > > > > > > > > > > > -- > > > > > > Alec Li > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > > Alec Li > -- -- Guozhang
Re: kafka producer performance test
I can't really gradle through, even clone the latest trunk, anyone having same issue? On Thu, Oct 2, 2014 at 1:55 PM, Sa Li wrote: > Thanks Guozhang > > I tried this as in KAFKA-1490: > > git clone https://git-wip-us.apache.org/repos/asf/kafka.git > > cd kafka > > gradle > > > but fails to build: > > FAILURE: Build failed with an exception. > > * Where: > > Script '/home/stuser/trunk/gradle/license.gradle' line: 2 > > * What went wrong: > > A problem occurred evaluating script. > > > Could not find method create() for arguments [downloadLicenses, class > > nl.javadude.gradle.plugins.license.DownloadLicenses] on task set. > > * Try: > > Run with --stacktrace option to get the stack trace. Run with --info or > --debug option to get more log output. > > BUILD FAILED > > > > > Seems it is really not that straightforward to build > > > thanks > > > > > On Thu, Oct 2, 2014 at 12:56 PM, Guozhang Wang wrote: > >> Hello Sa, >> >> KAFKA-1490 introduces a new step of downloading the wrapper, details are >> included in the latest README file. >> >> Guozhang >> >> On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: >> >> > Thanks, Jay, >> > >> > Here is what I did this morning, I git clone the latest version of kafka >> > from git, (I am currently using kafka 8.0) now it is 8.1.1, and it use >> > gradle to build project. I am having trouble to build it. I installed >> > gradle, and run ./gradlew jar in kafka root directory, it comes out: >> > Error: Could not find or load main class >> > org.gradle.wrapper.GradleWrapperMain >> > >> > Any idea about this. >> > >> > Thanks >> > >> > Alec >> > >> > On Wed, Oct 1, 2014 at 9:21 PM, Jay Kreps wrote: >> > >> > > Hi Sa, >> > > >> > > That script was developed with the new producer that is included on >> > > trunk. Checkout trunk and build and it should be there. >> > > >> > > -Jay >> > > >> > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: >> > > > Hi, All >> > > > >> > > > I built a 3-node kafka cluster, I want to make performance test, I >> > found >> > > someone post following thread, that is exactly the problem I have: >> > > > - >> > > > While testing kafka producer performance, I found 2 testing scripts. >> > > > >> > > > 1) performance testing script in kafka distribution >> > > > >> > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 >> --messages >> > > > 1000 --topic test --threads 10 --message-size 100 --batch-size >> > 1 >> > > > --compression-codec 1 >> > > > >> > > > 2) performance testing script mentioned in >> > > > >> > > > >> > > >> > >> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines >> > > > >> > > > bin/kafka-run-class.sh >> > > > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 >> 100 >> > > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 >> > > > buffer.memory=67108864 batch.size=8196 >> > > > >> > > > based on org.apache.kafka.clients.producer.Producer. >> > > > >> > > > —— >> > > > >> > > > >> > > > I was unable to duplicate either of above method, I figure the >> commands >> > > are outdated, anyone point me how to do such test with new command? >> > > > >> > > > >> > > > thanks >> > > > >> > > > Alec >> > > >> > >> > >> > >> > -- >> > >> > Alec Li >> > >> >> >> >> -- >> -- Guozhang >> > > > > -- > > Alec Li > -- Alec Li
Re: kafka producer performance test
Thanks Guozhang I tried this as in KAFKA-1490: git clone https://git-wip-us.apache.org/repos/asf/kafka.git cd kafka gradle but fails to build: FAILURE: Build failed with an exception. * Where: Script '/home/stuser/trunk/gradle/license.gradle' line: 2 * What went wrong: A problem occurred evaluating script. > Could not find method create() for arguments [downloadLicenses, class > nl.javadude.gradle.plugins.license.DownloadLicenses] on task set. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Seems it is really not that straightforward to build thanks On Thu, Oct 2, 2014 at 12:56 PM, Guozhang Wang wrote: > Hello Sa, > > KAFKA-1490 introduces a new step of downloading the wrapper, details are > included in the latest README file. > > Guozhang > > On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: > > > Thanks, Jay, > > > > Here is what I did this morning, I git clone the latest version of kafka > > from git, (I am currently using kafka 8.0) now it is 8.1.1, and it use > > gradle to build project. I am having trouble to build it. I installed > > gradle, and run ./gradlew jar in kafka root directory, it comes out: > > Error: Could not find or load main class > > org.gradle.wrapper.GradleWrapperMain > > > > Any idea about this. > > > > Thanks > > > > Alec > > > > On Wed, Oct 1, 2014 at 9:21 PM, Jay Kreps wrote: > > > > > Hi Sa, > > > > > > That script was developed with the new producer that is included on > > > trunk. Checkout trunk and build and it should be there. > > > > > > -Jay > > > > > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > > > > Hi, All > > > > > > > > I built a 3-node kafka cluster, I want to make performance test, I > > found > > > someone post following thread, that is exactly the problem I have: > > > > - > > > > While testing kafka producer performance, I found 2 testing scripts. > > > > > > > > 1) performance testing script in kafka distribution > > > > > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 > --messages > > > > 1000 --topic test --threads 10 --message-size 100 --batch-size > > 1 > > > > --compression-codec 1 > > > > > > > > 2) performance testing script mentioned in > > > > > > > > > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > > > > > bin/kafka-run-class.sh > > > > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 100 > > > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > > buffer.memory=67108864 batch.size=8196 > > > > > > > > based on org.apache.kafka.clients.producer.Producer. > > > > > > > > —— > > > > > > > > > > > > I was unable to duplicate either of above method, I figure the > commands > > > are outdated, anyone point me how to do such test with new command? > > > > > > > > > > > > thanks > > > > > > > > Alec > > > > > > > > > > > -- > > > > Alec Li > > > > > > -- > -- Guozhang > -- Alec Li
Re: kafka producer performance test
Hello Sa, KAFKA-1490 introduces a new step of downloading the wrapper, details are included in the latest README file. Guozhang On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: > Thanks, Jay, > > Here is what I did this morning, I git clone the latest version of kafka > from git, (I am currently using kafka 8.0) now it is 8.1.1, and it use > gradle to build project. I am having trouble to build it. I installed > gradle, and run ./gradlew jar in kafka root directory, it comes out: > Error: Could not find or load main class > org.gradle.wrapper.GradleWrapperMain > > Any idea about this. > > Thanks > > Alec > > On Wed, Oct 1, 2014 at 9:21 PM, Jay Kreps wrote: > > > Hi Sa, > > > > That script was developed with the new producer that is included on > > trunk. Checkout trunk and build and it should be there. > > > > -Jay > > > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > > > Hi, All > > > > > > I built a 3-node kafka cluster, I want to make performance test, I > found > > someone post following thread, that is exactly the problem I have: > > > - > > > While testing kafka producer performance, I found 2 testing scripts. > > > > > > 1) performance testing script in kafka distribution > > > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > > > 1000 --topic test --threads 10 --message-size 100 --batch-size > 1 > > > --compression-codec 1 > > > > > > 2) performance testing script mentioned in > > > > > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > > > bin/kafka-run-class.sh > > > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 100 > > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > buffer.memory=67108864 batch.size=8196 > > > > > > based on org.apache.kafka.clients.producer.Producer. > > > > > > —— > > > > > > > > > I was unable to duplicate either of above method, I figure the commands > > are outdated, anyone point me how to do such test with new command? > > > > > > > > > thanks > > > > > > Alec > > > > > > -- > > Alec Li > -- -- Guozhang
Re: kafka producer performance test
Thanks, Jay, Here is what I did this morning, I git clone the latest version of kafka from git, (I am currently using kafka 8.0) now it is 8.1.1, and it use gradle to build project. I am having trouble to build it. I installed gradle, and run ./gradlew jar in kafka root directory, it comes out: Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain Any idea about this. Thanks Alec On Wed, Oct 1, 2014 at 9:21 PM, Jay Kreps wrote: > Hi Sa, > > That script was developed with the new producer that is included on > trunk. Checkout trunk and build and it should be there. > > -Jay > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > > Hi, All > > > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following thread, that is exactly the problem I have: > > - > > While testing kafka producer performance, I found 2 testing scripts. > > > > 1) performance testing script in kafka distribution > > > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > > 1000 --topic test --threads 10 --message-size 100 --batch-size 1 > > --compression-codec 1 > > > > 2) performance testing script mentioned in > > > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > > > bin/kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 100 > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > buffer.memory=67108864 batch.size=8196 > > > > based on org.apache.kafka.clients.producer.Producer. > > > > —— > > > > > > I was unable to duplicate either of above method, I figure the commands > are outdated, anyone point me how to do such test with new command? > > > > > > thanks > > > > Alec > -- Alec Li
Re: kafka producer performance test
Hi Sa, That script was developed with the new producer that is included on trunk. Checkout trunk and build and it should be there. -Jay On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > Hi, All > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following thread, that is exactly the problem I have: > - > While testing kafka producer performance, I found 2 testing scripts. > > 1) performance testing script in kafka distribution > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > 1000 --topic test --threads 10 --message-size 100 --batch-size 1 > --compression-codec 1 > > 2) performance testing script mentioned in > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > bin/kafka-run-class.sh > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 100 > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > buffer.memory=67108864 batch.size=8196 > > based on org.apache.kafka.clients.producer.Producer. > > —— > > > I was unable to duplicate either of above method, I figure the commands are > outdated, anyone point me how to do such test with new command? > > > thanks > > Alec
Re: kafka producer performance test
Hi, Ravi Thanks for reply, this is how I build the kafka package 0.8 $ git clone https://git-wip-us.apache.org/repos/asf/kafka.git $ cd /etc/kafka $ git checkout -b 0.8 remotes/origin/0.8 $ ./sbt update $ ./sbt package $ ./sbt assembly-package-dependency So I believe I already build it, but still not able to run it, any clues for that? thanks Alec On Oct 1, 2014, at 9:13 PM, ravi singh wrote: > It is available with Kafka package containing the source code. Download > the package, build it and run the above command. > > Regards, > Ravi > > On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > >> Hi, All >> >> I built a 3-node kafka cluster, I want to make performance test, I found >> someone post following thread, that is exactly the problem I have: >> - >> While testing kafka producer performance, I found 2 testing scripts. >> >> 1) performance testing script in kafka distribution >> >> bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages >> 1000 --topic test --threads 10 --message-size 100 --batch-size 1 >> --compression-codec 1 >> >> 2) performance testing script mentioned in >> >> >> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines >> >> bin/kafka-run-class.sh >> org.apache.kafka.clients.tools.ProducerPerformance test6 5000 100 >> -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 >> buffer.memory=67108864 batch.size=8196 >> >> based on org.apache.kafka.clients.producer.Producer. >> >> —— >> >> >> I was unable to duplicate either of above method, I figure the commands >> are outdated, anyone point me how to do such test with new command? >> >> >> thanks >> >> Alec > > > > > -- > *Regards,* > *Ravi*
Re: kafka producer performance test
It is available with Kafka package containing the source code. Download the package, build it and run the above command. Regards, Ravi On Wed, Oct 1, 2014 at 7:55 PM, Sa Li wrote: > Hi, All > > I built a 3-node kafka cluster, I want to make performance test, I found > someone post following thread, that is exactly the problem I have: > - > While testing kafka producer performance, I found 2 testing scripts. > > 1) performance testing script in kafka distribution > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages > 1000 --topic test --threads 10 --message-size 100 --batch-size 1 > --compression-codec 1 > > 2) performance testing script mentioned in > > > https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines > > bin/kafka-run-class.sh > org.apache.kafka.clients.tools.ProducerPerformance test6 5000 100 > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > buffer.memory=67108864 batch.size=8196 > > based on org.apache.kafka.clients.producer.Producer. > > —— > > > I was unable to duplicate either of above method, I figure the commands > are outdated, anyone point me how to do such test with new command? > > > thanks > > Alec -- *Regards,* *Ravi*
Re: Kafka producer performance test sending 0x0 byte messages
Jun, let me see if I can fix first and then will submit back. Daniel, I was looking at the code some more and was thinking this might work https://github.com/apache/kafka/blob/0.8.1/perf/src/main/scala/kafka/perf/ProducerPerformance.scala on line 246 instead of looping to create messages I could open a sample file and add the rows as messages lien by line until I hit configured message cap. If I hit end of file I would start at the top. I think I can figure this out. Bert On Mon, Jun 30, 2014 at 1:46 PM, Daniel Compton wrote: > Hi Bert > > What you are describing could be done partially with the console producer. > It will read from a file and send each line to the Kafka broker. You could > make a really big file or alter that code to repeat a certain number of > times. The source is pretty readable, I think that might be an easier route > to take. > > Daniel. > > > On 1/07/2014, at 2:07 am, Bert Corderman wrote: > > > > Daniel, > > > > > > > > We have the same question. We noticed that the compression tests we ran > > using the built in performance tester was not realistic. I think on disk > > compression was 200:1. (yes that is two hundred to one) I had planned to > > try and edit the producer performance tester source and do the following > > > > > > > > 1. Add an option to read sample data from provided text file. > > (thought would be to add a file with 1-5000 rows, whatever I thought my > > batch size might be) > > > > 2. Load sample file into array > > > > 3. Change code that creates message to pull a random row from array > > > > > > > > I also am not a Scala developer so would take me a little bit to figure > > this out. This is on hold right now as I am looking at options of > > compression of the message before sending to kafka. We had originally > not > > wanted to do this as we are assuming that we would not get efficient > > compression ratios as we are only doing a single message however we are > > also talking about sending multiple messages from our application as a > > single Kafka message. Our concern with using kafka compression is the > > overhead required from decompression on the broker to assign Ids. Here > is > > a good article that describes this > > > http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/ > > > > > > > > But again we haven’t decided just yet. Would like to test and evaluate. > > > > > > > > Bert > > > > > > On Mon, Jun 30, 2014 at 2:24 AM, Daniel Compton > > wrote: > > > >> Hi folks > >> > >> I was doing some performance testing using the built in Kafka > performance > >> tester and it seems like it sends messages of size n bytes but with all > >> bytes having the value 0x0. Is that correct? Reading the source seemed > to > >> indicate that too but I'm not a Scala developer so I could be wrong. > >> > >> Would this affect the performance compared to a real world scenario? > >> Obviously you will get very efficient compression rates but apart from > >> that, is there likely to be optimisations carried out anywhere between > the > >> JVM and the network card that won't hold for messages with non zero > entropy? > >> > >> We're going to test this against our production workload so it's not a > big > >> deal for us but I wondered if this could give others skewed results? > >> > >> --- > >> Daniel >
Re: Kafka producer performance test sending 0x0 byte messages
Hi Bert What you are describing could be done partially with the console producer. It will read from a file and send each line to the Kafka broker. You could make a really big file or alter that code to repeat a certain number of times. The source is pretty readable, I think that might be an easier route to take. Daniel. > On 1/07/2014, at 2:07 am, Bert Corderman wrote: > > Daniel, > > > > We have the same question. We noticed that the compression tests we ran > using the built in performance tester was not realistic. I think on disk > compression was 200:1. (yes that is two hundred to one) I had planned to > try and edit the producer performance tester source and do the following > > > > 1. Add an option to read sample data from provided text file. > (thought would be to add a file with 1-5000 rows, whatever I thought my > batch size might be) > > 2. Load sample file into array > > 3. Change code that creates message to pull a random row from array > > > > I also am not a Scala developer so would take me a little bit to figure > this out. This is on hold right now as I am looking at options of > compression of the message before sending to kafka. We had originally not > wanted to do this as we are assuming that we would not get efficient > compression ratios as we are only doing a single message however we are > also talking about sending multiple messages from our application as a > single Kafka message. Our concern with using kafka compression is the > overhead required from decompression on the broker to assign Ids. Here is > a good article that describes this > http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/ > > > > But again we haven’t decided just yet. Would like to test and evaluate. > > > > Bert > > > On Mon, Jun 30, 2014 at 2:24 AM, Daniel Compton > wrote: > >> Hi folks >> >> I was doing some performance testing using the built in Kafka performance >> tester and it seems like it sends messages of size n bytes but with all >> bytes having the value 0x0. Is that correct? Reading the source seemed to >> indicate that too but I'm not a Scala developer so I could be wrong. >> >> Would this affect the performance compared to a real world scenario? >> Obviously you will get very efficient compression rates but apart from >> that, is there likely to be optimisations carried out anywhere between the >> JVM and the network card that won't hold for messages with non zero entropy? >> >> We're going to test this against our production workload so it's not a big >> deal for us but I wondered if this could give others skewed results? >> >> --- >> Daniel
Re: Kafka producer performance test sending 0x0 byte messages
Yes, this is a problem and will indeed affect the producer performance when compression is turned on. Perhaps we should fill in the values with some randomized bytes. Could you file a jira for this? Thanks, Jun On Sun, Jun 29, 2014 at 11:24 PM, Daniel Compton wrote: > Hi folks > > I was doing some performance testing using the built in Kafka performance > tester and it seems like it sends messages of size n bytes but with all > bytes having the value 0x0. Is that correct? Reading the source seemed to > indicate that too but I'm not a Scala developer so I could be wrong. > > Would this affect the performance compared to a real world scenario? > Obviously you will get very efficient compression rates but apart from > that, is there likely to be optimisations carried out anywhere between the > JVM and the network card that won't hold for messages with non zero entropy? > > We're going to test this against our production workload so it's not a big > deal for us but I wondered if this could give others skewed results? > > --- > Daniel
Re: Kafka producer performance test sending 0x0 byte messages
Daniel, We have the same question. We noticed that the compression tests we ran using the built in performance tester was not realistic. I think on disk compression was 200:1. (yes that is two hundred to one) I had planned to try and edit the producer performance tester source and do the following 1. Add an option to read sample data from provided text file. (thought would be to add a file with 1-5000 rows, whatever I thought my batch size might be) 2. Load sample file into array 3. Change code that creates message to pull a random row from array I also am not a Scala developer so would take me a little bit to figure this out. This is on hold right now as I am looking at options of compression of the message before sending to kafka. We had originally not wanted to do this as we are assuming that we would not get efficient compression ratios as we are only doing a single message however we are also talking about sending multiple messages from our application as a single Kafka message. Our concern with using kafka compression is the overhead required from decompression on the broker to assign Ids. Here is a good article that describes this http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/ But again we haven’t decided just yet. Would like to test and evaluate. Bert On Mon, Jun 30, 2014 at 2:24 AM, Daniel Compton wrote: > Hi folks > > I was doing some performance testing using the built in Kafka performance > tester and it seems like it sends messages of size n bytes but with all > bytes having the value 0x0. Is that correct? Reading the source seemed to > indicate that too but I'm not a Scala developer so I could be wrong. > > Would this affect the performance compared to a real world scenario? > Obviously you will get very efficient compression rates but apart from > that, is there likely to be optimisations carried out anywhere between the > JVM and the network card that won't hold for messages with non zero entropy? > > We're going to test this against our production workload so it's not a big > deal for us but I wondered if this could give others skewed results? > > --- > Daniel
