Re: kafka-producer-perf-test.sh transaction support?
Hi Anindya, Currently, setting transaction.id in producer props won't enable transactions. Users need to set --transaction-duration-ms to enable transactions in kafka-producer-perf-test. We recognize this can be confusing, so there's a JIRA ticket for it: https://issues.apache.org/jira/browse/KAFKA-16900 I've submitted a pull request to address this issue. With the update, users can enable transactions by either: - Setting transaction.id= via --producer-props - Setting transaction.id= in the config file via --producer.config - Setting --transaction-id - Setting --transaction-duration-ms= I believe this will resolve your issue. If you have any thoughts or suggestions, please feel free to share them on the PR or JIRA. On 2019/10/18 19:18:15 Anindya Haldar wrote: > Anyone who might have experienced this, or have a known solution? Would > appreciate some insights here. > > Sincerely, > Anindya Haldar > Oracle Responsys > > > > On Oct 17, 2019, at 5:38 PM, Anindya Haldar > > wrote: > > > > I am trying to test a 3 node Kafka cluster using the producer and consumer > > test perf scripts that ship with the binaries. When I am trying to enable > > transaction in the producer script it runs into errors. For example: > > > > # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1 > > --throughput -1 --record-size 256 --producer-props > > bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all > > transactional.id= enable.idempotence=true > > Exception in thread "main" java.lang.IllegalStateException: Cannot perform > > a 'send' before completing a call to initTransactions when transactions are > > enabled. > > at > > org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:362) > > at > > org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341) > > at > > org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915) > > at > > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856) > > at > > org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143) > > > > A non-transactional test succeeds though: > > > > bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1 > > --throughput -1 --record-size 256 --producer-props > > bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all > > 1 records sent, 6997.900630 records/sec (1.71 MB/sec), 112.32 ms avg > > latency, 499.00 ms max latency, 101 ms 50th, 206 ms 95th, 213 ms 99th, 214 > > ms 99.9th. > > > > The version I am using is: kafka_2.12-2.3.0 > > > > Does the producer perf test script support transactions? If it does, how do > > we run the producer with transaction support? > > > > Sincerely, > > Anindya Haldar > > Oracle Responsys > > > >
Re: kafka-producer-perf-test.sh maximum throughput limits for broker on single node
Hi Tomasz, There are some configurations needed to be tuned in producer side. Try searching kafka producer tuning, you should find many good articles describing it. Good luck. Luke On Fri, Feb 24, 2023 at 6:16 AM Tomasz Sąciński < tomasz.sacin...@linuxpolska.pl> wrote: > Hello users. > > I test max throughput to a single kafka broker but results do not match > speed limits disk or network. > > /kafka-producer-perf-test.sh --topic perf-test --num-records 3000 > --record-size 100 --throughput -1 --producer-props > bootstrap.servers=localhost:9092 > > 3000 records sent, 98.872850 records/sec (94.29 MB/sec), 13.51 ms avg > latency, 528.00 ms max latency, 9 ms 50th, 15 ms 95th, 162 ms 99th, 350 ms > 99.9th. > > Smaller record size gets lower throughput. > > Is upper limit throughput for broker or something another limit throughput? > > What is bottleneck for this? > > There are detailed information: > > hardware: > CPU: 6230R 2.1GHz 2x26core, > RAM: 192 GB, > disks SATA SSD – 20 x1,92T (test run on single disk) > > > Java version: > java-11-openjdk-11.0.17.0.8-2.el8_6.x86_64 > > > Kafka version: ./kafka-topics.sh --version > > 3.3.1 (Commit:e23c59d00e687ff5) > > kafka_2.13-3.3.1.jar > > Network speed tested by iperf3: 5,46 GB/s (testing kafka both localhost and > second node) > Disk speed: > > hdparm -tT /dev/mapper/vg1-kafka > > /dev/mapper/vg1-kafka: > > Timing cached reads: 19176 MB in 2.00 seconds = 9600.40 MB/sec > > Timing buffered disk reads: 10338 MB in 3.00 seconds = 3445.37 MB/sec > > > > writes: > > dd if=/dev/zero of=/kafka/test1.img bs=1G count=1 oflag=dsync > > 1+0 records in > > 1+0 records out > > 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.13398 s, 947 MB/s > > Testing both on default jvm settings and with 6GB RAM: > >-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 >-XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M >-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 >-XX:+ExplicitGCInvokesConcurrent > > Kafka installed and ran from instructions > https://kafka.apache.org/quickstart on a single node. > > > Pozdrawiam/Regards > > > > Tomasz Sąciński > > -- > <https://linuxpolska.pl/> > > > Linux Polska Sp. z o.o. > > Al. Jerozolimskie > 100, 00-807 Warszawa > < > https://www.google.com/maps/place/Linux+Polska+Sp.+z+o.+o./@52.221986,20.9744718,17z/data=!3m1!4b1!4m5!3m4!1s0x0:0x5756147639e870d6!8m2!3d52.221986!4d20.9766605?authuser=2 > > > > tel. +48 22 213 95 71, fax +48 22 213 96 71 > > KRS 0326158, Sąd Rejonowy > dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS > > Kapitał > zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601 > > > > www.linuxpolska.pl <https://linuxpolska.pl/> | > <https://www.linkedin.com/company/linux-polska/> > <https://www.facebook.com/linuxpolskapl> > <https://twitter.com/linuxpolska> > > _ > > > > > This message may contain confidential information that is covered by > legal privilege. If you are not the intended recipient or if you have > received this message by mistake, please notify the sender immediately and > delete this e-mail and its attachments from your system. Any unauthorized > copying, disclosure or distribution of the material in this e-mail and its > attachments is strictly forbidden. > > > -- > <https://linuxpolska.pl/baner-lp/> >
kafka-producer-perf-test.sh maximum throughput limits for broker on single node
Hello users. I test max throughput to a single kafka broker but results do not match speed limits disk or network. /kafka-producer-perf-test.sh --topic perf-test --num-records 3000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=localhost:9092 3000 records sent, 98.872850 records/sec (94.29 MB/sec), 13.51 ms avg latency, 528.00 ms max latency, 9 ms 50th, 15 ms 95th, 162 ms 99th, 350 ms 99.9th. Smaller record size gets lower throughput. Is upper limit throughput for broker or something another limit throughput? What is bottleneck for this? There are detailed information: hardware: CPU: 6230R 2.1GHz 2x26core, RAM: 192 GB, disks SATA SSD – 20 x1,92T (test run on single disk) Java version: java-11-openjdk-11.0.17.0.8-2.el8_6.x86_64 Kafka version: ./kafka-topics.sh --version 3.3.1 (Commit:e23c59d00e687ff5) kafka_2.13-3.3.1.jar Network speed tested by iperf3: 5,46 GB/s (testing kafka both localhost and second node) Disk speed: hdparm -tT /dev/mapper/vg1-kafka /dev/mapper/vg1-kafka: Timing cached reads: 19176 MB in 2.00 seconds = 9600.40 MB/sec Timing buffered disk reads: 10338 MB in 3.00 seconds = 3445.37 MB/sec writes: dd if=/dev/zero of=/kafka/test1.img bs=1G count=1 oflag=dsync 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.13398 s, 947 MB/s Testing both on default jvm settings and with 6GB RAM: -Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+ExplicitGCInvokesConcurrent Kafka installed and ran from instructions https://kafka.apache.org/quickstart on a single node. Pozdrawiam/Regards Tomasz Sąciński -- <https://linuxpolska.pl/> Linux Polska Sp. z o.o. Al. Jerozolimskie 100, 00-807 Warszawa <https://www.google.com/maps/place/Linux+Polska+Sp.+z+o.+o./@52.221986,20.9744718,17z/data=!3m1!4b1!4m5!3m4!1s0x0:0x5756147639e870d6!8m2!3d52.221986!4d20.9766605?authuser=2> tel. +48 22 213 95 71, fax +48 22 213 96 71 KRS 0326158, Sąd Rejonowy dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS Kapitał zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601 www.linuxpolska.pl <https://linuxpolska.pl/> | <https://www.linkedin.com/company/linux-polska/> <https://www.facebook.com/linuxpolskapl> <https://twitter.com/linuxpolska> _ This message may contain confidential information that is covered by legal privilege. If you are not the intended recipient or if you have received this message by mistake, please notify the sender immediately and delete this e-mail and its attachments from your system. Any unauthorized copying, disclosure or distribution of the material in this e-mail and its attachments is strictly forbidden. -- <https://linuxpolska.pl/baner-lp/>
Re: kafka-producer-perf-test.sh transaction support?
Anyone who might have experienced this, or have a known solution? Would appreciate some insights here. Sincerely, Anindya Haldar Oracle Responsys > On Oct 17, 2019, at 5:38 PM, Anindya Haldar wrote: > > I am trying to test a 3 node Kafka cluster using the producer and consumer > test perf scripts that ship with the binaries. When I am trying to enable > transaction in the producer script it runs into errors. For example: > > # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1 > --throughput -1 --record-size 256 --producer-props > bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all > transactional.id= enable.idempotence=true > Exception in thread "main" java.lang.IllegalStateException: Cannot perform a > 'send' before completing a call to initTransactions when transactions are > enabled. > at > org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:362) > at > org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341) > at > org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915) > at > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856) > at > org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143) > > A non-transactional test succeeds though: > > bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1 > --throughput -1 --record-size 256 --producer-props > bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all > 1 records sent, 6997.900630 records/sec (1.71 MB/sec), 112.32 ms avg > latency, 499.00 ms max latency, 101 ms 50th, 206 ms 95th, 213 ms 99th, 214 ms > 99.9th. > > The version I am using is: kafka_2.12-2.3.0 > > Does the producer perf test script support transactions? If it does, how do > we run the producer with transaction support? > > Sincerely, > Anindya Haldar > Oracle Responsys >
kafka-producer-perf-test.sh transaction support?
I am trying to test a 3 node Kafka cluster using the producer and consumer test perf scripts that ship with the binaries. When I am trying to enable transaction in the producer script it runs into errors. For example: # bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1 --throughput -1 --record-size 256 --producer-props bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all transactional.id= enable.idempotence=true Exception in thread "main" java.lang.IllegalStateException: Cannot perform a 'send' before completing a call to initTransactions when transactions are enabled. at org.apache.kafka.clients.producer.internals.TransactionManager.failIfNotReadyForSend(TransactionManager.java:362) at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartitionToTransaction(TransactionManager.java:341) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:915) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856) at org.apache.kafka.tools.ProducerPerformance.main(ProducerPerformance.java:143) A non-transactional test succeeds though: bin/kafka-producer-perf-test.sh --topic testtopic1 --num-records 1 --throughput -1 --record-size 256 --producer-props bootstrap.servers=172.18.0.4:9092,172.18.0.5:9092,172.18.0.6:9092 acks=all 1 records sent, 6997.900630 records/sec (1.71 MB/sec), 112.32 ms avg latency, 499.00 ms max latency, 101 ms 50th, 206 ms 95th, 213 ms 99th, 214 ms 99.9th. The version I am using is: kafka_2.12-2.3.0 Does the producer perf test script support transactions? If it does, how do we run the producer with transaction support? Sincerely, Anindya Haldar Oracle Responsys
Re: Errors observed in performance test using kafka-producer-perf-test.sh|
I have another very disturbing observation. The errors go away if I start 2 kafka-producer-perf-test.sh with the same configs on different hosts. If I cancel 1 kafka-producer-perf-test.sh then after some time the below errors start reappearing. org.apache.kafka.common.errors.TimeoutException: The request timed out. org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) for benchmark-6-3r-2isr-none-0: 182806 ms has passed since last append --Unilocal On Wed, May 30, 2018 at 1:19 AM Localhost shell < universal.localh...@gmail.com> wrote: > Hello All, > > I am trying to perform a benchmark test in our kafka env. I have played > with few configurations such as request.timeout.ms and max.block.ms and > throughout but not able to avoid the error: > org.apache.kafka.common.errors.TimeoutException: The request timed out. > org.apache.kafka.common.errors.NetworkException: The server disconnected > before a response was received. > org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) > for benchmark-6-3r-2isr-none-0: 182806 ms has passed since last append > > Produce Perf Test command: > nohup sh ~/kafka/kafka_2.11-1.0.0/bin/kafka-producer-perf-test.sh --topic > benchmark-6p-3r-2isr-none --num-records 1000 --record-size 100 > --throughput -1 --print-metrics --producer-props acks=all > bootstrap.servers=node1:9092,node2:9092,node3:9092 request.timeout.ms=18 > max.block.ms=18 buffer.memory=1 > > ~/kafka/load_test/results/6p-3r-10M-100B-t-1-ackall-rto3m-block2m-bm100m-2 > 2>&1 > > Cluster: 3 nodes, topic: 6 partitions, RF=3 and minISR=2 > I am monitoring the kafka metrics using a tsdb and grafana. I know that > disk IO perf is bad [disk await(1.5 secs), IO queue size and disk > utilization metrics are high(60-75%)] but I don't see any issue in kafka > logs that can relate slow disk io to the above perf errors. > > I have even run the test with throughput=1000(all above params same) but > still get timeout exceptions. > > Need suggestions to understand the issue and fix the above errors? > > --Unilocal > -- --Unilocal
Errors observed in performance test using kafka-producer-perf-test.sh|
Hello All, I am trying to perform a benchmark test in our kafka env. I have played with few configurations such as request.timeout.ms and max.block.ms and throughout but not able to avoid the error: org.apache.kafka.common.errors.TimeoutException: The request timed out. org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) for benchmark-6-3r-2isr-none-0: 182806 ms has passed since last append Produce Perf Test command: nohup sh ~/kafka/kafka_2.11-1.0.0/bin/kafka-producer-perf-test.sh --topic benchmark-6p-3r-2isr-none --num-records 1000 --record-size 100 --throughput -1 --print-metrics --producer-props acks=all bootstrap.servers=node1:9092,node2:9092,node3:9092 request.timeout.ms=18 max.block.ms=18 buffer.memory=1 > ~/kafka/load_test/results/6p-3r-10M-100B-t-1-ackall-rto3m-block2m-bm100m-2 2>&1 Cluster: 3 nodes, topic: 6 partitions, RF=3 and minISR=2 I am monitoring the kafka metrics using a tsdb and grafana. I know that disk IO perf is bad [disk await(1.5 secs), IO queue size and disk utilization metrics are high(60-75%)] but I don't see any issue in kafka logs that can relate slow disk io to the above perf errors. I have even run the test with throughput=1000(all above params same) but still get timeout exceptions. Need suggestions to understand the issue and fix the above errors? --Unilocal
Re: kafka-producer-perf-test.sh - 0.8.2.1
Thanks! I also found old thread and Ewen did reply to that. So I will just re-post so somebody else might find it helpful: Ewen: ... EndToEndLatency works with a single message at a time. It produces the message then waits for the consumer to receive it. This approach guarantees there is no delay due to queuing. The goal with this test is to evaluate the *minimum* *latency*. *ProducerPerformance* focuses on achieving maximum throughput. This means it will enqueue lots of records so it will always have more data to send (and can use batching to increase the throughput). Unlike EndToEndLatency, this means records may just sit in a queue on the producer for awhile because the maximum number of in flight requests has been reached and it needs to wait for responses for those requests. Since EndToEndLatency only ever has one record outstanding, it will never encounter this case. ... Now it does make sense to me. Thanks for your help! Andrew On Mon, Jan 11, 2016 at 1:31 PM, Jay Kreps wrote: > If you use the perf test without any bound on throughput it will > always try to send data faster than it can go out and build up a queue > of unsent data. So e.g. if your buffer is 1MB each send will be > blocked on waiting for the full 1MB of queued data to clear out and > get sent. This makes sense if you think about it. > > If you want to test latency under load you need to throttle the > maximum throughput to something like what you think you would see in > your application (there is an option for that in the command line > options). > > -Jay > > On Mon, Jan 11, 2016 at 11:02 AM, Andrej Vladimirovich > wrote: > > Ewen, > > > > One more question. I mentioned that *kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance* latency is a lot > higher > > than *kafka-run-class.sh kafka.tools.TestEndToEndLatency.* > > > > Example: > > > > *ProducerPerformance:* > > > > 5000 records sent, 337463.891364 records/sec (32.18 MB/sec), *1548.51 > > ms avg latency*, 3186.00 ms max latency, 2478 ms 50th, 3071 ms 95th, 3118 > > ms 99th, 3179 ms 99.9th. > > > > *TestEndToEndLatency:* > > > > Percentiles: 50th = 8, 99th = 9, 99.9th = 20 > > > > So 1548.51 ms vs 9 ms.Huge difference. > > > > I am using the same cluster, same server and same topic to run both > tests. > > It does not make any sense to me why would End to End be so low and > > Producer to Kafka is so large? > > > > I did some research online and found other people having the same > question > > without any responses. > > > > Thanks a lot for your help! > > > > Andrew > > > > On Fri, Jan 8, 2016 at 5:44 PM, Ewen Cheslack-Postava > > > wrote: > > > >> It is single threaded in the sense that you can not request that > multiple > >> threads be used to call producer.send(). However, the producer has its > own > >> internal thread for doing network IO. When you have such a simple > producer, > >> depending on the size of messages you can saturate a 1Gbps link with a > >> single thread, so usually using more threads isn't much help. If you > still > >> need more throughput, you can just use more processes. > >> > >> -Ewen > >> > >> On Fri, Jan 8, 2016 at 1:24 PM, Andrej Vladimirovich < > >> udodizdu...@gmail.com> > >> wrote: > >> > >> > Thanks Ewen. Do you know if kafka-run-class.sh > >> > org.apache.kafka.clients.tools.ProducerPerformance > >> > is single threaded? Or is there any way to specify number of threads? > >> > > >> > On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava < > e...@confluent.io > >> > > >> > wrote: > >> > > >> > > Ah, sorry, I missed the version number in your title. I think this > tool > >> > saw > >> > > some rearrangement in 0.9.0 and I was looking at the latest version. > >> > > Unfortunately it doesn't look like the old > >> > kafka.tools.ProducerPerformance > >> > > that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports > passing > >> > in > >> > > additional properties. > >> > > > >> > > -Ewen > >> > > > >> > > On Fri, Jan 8, 2016 at 9:10 AM, Andrej Vladimirovich < > >> > > udodizdu...@gmail.com> > >> > > wrote: > >> > > > >> > > > Ewen, > >> > > > > >> > > > I tried that before like this: > >> > > > > >>
Re: kafka-producer-perf-test.sh - 0.8.2.1
If you use the perf test without any bound on throughput it will always try to send data faster than it can go out and build up a queue of unsent data. So e.g. if your buffer is 1MB each send will be blocked on waiting for the full 1MB of queued data to clear out and get sent. This makes sense if you think about it. If you want to test latency under load you need to throttle the maximum throughput to something like what you think you would see in your application (there is an option for that in the command line options). -Jay On Mon, Jan 11, 2016 at 11:02 AM, Andrej Vladimirovich wrote: > Ewen, > > One more question. I mentioned that *kafka-run-class.sh > org.apache.kafka.clients.tools.ProducerPerformance* latency is a lot higher > than *kafka-run-class.sh kafka.tools.TestEndToEndLatency.* > > Example: > > *ProducerPerformance:* > > 5000 records sent, 337463.891364 records/sec (32.18 MB/sec), *1548.51 > ms avg latency*, 3186.00 ms max latency, 2478 ms 50th, 3071 ms 95th, 3118 > ms 99th, 3179 ms 99.9th. > > *TestEndToEndLatency:* > > Percentiles: 50th = 8, 99th = 9, 99.9th = 20 > > So 1548.51 ms vs 9 ms.Huge difference. > > I am using the same cluster, same server and same topic to run both tests. > It does not make any sense to me why would End to End be so low and > Producer to Kafka is so large? > > I did some research online and found other people having the same question > without any responses. > > Thanks a lot for your help! > > Andrew > > On Fri, Jan 8, 2016 at 5:44 PM, Ewen Cheslack-Postava > wrote: > >> It is single threaded in the sense that you can not request that multiple >> threads be used to call producer.send(). However, the producer has its own >> internal thread for doing network IO. When you have such a simple producer, >> depending on the size of messages you can saturate a 1Gbps link with a >> single thread, so usually using more threads isn't much help. If you still >> need more throughput, you can just use more processes. >> >> -Ewen >> >> On Fri, Jan 8, 2016 at 1:24 PM, Andrej Vladimirovich < >> udodizdu...@gmail.com> >> wrote: >> >> > Thanks Ewen. Do you know if kafka-run-class.sh >> > org.apache.kafka.clients.tools.ProducerPerformance >> > is single threaded? Or is there any way to specify number of threads? >> > >> > On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava > > >> > wrote: >> > >> > > Ah, sorry, I missed the version number in your title. I think this tool >> > saw >> > > some rearrangement in 0.9.0 and I was looking at the latest version. >> > > Unfortunately it doesn't look like the old >> > kafka.tools.ProducerPerformance >> > > that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing >> > in >> > > additional properties. >> > > >> > > -Ewen >> > > >> > > On Fri, Jan 8, 2016 at 9:10 AM, Andrej Vladimirovich < >> > > udodizdu...@gmail.com> >> > > wrote: >> > > >> > > > Ewen, >> > > > >> > > > I tried that before like this: >> > > > >> > > > ./kafka-producer-perf-test.sh --broker-list test:9092 --topics >> test8-3 >> > > > --messages 200 --new-producer --message-size 200 >> > > > --show-detailed-stats max.request.size=1000 >> > > > >> > > > and it does not work. It comletly ignore this option. >> > > > >> > > > And --producer-props is not a valid option for >> > > kafka-producer-perf-test.sh. >> > > > Maybe it is not the right syntax? But I tried a lot of different ways >> > and >> > > > have yet to find the right one. >> > > > >> > > > Thanks! >> > > > >> > > > Andrew >> > > > >> > > > On Fri, Jan 8, 2016 at 10:54 AM, Ewen Cheslack-Postava < >> > > e...@confluent.io> >> > > > wrote: >> > > > >> > > > > Andrew, >> > > > > >> > > > > kafka-producer-perf-test.sh is just a wrapper around >> > > > > orga.apache.kafka.clients.tools.ProducerPerformance and all command >> > > line >> > > > > options should be forwarded. Can you just pass a --producer-props >> to >> > > set >> > > > > max.request.size to a larger value? >> > > > > >> > > > > -Ewen >> > > >
Re: kafka-producer-perf-test.sh - 0.8.2.1
Ewen, One more question. I mentioned that *kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance* latency is a lot higher than *kafka-run-class.sh kafka.tools.TestEndToEndLatency.* Example: *ProducerPerformance:* 5000 records sent, 337463.891364 records/sec (32.18 MB/sec), *1548.51 ms avg latency*, 3186.00 ms max latency, 2478 ms 50th, 3071 ms 95th, 3118 ms 99th, 3179 ms 99.9th. *TestEndToEndLatency:* Percentiles: 50th = 8, 99th = 9, 99.9th = 20 So 1548.51 ms vs 9 ms.Huge difference. I am using the same cluster, same server and same topic to run both tests. It does not make any sense to me why would End to End be so low and Producer to Kafka is so large? I did some research online and found other people having the same question without any responses. Thanks a lot for your help! Andrew On Fri, Jan 8, 2016 at 5:44 PM, Ewen Cheslack-Postava wrote: > It is single threaded in the sense that you can not request that multiple > threads be used to call producer.send(). However, the producer has its own > internal thread for doing network IO. When you have such a simple producer, > depending on the size of messages you can saturate a 1Gbps link with a > single thread, so usually using more threads isn't much help. If you still > need more throughput, you can just use more processes. > > -Ewen > > On Fri, Jan 8, 2016 at 1:24 PM, Andrej Vladimirovich < > udodizdu...@gmail.com> > wrote: > > > Thanks Ewen. Do you know if kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance > > is single threaded? Or is there any way to specify number of threads? > > > > On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava > > > wrote: > > > > > Ah, sorry, I missed the version number in your title. I think this tool > > saw > > > some rearrangement in 0.9.0 and I was looking at the latest version. > > > Unfortunately it doesn't look like the old > > kafka.tools.ProducerPerformance > > > that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing > > in > > > additional properties. > > > > > > -Ewen > > > > > > On Fri, Jan 8, 2016 at 9:10 AM, Andrej Vladimirovich < > > > udodizdu...@gmail.com> > > > wrote: > > > > > > > Ewen, > > > > > > > > I tried that before like this: > > > > > > > > ./kafka-producer-perf-test.sh --broker-list test:9092 --topics > test8-3 > > > > --messages 200 --new-producer --message-size 200 > > > > --show-detailed-stats max.request.size=1000 > > > > > > > > and it does not work. It comletly ignore this option. > > > > > > > > And --producer-props is not a valid option for > > > kafka-producer-perf-test.sh. > > > > Maybe it is not the right syntax? But I tried a lot of different ways > > and > > > > have yet to find the right one. > > > > > > > > Thanks! > > > > > > > > Andrew > > > > > > > > On Fri, Jan 8, 2016 at 10:54 AM, Ewen Cheslack-Postava < > > > e...@confluent.io> > > > > wrote: > > > > > > > > > Andrew, > > > > > > > > > > kafka-producer-perf-test.sh is just a wrapper around > > > > > orga.apache.kafka.clients.tools.ProducerPerformance and all command > > > line > > > > > options should be forwarded. Can you just pass a --producer-props > to > > > set > > > > > max.request.size to a larger value? > > > > > > > > > > -Ewen > > > > > > > > > > On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimirovich < > > > > > udodizdu...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi! > > > > > > > > > > > > I am testing Kafka's performance with large messages and would > like > > > to > > > > > > specify maximum request size when I run > > kafka-producer-perf-test.sh: > > > > > > > > > > > > ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics > > > > test8-3 > > > > > > --messages 100 --new-producer --msage-size 110 > > > > > > --show-detailed-stats > > > > > > > > > > > > I always get this message if I specify somethine larger than 1MB: > > > > > > > > > > > > ERROR Error when sending message to topic test8-3 with key: 1 > > bytes, > > > > > value: > > > > > > 110 bytes with error: The message is 1100027 bytes when > > > serialized > > > > > > which is larger than the maximum request size you have configured > > > with > > > > > the > > > > > > max.request.size configuration. > > > > > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > > > > > > > > > > > I know I can specify maximum request size with kafka-run-class.sh > > > > > > org.apache.kafka.clients.tools.ProducerPerformance but I would > like > > > to > > > > > use > > > > > > kafka-producer-perf-test.sh if possible. > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Ewen > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Ewen > > > > > > > > > -- > Thanks, > Ewen >
Re: kafka-producer-perf-test.sh - 0.8.2.1
It is single threaded in the sense that you can not request that multiple threads be used to call producer.send(). However, the producer has its own internal thread for doing network IO. When you have such a simple producer, depending on the size of messages you can saturate a 1Gbps link with a single thread, so usually using more threads isn't much help. If you still need more throughput, you can just use more processes. -Ewen On Fri, Jan 8, 2016 at 1:24 PM, Andrej Vladimirovich wrote: > Thanks Ewen. Do you know if kafka-run-class.sh > org.apache.kafka.clients.tools.ProducerPerformance > is single threaded? Or is there any way to specify number of threads? > > On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava > wrote: > > > Ah, sorry, I missed the version number in your title. I think this tool > saw > > some rearrangement in 0.9.0 and I was looking at the latest version. > > Unfortunately it doesn't look like the old > kafka.tools.ProducerPerformance > > that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing > in > > additional properties. > > > > -Ewen > > > > On Fri, Jan 8, 2016 at 9:10 AM, Andrej Vladimirovich < > > udodizdu...@gmail.com> > > wrote: > > > > > Ewen, > > > > > > I tried that before like this: > > > > > > ./kafka-producer-perf-test.sh --broker-list test:9092 --topics test8-3 > > > --messages 200 --new-producer --message-size 200 > > > --show-detailed-stats max.request.size=1000 > > > > > > and it does not work. It comletly ignore this option. > > > > > > And --producer-props is not a valid option for > > kafka-producer-perf-test.sh. > > > Maybe it is not the right syntax? But I tried a lot of different ways > and > > > have yet to find the right one. > > > > > > Thanks! > > > > > > Andrew > > > > > > On Fri, Jan 8, 2016 at 10:54 AM, Ewen Cheslack-Postava < > > e...@confluent.io> > > > wrote: > > > > > > > Andrew, > > > > > > > > kafka-producer-perf-test.sh is just a wrapper around > > > > orga.apache.kafka.clients.tools.ProducerPerformance and all command > > line > > > > options should be forwarded. Can you just pass a --producer-props to > > set > > > > max.request.size to a larger value? > > > > > > > > -Ewen > > > > > > > > On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimirovich < > > > > udodizdu...@gmail.com> > > > > wrote: > > > > > > > > > Hi! > > > > > > > > > > I am testing Kafka's performance with large messages and would like > > to > > > > > specify maximum request size when I run > kafka-producer-perf-test.sh: > > > > > > > > > > ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics > > > test8-3 > > > > > --messages 100 --new-producer --msage-size 110 > > > > > --show-detailed-stats > > > > > > > > > > I always get this message if I specify somethine larger than 1MB: > > > > > > > > > > ERROR Error when sending message to topic test8-3 with key: 1 > bytes, > > > > value: > > > > > 110 bytes with error: The message is 1100027 bytes when > > serialized > > > > > which is larger than the maximum request size you have configured > > with > > > > the > > > > > max.request.size configuration. > > > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > > > > > > > > > I know I can specify maximum request size with kafka-run-class.sh > > > > > org.apache.kafka.clients.tools.ProducerPerformance but I would like > > to > > > > use > > > > > kafka-producer-perf-test.sh if possible. > > > > > > > > > > Thanks! > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Ewen > > > > > > > > > > > > > > > -- > > Thanks, > > Ewen > > > -- Thanks, Ewen
Re: kafka-producer-perf-test.sh - 0.8.2.1
Thanks Ewen. Do you know if kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance is single threaded? Or is there any way to specify number of threads? On Fri, Jan 8, 2016 at 1:24 PM, Ewen Cheslack-Postava wrote: > Ah, sorry, I missed the version number in your title. I think this tool saw > some rearrangement in 0.9.0 and I was looking at the latest version. > Unfortunately it doesn't look like the old kafka.tools.ProducerPerformance > that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing in > additional properties. > > -Ewen > > On Fri, Jan 8, 2016 at 9:10 AM, Andrej Vladimirovich < > udodizdu...@gmail.com> > wrote: > > > Ewen, > > > > I tried that before like this: > > > > ./kafka-producer-perf-test.sh --broker-list test:9092 --topics test8-3 > > --messages 200 --new-producer --message-size 200 > > --show-detailed-stats max.request.size=1000 > > > > and it does not work. It comletly ignore this option. > > > > And --producer-props is not a valid option for > kafka-producer-perf-test.sh. > > Maybe it is not the right syntax? But I tried a lot of different ways and > > have yet to find the right one. > > > > Thanks! > > > > Andrew > > > > On Fri, Jan 8, 2016 at 10:54 AM, Ewen Cheslack-Postava < > e...@confluent.io> > > wrote: > > > > > Andrew, > > > > > > kafka-producer-perf-test.sh is just a wrapper around > > > orga.apache.kafka.clients.tools.ProducerPerformance and all command > line > > > options should be forwarded. Can you just pass a --producer-props to > set > > > max.request.size to a larger value? > > > > > > -Ewen > > > > > > On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimirovich < > > > udodizdu...@gmail.com> > > > wrote: > > > > > > > Hi! > > > > > > > > I am testing Kafka's performance with large messages and would like > to > > > > specify maximum request size when I run kafka-producer-perf-test.sh: > > > > > > > > ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics > > test8-3 > > > > --messages 100 --new-producer --msage-size 110 > > > > --show-detailed-stats > > > > > > > > I always get this message if I specify somethine larger than 1MB: > > > > > > > > ERROR Error when sending message to topic test8-3 with key: 1 bytes, > > > value: > > > > 110 bytes with error: The message is 1100027 bytes when > serialized > > > > which is larger than the maximum request size you have configured > with > > > the > > > > max.request.size configuration. > > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > > > > > > > I know I can specify maximum request size with kafka-run-class.sh > > > > org.apache.kafka.clients.tools.ProducerPerformance but I would like > to > > > use > > > > kafka-producer-perf-test.sh if possible. > > > > > > > > Thanks! > > > > > > > > Andrew > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Ewen > > > > > > > > > -- > Thanks, > Ewen >
Re: kafka-producer-perf-test.sh - 0.8.2.1
Ah, sorry, I missed the version number in your title. I think this tool saw some rearrangement in 0.9.0 and I was looking at the latest version. Unfortunately it doesn't look like the old kafka.tools.ProducerPerformance that is used in kafka-producer-perf-test.sh in 0.8.2.1 supports passing in additional properties. -Ewen On Fri, Jan 8, 2016 at 9:10 AM, Andrej Vladimirovich wrote: > Ewen, > > I tried that before like this: > > ./kafka-producer-perf-test.sh --broker-list test:9092 --topics test8-3 > --messages 200 --new-producer --message-size 200 > --show-detailed-stats max.request.size=1000 > > and it does not work. It comletly ignore this option. > > And --producer-props is not a valid option for kafka-producer-perf-test.sh. > Maybe it is not the right syntax? But I tried a lot of different ways and > have yet to find the right one. > > Thanks! > > Andrew > > On Fri, Jan 8, 2016 at 10:54 AM, Ewen Cheslack-Postava > wrote: > > > Andrew, > > > > kafka-producer-perf-test.sh is just a wrapper around > > orga.apache.kafka.clients.tools.ProducerPerformance and all command line > > options should be forwarded. Can you just pass a --producer-props to set > > max.request.size to a larger value? > > > > -Ewen > > > > On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimirovich < > > udodizdu...@gmail.com> > > wrote: > > > > > Hi! > > > > > > I am testing Kafka's performance with large messages and would like to > > > specify maximum request size when I run kafka-producer-perf-test.sh: > > > > > > ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics > test8-3 > > > --messages 100 --new-producer --msage-size 110 > > > --show-detailed-stats > > > > > > I always get this message if I specify somethine larger than 1MB: > > > > > > ERROR Error when sending message to topic test8-3 with key: 1 bytes, > > value: > > > 110 bytes with error: The message is 1100027 bytes when serialized > > > which is larger than the maximum request size you have configured with > > the > > > max.request.size configuration. > > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > > > > > I know I can specify maximum request size with kafka-run-class.sh > > > org.apache.kafka.clients.tools.ProducerPerformance but I would like to > > use > > > kafka-producer-perf-test.sh if possible. > > > > > > Thanks! > > > > > > Andrew > > > > > > > > > > > -- > > Thanks, > > Ewen > > > -- Thanks, Ewen
Re: kafka-producer-perf-test.sh - 0.8.2.1
Ewen, I tried that before like this: ./kafka-producer-perf-test.sh --broker-list test:9092 --topics test8-3 --messages 200 --new-producer --message-size 200 --show-detailed-stats max.request.size=1000 and it does not work. It comletly ignore this option. And --producer-props is not a valid option for kafka-producer-perf-test.sh. Maybe it is not the right syntax? But I tried a lot of different ways and have yet to find the right one. Thanks! Andrew On Fri, Jan 8, 2016 at 10:54 AM, Ewen Cheslack-Postava wrote: > Andrew, > > kafka-producer-perf-test.sh is just a wrapper around > orga.apache.kafka.clients.tools.ProducerPerformance and all command line > options should be forwarded. Can you just pass a --producer-props to set > max.request.size to a larger value? > > -Ewen > > On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimirovich < > udodizdu...@gmail.com> > wrote: > > > Hi! > > > > I am testing Kafka's performance with large messages and would like to > > specify maximum request size when I run kafka-producer-perf-test.sh: > > > > ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics test8-3 > > --messages 100 --new-producer --msage-size 110 > > --show-detailed-stats > > > > I always get this message if I specify somethine larger than 1MB: > > > > ERROR Error when sending message to topic test8-3 with key: 1 bytes, > value: > > 110 bytes with error: The message is 1100027 bytes when serialized > > which is larger than the maximum request size you have configured with > the > > max.request.size configuration. > > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > > > I know I can specify maximum request size with kafka-run-class.sh > > org.apache.kafka.clients.tools.ProducerPerformance but I would like to > use > > kafka-producer-perf-test.sh if possible. > > > > Thanks! > > > > Andrew > > > > > > -- > Thanks, > Ewen >
Re: kafka-producer-perf-test.sh - 0.8.2.1
Andrew, kafka-producer-perf-test.sh is just a wrapper around orga.apache.kafka.clients.tools.ProducerPerformance and all command line options should be forwarded. Can you just pass a --producer-props to set max.request.size to a larger value? -Ewen On Fri, Jan 8, 2016 at 7:51 AM, Andrej Vladimirovich wrote: > Hi! > > I am testing Kafka's performance with large messages and would like to > specify maximum request size when I run kafka-producer-perf-test.sh: > > ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics test8-3 > --messages 100 --new-producer --msage-size 110 > --show-detailed-stats > > I always get this message if I specify somethine larger than 1MB: > > ERROR Error when sending message to topic test8-3 with key: 1 bytes, value: > 110 bytes with error: The message is 1100027 bytes when serialized > which is larger than the maximum request size you have configured with the > max.request.size configuration. > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > I know I can specify maximum request size with kafka-run-class.sh > org.apache.kafka.clients.tools.ProducerPerformance but I would like to use > kafka-producer-perf-test.sh if possible. > > Thanks! > > Andrew > -- Thanks, Ewen
kafka-producer-perf-test.sh - 0.8.2.1
Hi! I am testing Kafka's performance with large messages and would like to specify maximum request size when I run kafka-producer-perf-test.sh: ./kafka-producer-perf-test.sh --broker-list "test1:9092" --topics test8-3 --messages 100 --new-producer --msage-size 110 --show-detailed-stats I always get this message if I specify somethine larger than 1MB: ERROR Error when sending message to topic test8-3 with key: 1 bytes, value: 110 bytes with error: The message is 1100027 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) I know I can specify maximum request size with kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance but I would like to use kafka-producer-perf-test.sh if possible. Thanks! Andrew
Re: kafka producer-perf-test.sh compression-codec not working
Hi Erik, Thanks for these inputs I'll implement it Regards, Prabhjot On Aug 25, 2015 11:53 PM, "Helleren, Erik" wrote: > Prabhjot, > You can’t do it with producer perf test, but its relatively simple to > implement. The message body includes a timestamp of when your producer > produces, and the consumer looks at the difference between between the > timestamp in the body and the current timestamp. > > Or, if you were looking for ack latency, you can use the producer’s async > callback to measure latency. > -Erik > > From: Prabhjot Bharaj mailto:prabhbha...@gmail.com > >> > Date: Tuesday, August 25, 2015 at 9:22 AM > To: Erik Helleren erik.helle...@cmegroup.com>> > Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org>" < > users@kafka.apache.org<mailto:users@kafka.apache.org>>, " > d...@kafka.apache.org<mailto:d...@kafka.apache.org>" <mailto:d...@kafka.apache.org>> > Subject: Re: kafka producer-perf-test.sh compression-codec not working > > Hi Erik, > > Thanks for your inputs. > > How can we measure round trip latency using kafka-producer-perf-test.sh ? > > or any other tool ? > > Regards, > Prabhjot > > On Tue, Aug 25, 2015 at 7:41 PM, Helleren, Erik < > erik.helle...@cmegroup.com<mailto:erik.helle...@cmegroup.com>> wrote: > Prabhjot, > When no compression is being used, it should have only a tiny impact on > performance. But when it is enabled it will make it as though the message > payload is small and nearly constant, regardless as to how large the > configured message size is. > > I think that the answer is that this is room for improvement in the perf > test, especially where compression is concerned. If you do implement an > improvement, a patch might be helpful to the community. But something to > consider is that threwput alone isn’t the only important performance > measure. Round trip latency is also important. > Thanks, > -Erik > > > From: Prabhjot Bharaj mailto:prabhbha...@gmail.com > ><mailto:prabhbha...@gmail.com<mailto:prabhbha...@gmail.com>>> > Date: Tuesday, August 25, 2015 at 8:41 AM > To: Erik Helleren erik.helle...@cmegroup.com><mailto:erik.helle...@cmegroup.com erik.helle...@cmegroup.com>>> > Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org> users@kafka.apache.org<mailto:users@kafka.apache.org>>" < > users@kafka.apache.org<mailto:users@kafka.apache.org> users@kafka.apache.org<mailto:users@kafka.apache.org>>>, " > d...@kafka.apache.org<mailto:d...@kafka.apache.org> d...@kafka.apache.org<mailto:d...@kafka.apache.org>>" <mailto:d...@kafka.apache.org><mailto:d...@kafka.apache.org d...@kafka.apache.org>>> > Subject: Re: kafka producer-perf-test.sh compression-codec not working > > Hi Erik, > > I have put my efforts on the produce side till now, Thanks for making me > aware that consumer will decompress automatically. > > I'll also consider your point on creating real-life messages > > But, I have still have one confusion - > > Why would the current ProducerPerformance.scala compress an Array of Bytes > with all zeros ? > That will anyways give better throughput. correct ? > > Regards, > Prabhjot > > On Tue, Aug 25, 2015 at 7:05 PM, Helleren, Erik < > erik.helle...@cmegroup.com<mailto:erik.helle...@cmegroup.com> erik.helle...@cmegroup.com<mailto:erik.helle...@cmegroup.com>>> wrote: > Hi Prabhjot, > There are two important things to know about kafka compression: First > uncompression happens automatically in the consumer > (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you > should see ascii returned on the consumer side. The best way to see if > compression has happened that I know of is to actually look at a packet > capture. > > Second, the producer does not compress individual messages, but actually > batches several sequential messages to the same topic and partition > together and compresses that compound message. > ( > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Pro > tocol#AGuideToTheKafkaProtocol-Compression) Thus, a fixed string will > still see far better compression ratios than a Œtypical' real life > message. > > Making a real-life-like message isn¹t easy, and depends heavily on your > domain. But a general approach would be to generate messages by randomly > selected words from a dictionary. And having a dictionary around thousand > large words means there is a reasonable chance of the same words appearing > multiple times in the same message. Also words can be non-sence like > ³
Re: kafka producer-perf-test.sh compression-codec not working
Prabhjot, You can’t do it with producer perf test, but its relatively simple to implement. The message body includes a timestamp of when your producer produces, and the consumer looks at the difference between between the timestamp in the body and the current timestamp. Or, if you were looking for ack latency, you can use the producer’s async callback to measure latency. -Erik From: Prabhjot Bharaj mailto:prabhbha...@gmail.com>> Date: Tuesday, August 25, 2015 at 9:22 AM To: Erik Helleren mailto:erik.helle...@cmegroup.com>> Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>>, "d...@kafka.apache.org<mailto:d...@kafka.apache.org>" mailto:d...@kafka.apache.org>> Subject: Re: kafka producer-perf-test.sh compression-codec not working Hi Erik, Thanks for your inputs. How can we measure round trip latency using kafka-producer-perf-test.sh ? or any other tool ? Regards, Prabhjot On Tue, Aug 25, 2015 at 7:41 PM, Helleren, Erik mailto:erik.helle...@cmegroup.com>> wrote: Prabhjot, When no compression is being used, it should have only a tiny impact on performance. But when it is enabled it will make it as though the message payload is small and nearly constant, regardless as to how large the configured message size is. I think that the answer is that this is room for improvement in the perf test, especially where compression is concerned. If you do implement an improvement, a patch might be helpful to the community. But something to consider is that threwput alone isn’t the only important performance measure. Round trip latency is also important. Thanks, -Erik From: Prabhjot Bharaj mailto:prabhbha...@gmail.com><mailto:prabhbha...@gmail.com<mailto:prabhbha...@gmail.com>>> Date: Tuesday, August 25, 2015 at 8:41 AM To: Erik Helleren mailto:erik.helle...@cmegroup.com><mailto:erik.helle...@cmegroup.com<mailto:erik.helle...@cmegroup.com>>> Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org><mailto:users@kafka.apache.org<mailto:users@kafka.apache.org>>" mailto:users@kafka.apache.org><mailto:users@kafka.apache.org<mailto:users@kafka.apache.org>>>, "d...@kafka.apache.org<mailto:d...@kafka.apache.org><mailto:d...@kafka.apache.org<mailto:d...@kafka.apache.org>>" mailto:d...@kafka.apache.org><mailto:d...@kafka.apache.org<mailto:d...@kafka.apache.org>>> Subject: Re: kafka producer-perf-test.sh compression-codec not working Hi Erik, I have put my efforts on the produce side till now, Thanks for making me aware that consumer will decompress automatically. I'll also consider your point on creating real-life messages But, I have still have one confusion - Why would the current ProducerPerformance.scala compress an Array of Bytes with all zeros ? That will anyways give better throughput. correct ? Regards, Prabhjot On Tue, Aug 25, 2015 at 7:05 PM, Helleren, Erik mailto:erik.helle...@cmegroup.com><mailto:erik.helle...@cmegroup.com<mailto:erik.helle...@cmegroup.com>>> wrote: Hi Prabhjot, There are two important things to know about kafka compression: First uncompression happens automatically in the consumer (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you should see ascii returned on the consumer side. The best way to see if compression has happened that I know of is to actually look at a packet capture. Second, the producer does not compress individual messages, but actually batches several sequential messages to the same topic and partition together and compresses that compound message. (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Pro tocol#AGuideToTheKafkaProtocol-Compression) Thus, a fixed string will still see far better compression ratios than a Œtypical' real life message. Making a real-life-like message isn¹t easy, and depends heavily on your domain. But a general approach would be to generate messages by randomly selected words from a dictionary. And having a dictionary around thousand large words means there is a reasonable chance of the same words appearing multiple times in the same message. Also words can be non-sence like ³asdfasdfasdfasdf², or large words in the language of your choice. The goal is for each message to be unique, but still have similar chunks that a compression algorithm can detect and compress. -Erik On 8/25/15, 6:47 AM, "Prabhjot Bharaj" mailto:prabhbha...@gmail.com><mailto:prabhbha...@gmail.com<mailto:prabhbha...@gmail.com>>> wrote: >Hi, > >I have bene trying to use kafka-producer-perf-test.sh to arrive at certain >benchmarks. >When I try to run it with --compression-codec values of 1, 2 and 3, I >notice increased throughput compared to NoCompressionCodec > >But, When I checked t
Re: kafka producer-perf-test.sh compression-codec not working
Hi Erik, Thanks for your inputs. How can we measure round trip latency using kafka-producer-perf-test.sh ? or any other tool ? Regards, Prabhjot On Tue, Aug 25, 2015 at 7:41 PM, Helleren, Erik wrote: > Prabhjot, > When no compression is being used, it should have only a tiny impact on > performance. But when it is enabled it will make it as though the message > payload is small and nearly constant, regardless as to how large the > configured message size is. > > I think that the answer is that this is room for improvement in the perf > test, especially where compression is concerned. If you do implement an > improvement, a patch might be helpful to the community. But something to > consider is that threwput alone isn’t the only important performance > measure. Round trip latency is also important. > Thanks, > -Erik > > > From: Prabhjot Bharaj mailto:prabhbha...@gmail.com > >> > Date: Tuesday, August 25, 2015 at 8:41 AM > To: Erik Helleren erik.helle...@cmegroup.com>> > Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org>" < > users@kafka.apache.org<mailto:users@kafka.apache.org>>, " > d...@kafka.apache.org<mailto:d...@kafka.apache.org>" <mailto:d...@kafka.apache.org>> > Subject: Re: kafka producer-perf-test.sh compression-codec not working > > Hi Erik, > > I have put my efforts on the produce side till now, Thanks for making me > aware that consumer will decompress automatically. > > I'll also consider your point on creating real-life messages > > But, I have still have one confusion - > > Why would the current ProducerPerformance.scala compress an Array of Bytes > with all zeros ? > That will anyways give better throughput. correct ? > > Regards, > Prabhjot > > On Tue, Aug 25, 2015 at 7:05 PM, Helleren, Erik < > erik.helle...@cmegroup.com<mailto:erik.helle...@cmegroup.com>> wrote: > Hi Prabhjot, > There are two important things to know about kafka compression: First > uncompression happens automatically in the consumer > (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you > should see ascii returned on the consumer side. The best way to see if > compression has happened that I know of is to actually look at a packet > capture. > > Second, the producer does not compress individual messages, but actually > batches several sequential messages to the same topic and partition > together and compresses that compound message. > ( > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Pro > tocol#AGuideToTheKafkaProtocol-Compression) Thus, a fixed string will > still see far better compression ratios than a Œtypical' real life > message. > > Making a real-life-like message isn¹t easy, and depends heavily on your > domain. But a general approach would be to generate messages by randomly > selected words from a dictionary. And having a dictionary around thousand > large words means there is a reasonable chance of the same words appearing > multiple times in the same message. Also words can be non-sence like > ³asdfasdfasdfasdf², or large words in the language of your choice. The > goal is for each message to be unique, but still have similar chunks that > a compression algorithm can detect and compress. > > -Erik > > > On 8/25/15, 6:47 AM, "Prabhjot Bharaj" prabhbha...@gmail.com>> wrote: > > >Hi, > > > >I have bene trying to use kafka-producer-perf-test.sh to arrive at certain > >benchmarks. > >When I try to run it with --compression-codec values of 1, 2 and 3, I > >notice increased throughput compared to NoCompressionCodec > > > >But, When I checked the Producerperformance.scala, I saw that the the > >`producer.send` is getting data from the method: `generateProducerData`. > >But, this data is just an empty array of Bytes. > > > >Now, as per my basic understanding of compression algorithms, I think a > >byte sequence of zeros will eventually result in a very small message, > >because of which I thought I might be observing better throughput. > > > >So, in line: 247 of ProducerPerformance.scala, I did this minor code > >change:- > > > > > > > >*val message = > >"qopwr11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1- > >1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csm > >il/bitrate=11subcategory > >71Title > >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- > >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a > >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FN
Re: kafka producer-perf-test.sh compression-codec not working
Prabhjot, When no compression is being used, it should have only a tiny impact on performance. But when it is enabled it will make it as though the message payload is small and nearly constant, regardless as to how large the configured message size is. I think that the answer is that this is room for improvement in the perf test, especially where compression is concerned. If you do implement an improvement, a patch might be helpful to the community. But something to consider is that threwput alone isn’t the only important performance measure. Round trip latency is also important. Thanks, -Erik From: Prabhjot Bharaj mailto:prabhbha...@gmail.com>> Date: Tuesday, August 25, 2015 at 8:41 AM To: Erik Helleren mailto:erik.helle...@cmegroup.com>> Cc: "users@kafka.apache.org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>>, "d...@kafka.apache.org<mailto:d...@kafka.apache.org>" mailto:d...@kafka.apache.org>> Subject: Re: kafka producer-perf-test.sh compression-codec not working Hi Erik, I have put my efforts on the produce side till now, Thanks for making me aware that consumer will decompress automatically. I'll also consider your point on creating real-life messages But, I have still have one confusion - Why would the current ProducerPerformance.scala compress an Array of Bytes with all zeros ? That will anyways give better throughput. correct ? Regards, Prabhjot On Tue, Aug 25, 2015 at 7:05 PM, Helleren, Erik mailto:erik.helle...@cmegroup.com>> wrote: Hi Prabhjot, There are two important things to know about kafka compression: First uncompression happens automatically in the consumer (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you should see ascii returned on the consumer side. The best way to see if compression has happened that I know of is to actually look at a packet capture. Second, the producer does not compress individual messages, but actually batches several sequential messages to the same topic and partition together and compresses that compound message. (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Pro tocol#AGuideToTheKafkaProtocol-Compression) Thus, a fixed string will still see far better compression ratios than a Œtypical' real life message. Making a real-life-like message isn¹t easy, and depends heavily on your domain. But a general approach would be to generate messages by randomly selected words from a dictionary. And having a dictionary around thousand large words means there is a reasonable chance of the same words appearing multiple times in the same message. Also words can be non-sence like ³asdfasdfasdfasdf², or large words in the language of your choice. The goal is for each message to be unique, but still have similar chunks that a compression algorithm can detect and compress. -Erik On 8/25/15, 6:47 AM, "Prabhjot Bharaj" mailto:prabhbha...@gmail.com>> wrote: >Hi, > >I have bene trying to use kafka-producer-perf-test.sh to arrive at certain >benchmarks. >When I try to run it with --compression-codec values of 1, 2 and 3, I >notice increased throughput compared to NoCompressionCodec > >But, When I checked the Producerperformance.scala, I saw that the the >`producer.send` is getting data from the method: `generateProducerData`. >But, this data is just an empty array of Bytes. > >Now, as per my basic understanding of compression algorithms, I think a >byte sequence of zeros will eventually result in a very small message, >because of which I thought I might be observing better throughput. > >So, in line: 247 of ProducerPerformance.scala, I did this minor code >change:- > > > >*val message = >"qopwr11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1- >1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csm >il/bitrate=11subcategory >71Title >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,M >ED,HIGH,HD,.mp4.csmil/bitrate=11subcategory >71Title >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,M >ED,HIGH,HD,.mp4.csmil/bitrate=11subcategory >71Title >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- >1-1-1-1-1-1-111-1-1-"message.getBytes().slice(0,msgSize)* > > >This makes sure that I have a big message, and I can slice that >message to the message size passed in the command line options > > >But, the problem is that when I try running the same with >
Re: kafka producer-perf-test.sh compression-codec not working
Hi Erik, I have put my efforts on the produce side till now, Thanks for making me aware that consumer will decompress automatically. I'll also consider your point on creating real-life messages But, I have still have one confusion - Why would the current ProducerPerformance.scala compress an Array of Bytes with all zeros ? That will anyways give better throughput. correct ? Regards, Prabhjot On Tue, Aug 25, 2015 at 7:05 PM, Helleren, Erik wrote: > Hi Prabhjot, > There are two important things to know about kafka compression: First > uncompression happens automatically in the consumer > (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you > should see ascii returned on the consumer side. The best way to see if > compression has happened that I know of is to actually look at a packet > capture. > > Second, the producer does not compress individual messages, but actually > batches several sequential messages to the same topic and partition > together and compresses that compound message. > ( > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Pro > tocol#AGuideToTheKafkaProtocol-Compression) Thus, a fixed string will > still see far better compression ratios than a Œtypical' real life > message. > > Making a real-life-like message isn¹t easy, and depends heavily on your > domain. But a general approach would be to generate messages by randomly > selected words from a dictionary. And having a dictionary around thousand > large words means there is a reasonable chance of the same words appearing > multiple times in the same message. Also words can be non-sence like > ³asdfasdfasdfasdf², or large words in the language of your choice. The > goal is for each message to be unique, but still have similar chunks that > a compression algorithm can detect and compress. > > -Erik > > > On 8/25/15, 6:47 AM, "Prabhjot Bharaj" wrote: > > >Hi, > > > >I have bene trying to use kafka-producer-perf-test.sh to arrive at certain > >benchmarks. > >When I try to run it with --compression-codec values of 1, 2 and 3, I > >notice increased throughput compared to NoCompressionCodec > > > >But, When I checked the Producerperformance.scala, I saw that the the > >`producer.send` is getting data from the method: `generateProducerData`. > >But, this data is just an empty array of Bytes. > > > >Now, as per my basic understanding of compression algorithms, I think a > >byte sequence of zeros will eventually result in a very small message, > >because of which I thought I might be observing better throughput. > > > >So, in line: 247 of ProducerPerformance.scala, I did this minor code > >change:- > > > > > > > >*val message = > >"qopwr11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1- > >1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csm > >il/bitrate=11subcategory > >71Title > >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- > >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a > >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,M > >ED,HIGH,HD,.mp4.csmil/bitrate=11subcategory > >71Title > >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- > >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a > >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,M > >ED,HIGH,HD,.mp4.csmil/bitrate=11subcategory > >71Title > >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- > >1-1-1-1-1-1-111-1-1-"message.getBytes().slice(0,msgSize)* > > > > > >This makes sure that I have a big message, and I can slice that > >message to the message size passed in the command line options > > > > > >But, the problem is that when I try running the same with > >--compression-codec vlues of 1, 2 or 3, I still am seeing ASCII data > >(i.e. uncompressed one only) > > > > > >I want to ask whether this is a bug. And, using > >kafka-producer-perf-test.sh, how can I send my own compressed data ? > > > > > >Thanks, > > > >Prabhjot > > -- - "There are only 10 types of people in the world: Those who understand binary, and those who don't"
Re: kafka producer-perf-test.sh compression-codec not working
Hi Prabhjot, There are two important things to know about kafka compression: First uncompression happens automatically in the consumer (https://cwiki.apache.org/confluence/display/KAFKA/Compression) so you should see ascii returned on the consumer side. The best way to see if compression has happened that I know of is to actually look at a packet capture. Second, the producer does not compress individual messages, but actually batches several sequential messages to the same topic and partition together and compresses that compound message. (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Pro tocol#AGuideToTheKafkaProtocol-Compression) Thus, a fixed string will still see far better compression ratios than a Œtypical' real life message. Making a real-life-like message isn¹t easy, and depends heavily on your domain. But a general approach would be to generate messages by randomly selected words from a dictionary. And having a dictionary around thousand large words means there is a reasonable chance of the same words appearing multiple times in the same message. Also words can be non-sence like ³asdfasdfasdfasdf², or large words in the language of your choice. The goal is for each message to be unique, but still have similar chunks that a compression algorithm can detect and compress. -Erik On 8/25/15, 6:47 AM, "Prabhjot Bharaj" wrote: >Hi, > >I have bene trying to use kafka-producer-perf-test.sh to arrive at certain >benchmarks. >When I try to run it with --compression-codec values of 1, 2 and 3, I >notice increased throughput compared to NoCompressionCodec > >But, When I checked the Producerperformance.scala, I saw that the the >`producer.send` is getting data from the method: `generateProducerData`. >But, this data is just an empty array of Bytes. > >Now, as per my basic understanding of compression algorithms, I think a >byte sequence of zeros will eventually result in a very small message, >because of which I thought I might be observing better throughput. > >So, in line: 247 of ProducerPerformance.scala, I did this minor code >change:- > > > >*val message = >"qopwr11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1- >1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csm >il/bitrate=11subcategory >71Title >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,M >ED,HIGH,HD,.mp4.csmil/bitrate=11subcategory >71Title >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- >1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1a >n-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,M >ED,HIGH,HD,.mp4.csmil/bitrate=11subcategory >71Title >10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1- >1-1-1-1-1-1-111-1-1-"message.getBytes().slice(0,msgSize)* > > >This makes sure that I have a big message, and I can slice that >message to the message size passed in the command line options > > >But, the problem is that when I try running the same with >--compression-codec vlues of 1, 2 or 3, I still am seeing ASCII data >(i.e. uncompressed one only) > > >I want to ask whether this is a bug. And, using >kafka-producer-perf-test.sh, how can I send my own compressed data ? > > >Thanks, > >Prabhjot
kafka producer-perf-test.sh compression-codec not working
Hi, I have bene trying to use kafka-producer-perf-test.sh to arrive at certain benchmarks. When I try to run it with --compression-codec values of 1, 2 and 3, I notice increased throughput compared to NoCompressionCodec But, When I checked the Producerperformance.scala, I saw that the the `producer.send` is getting data from the method: `generateProducerData`. But, this data is just an empty array of Bytes. Now, as per my basic understanding of compression algorithms, I think a byte sequence of zeros will eventually result in a very small message, because of which I thought I might be observing better throughput. So, in line: 247 of ProducerPerformance.scala, I did this minor code change:- *val message = "qopwr11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csmil/bitrate=11subcategory 71Title 10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1-1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csmil/bitrate=11subcategory 71Title 10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1-1-1-1-1-1-1-111-1-1-r1VR-11591UPD113582260001AS1IL1-1N/A1Entertainment1-1an-example.com1-1-1-1-1-1-1-1011413/011413_factor_points_FNC_,LOW,MED_LOW,MED,HIGH,HD,.mp4.csmil/bitrate=11subcategory 71Title 10^D1-1-111-1-1-1-1-1-111-1-1-1-1-115101-1-1-1-1126112491-1-1-1-1-1-1-1-1-1-1-1-1-1-1-111-1-1-"message.getBytes().slice(0,msgSize)* This makes sure that I have a big message, and I can slice that message to the message size passed in the command line options But, the problem is that when I try running the same with --compression-codec vlues of 1, 2 or 3, I still am seeing ASCII data (i.e. uncompressed one only) I want to ask whether this is a bug. And, using kafka-producer-perf-test.sh, how can I send my own compressed data ? Thanks, Prabhjot
Re: kafka-producer-perf-test.sh - No visible difference between request-num-acks 1 and -1
Hi Prabhjot, Do you intent to use the old producer performance microbenchmark? Thanks, -Tao On Fri, Aug 21, 2015 at 2:41 AM, Prabhjot Bharaj wrote: > Hi, > > I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along > the bundle > > I have setup a 5 machine cluster and on the same 5 machines, I'm also > running zookeeper as well > > I am trying to see what is the maximum produce throughput I can get on this > 5 node cluster > > I have created only 1 topic - tops1 in the cluster:- > > root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper > localhost:2182 > > Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: > > Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 > > Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 > > Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 > > Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 > > > This is the output of the kafka-producer-perf-test.sh for request-num-acks > 1 and request-num-acks -1:- > > root@x.x.x.x:~# date;time kafka-producer-perf-test.sh --broker-list > x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages > 100 --message-size 500 --topics tops1 --show-detailed-stats --threads > 30 --request-num-acks 1 --batch-size 1000 --request-timeout-ms 1 > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > > 2015-08-21 09:36:01:021, 2015-08-21 09:36:49:351, 0, 500, 1000, 476.83, > 9.8662, 90, 20690.8752 > > > real 0m49.375s > > user 0m38.777s > > sys 0m5.951s > > > > root@x.x.x.x:~# date;time kafka-producer-perf-test.sh > --broker-list > x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 > --messages > 100 --message-size 500 --topics tops1 --show-detailed-stats --threads > 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms 1 > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > > 2015-08-21 09:31:19:329, 2015-08-21 09:32:08:627, 0, 500, 1000, 476.83, > 9.6724, 90, 20284.5957 > > > real 0m50.341s > > user 0m39.029s > > sys 0m6.051s > > > I'll continue to test with more number of messages, to see if I get any > difference. > > But, I just want to be sure whatever I'm doing is right. > > > Please share your thoughts on the results. > > > Regards, > > prabcs >
Re: SocketTimeoutException with kafka-producer-perf-test.sh
This link already had the answer I was searching for: http://grokbase.com/t/kafka/users/145vmz70cb/java-net-sockettimeoutexception-in-broker On Fri, Aug 21, 2015 at 3:03 PM, Prabhjot Bharaj wrote: > Hi, > > Never mind. I have solved this problem by referring to an earlier > question, where Neha had suggested to use a higher value for > request-timeout-ms > > Regards, > prabcs > > On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj > wrote: > >> Hello Folks, >> >> I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along >> the bundle >> >> I have setup a 5 machine cluster and on the same 5 machines, I'm also >> running zookeeper as well >> >> I am trying to see what is the maximum produce throughput I can get on >> this 5 node cluster >> >> I have created only 1 topic - tops1 in the cluster:- >> >> root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper >> localhost:2182 >> >> Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: >> >> Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 >> >> Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 >> >> Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 >> >> Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 >> >> >> When I run this command in parallel from 2 different machines, I get >> SocketTimeoutException:- >> >> time kafka-producer-perf-test.sh --broker-list >> x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages >> 100 --message-size 500 --topics tops1 --show-detailed-stats --threads >> 5 --request-num-acks -1 --batch-size 1000 >> >> >> My server.properties has these details:- >> >> >> broker.id=0 >> >> port=9092 >> >> num.network.threads=3 >> >> num.io.threads=8 >> >> socket.send.buffer.bytes=1048576 >> >> socket.receive.buffer.bytes=1048576 >> >> socket.request.max.bytes=104857600 >> >> log.dirs=/kafka-logs >> >> num.partitions=1 >> >> num.recovery.threads.per.data.dir=1 >> >> log.retention.hours=168 >> >> log.segment.bytes=1073741824 >> >> log.retention.check.interval.ms=30 >> >> log.cleaner.enable=false >> >> zookeeper.connect=localhost:2181 >> >> zookeeper.connection.timeout.ms=6000 >> >> num.replica.fetchers=4 >> >> >> 2 major changes in the server.properties, which gave a performance boost >> on this test were:- >> >> num.replica.fetchers=4 >> >> socket.send.buffer.bytes=1048576 >> >> socket.receive.buffer.bytes=1048576 >> >> >> I think I'm giving enough requests on the network, but not sure how I >> should correct it >> >> I am particularly interested in knowing how much max throughput per topic >> can I get, with the highest level of durability (i.e. request-num-acks = -1) >> >> >> Request you to share your thoughts on this. >> >> Thanks, >> >> Prabhjot >> > > > > -- > - > "There are only 10 types of people in the world: Those who understand > binary, and those who don't" > -- - "There are only 10 types of people in the world: Those who understand binary, and those who don't"
kafka-producer-perf-test.sh - No visible difference between request-num-acks 1 and -1
Hi, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only 1 topic - tops1 in the cluster:- root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper localhost:2182 Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 This is the output of the kafka-producer-perf-test.sh for request-num-acks 1 and request-num-acks -1:- root@x.x.x.x:~# date;time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 100 --message-size 500 --topics tops1 --show-detailed-stats --threads 30 --request-num-acks 1 --batch-size 1000 --request-timeout-ms 1 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2015-08-21 09:36:01:021, 2015-08-21 09:36:49:351, 0, 500, 1000, 476.83, 9.8662, 90, 20690.8752 real 0m49.375s user 0m38.777s sys 0m5.951s root@x.x.x.x:~# date;time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 100 --message-size 500 --topics tops1 --show-detailed-stats --threads 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms 1 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2015-08-21 09:31:19:329, 2015-08-21 09:32:08:627, 0, 500, 1000, 476.83, 9.6724, 90, 20284.5957 real 0m50.341s user 0m39.029s sys 0m6.051s I'll continue to test with more number of messages, to see if I get any difference. But, I just want to be sure whatever I'm doing is right. Please share your thoughts on the results. Regards, prabcs
Re: SocketTimeoutException with kafka-producer-perf-test.sh
Hi, Never mind. I have solved this problem by referring to an earlier question, where Neha had suggested to use a higher value for request-timeout-ms Regards, prabcs On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj wrote: > Hello Folks, > > I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along > the bundle > > I have setup a 5 machine cluster and on the same 5 machines, I'm also > running zookeeper as well > > I am trying to see what is the maximum produce throughput I can get on > this 5 node cluster > > I have created only 1 topic - tops1 in the cluster:- > > root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper > localhost:2182 > > Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: > > Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 > > Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 > > Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 > > Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 > > > When I run this command in parallel from 2 different machines, I get > SocketTimeoutException:- > > time kafka-producer-perf-test.sh --broker-list > x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages > 100 --message-size 500 --topics tops1 --show-detailed-stats --threads > 5 --request-num-acks -1 --batch-size 1000 > > > My server.properties has these details:- > > > broker.id=0 > > port=9092 > > num.network.threads=3 > > num.io.threads=8 > > socket.send.buffer.bytes=1048576 > > socket.receive.buffer.bytes=1048576 > > socket.request.max.bytes=104857600 > > log.dirs=/kafka-logs > > num.partitions=1 > > num.recovery.threads.per.data.dir=1 > > log.retention.hours=168 > > log.segment.bytes=1073741824 > > log.retention.check.interval.ms=30 > > log.cleaner.enable=false > > zookeeper.connect=localhost:2181 > > zookeeper.connection.timeout.ms=6000 > > num.replica.fetchers=4 > > > 2 major changes in the server.properties, which gave a performance boost > on this test were:- > > num.replica.fetchers=4 > > socket.send.buffer.bytes=1048576 > > socket.receive.buffer.bytes=1048576 > > > I think I'm giving enough requests on the network, but not sure how I > should correct it > > I am particularly interested in knowing how much max throughput per topic > can I get, with the highest level of durability (i.e. request-num-acks = -1) > > > Request you to share your thoughts on this. > > Thanks, > > Prabhjot > -- - "There are only 10 types of people in the world: Those who understand binary, and those who don't"
SocketTimeoutException with kafka-producer-perf-test.sh
Hello Folks, I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along the bundle I have setup a 5 machine cluster and on the same 5 machines, I'm also running zookeeper as well I am trying to see what is the maximum produce throughput I can get on this 5 node cluster I have created only 1 topic - tops1 in the cluster:- root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper localhost:2182 Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 When I run this command in parallel from 2 different machines, I get SocketTimeoutException:- time kafka-producer-perf-test.sh --broker-list x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages 100 --message-size 500 --topics tops1 --show-detailed-stats --threads 5 --request-num-acks -1 --batch-size 1000 My server.properties has these details:- broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 log.dirs=/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=localhost:2181 zookeeper.connection.timeout.ms=6000 num.replica.fetchers=4 2 major changes in the server.properties, which gave a performance boost on this test were:- num.replica.fetchers=4 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 I think I'm giving enough requests on the network, but not sure how I should correct it I am particularly interested in knowing how much max throughput per topic can I get, with the highest level of durability (i.e. request-num-acks = -1) Request you to share your thoughts on this. Thanks, Prabhjot
Re: kafka-producer-perf-test.sh
If that kafka-producer-per-test.sh doesn't work for you and you choose the DIY approach, https://github.com/sematext/ActionGenerator may be of help. Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Dec 22, 2014 at 11:09 AM, nitin sharma wrote: > have to conduce load and stress test by publishing at least 10K > messages/sec to my Kafka brokers. Here is the setup: > > Number of brokers : 2 > Topic : 1 > Partition :2. Each broker leader of one. > > I would like to know if i can use "kafka-producer-per-test.sh" in first > place. also, if any one knows the limitation of this utility, so that i > can think of writing my own in order to overcome the limitations. > > Regards, > Nitin Kumar Sharma. > > > On Mon, Dec 22, 2014 at 10:57 AM, Jun Rao wrote: > > > What kind of load testing do you plan to do? > > > > Thanks, > > > > Jun > > > > On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma < > kumarsharma.ni...@gmail.com > > > > > wrote: > > > > > Hi Team, > > > > > > I have to make a decision on whether i should go with Kafka producer > test > > > utility or build my own java tool for my load testing . > > > > > > Kindly let me know if anyone knows any limitation with > > > "kafka-producer-per-test.sh" when it come to simulation of messages > under > > > load condition? > > > > > > Regards, > > > Nitin Kumar Sharma. > > > > > >
Re: kafka-producer-perf-test.sh
have to conduce load and stress test by publishing at least 10K messages/sec to my Kafka brokers. Here is the setup: Number of brokers : 2 Topic : 1 Partition :2. Each broker leader of one. I would like to know if i can use "kafka-producer-per-test.sh" in first place. also, if any one knows the limitation of this utility, so that i can think of writing my own in order to overcome the limitations. Regards, Nitin Kumar Sharma. On Mon, Dec 22, 2014 at 10:57 AM, Jun Rao wrote: > What kind of load testing do you plan to do? > > Thanks, > > Jun > > On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma > > wrote: > > > Hi Team, > > > > I have to make a decision on whether i should go with Kafka producer test > > utility or build my own java tool for my load testing . > > > > Kindly let me know if anyone knows any limitation with > > "kafka-producer-per-test.sh" when it come to simulation of messages under > > load condition? > > > > Regards, > > Nitin Kumar Sharma. > > >
Re: kafka-producer-perf-test.sh
What kind of load testing do you plan to do? Thanks, Jun On Wed, Dec 17, 2014 at 3:42 AM, nitin sharma wrote: > Hi Team, > > I have to make a decision on whether i should go with Kafka producer test > utility or build my own java tool for my load testing . > > Kindly let me know if anyone knows any limitation with > "kafka-producer-per-test.sh" when it come to simulation of messages under > load condition? > > Regards, > Nitin Kumar Sharma. >
kafka-producer-perf-test.sh
Hi Team, I have to make a decision on whether i should go with Kafka producer test utility or build my own java tool for my load testing . Kindly let me know if anyone knows any limitation with "kafka-producer-per-test.sh" when it come to simulation of messages under load condition? Regards, Nitin Kumar Sharma.
Re: Question on output of kafka-producer-perf-test.sh
If you start with an empty topic, the log size on the broker gives you the total bytes sent. Thanks, Jun On Wed, May 28, 2014 at 10:38 AM, Maung Than wrote: > > >>>> The bytes-in-rate reported on the broker is the post-compression rate. > > Based on that fact, I am trying to calculate the volume by multiplying > duration and rate, i.e, (end.time - start.time) X MB.sec , > but it does not give me the total.data.sent.in.MB.I put the duration in > the brackets below: > > Any thoughts? > > > 2014-05-27 14:10:37:347,2014-05-27 14:11:15:546, 0, 420, 1000, > 2002.72, 52.4285, 500, 130893.4789 ( 34 sec ) > > > 2014-05-27 13:33:31:493,2014-05-27 13:34:11:862, 2, 420, 1000, > 2002.72, 49.6102, 500, 123857.4153 ( 40 sec ) > > Thanks, > Maung > > > On May 28, 2014, at 7:37 AM, Jun Rao wrote: > > > The bytes-in-rate reported on the broker is the post-compression rate. > > > > Thanks, > > > > Jun > > > > > > On Tue, May 27, 2014 at 9:16 PM, Maung Than > wrote: > > > >> Any idea when and which release that could be included in? > >> > >> We would like to have it sooner and can we do something about it? > >> > >> Thanks, > >> Maung > >> > >> On May 27, 2014, at 5:10 PM, Guozhang Wang wrote: > >> > >>> Maung, > >>> > >>> Yes, the throughput computed here is based on pre-compression bytes. In > >> the > >>> old producer we do not have a metric exposing the compression ratio, in > >> the > >>> new producer available in current trunk we can easily add that metric. > I > >>> have just created a ticket for this a moment ago. > >>> > >>> https://issues.apache.org/jira/browse/KAFKA-1472 > >>> > >>> Guozhang > >>> > >>> > >>> > >>> On Tue, May 27, 2014 at 2:41 PM, Maung Than > >> wrote: > >>> > >>>> > >>>> Hi All, > >>>> > >>>> We are seeing total data sent below is the same with or without > >>>> compression. > >>>> Is it always raw data? > >>>> If so is there a way we can get the compressed volume? > >>>> > >>>> start.time, end.time, compression, message.size, batch.size, > >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation > >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > >> further > >>>> details. > >>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, > 2002.72, > >>>> 52.4285, 500, 130893.4789 > >>>> > >>>> > >>>> -bash-4.1$ > >>>> > /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh > >>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic > >> imessage > >>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 > >>>> start.time, end.time, compression, message.size, batch.size, > >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation > >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > >> further > >>>> details. > >>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not > valid > >>>> (kafka.utils.VerifiableProperties) > >>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, > 2002.72, > >>>> 49.6102, 500, 123857.4153 > >>>> > >>>> Thanks, > >>>> Maung > >>>> > >>> > >>> > >>> > >>> -- > >>> -- Guozhang > >> > >> > >
Re: Question on output of kafka-producer-perf-test.sh
>>>> The bytes-in-rate reported on the broker is the post-compression rate. Based on that fact, I am trying to calculate the volume by multiplying duration and rate, i.e, (end.time - start.time) X MB.sec , but it does not give me the total.data.sent.in.MB.I put the duration in the brackets below: Any thoughts? 2014-05-27 14:10:37:347,2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, 52.4285, 500, 130893.4789 ( 34 sec ) 2014-05-27 13:33:31:493,2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, 49.6102, 500, 123857.4153 ( 40 sec ) Thanks, Maung On May 28, 2014, at 7:37 AM, Jun Rao wrote: > The bytes-in-rate reported on the broker is the post-compression rate. > > Thanks, > > Jun > > > On Tue, May 27, 2014 at 9:16 PM, Maung Than wrote: > >> Any idea when and which release that could be included in? >> >> We would like to have it sooner and can we do something about it? >> >> Thanks, >> Maung >> >> On May 27, 2014, at 5:10 PM, Guozhang Wang wrote: >> >>> Maung, >>> >>> Yes, the throughput computed here is based on pre-compression bytes. In >> the >>> old producer we do not have a metric exposing the compression ratio, in >> the >>> new producer available in current trunk we can easily add that metric. I >>> have just created a ticket for this a moment ago. >>> >>> https://issues.apache.org/jira/browse/KAFKA-1472 >>> >>> Guozhang >>> >>> >>> >>> On Tue, May 27, 2014 at 2:41 PM, Maung Than >> wrote: >>> >>>> >>>> Hi All, >>>> >>>> We are seeing total data sent below is the same with or without >>>> compression. >>>> Is it always raw data? >>>> If so is there a way we can get the compressed volume? >>>> >>>> start.time, end.time, compression, message.size, batch.size, >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec >>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >> further >>>> details. >>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, >>>> 52.4285, 500, 130893.4789 >>>> >>>> >>>> -bash-4.1$ >>>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh >>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic >> imessage >>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 >>>> start.time, end.time, compression, message.size, batch.size, >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec >>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >> further >>>> details. >>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid >>>> (kafka.utils.VerifiableProperties) >>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, >>>> 49.6102, 500, 123857.4153 >>>> >>>> Thanks, >>>> Maung >>>> >>> >>> >>> >>> -- >>> -- Guozhang >> >>
Re: Question on output of kafka-producer-perf-test.sh
The bytes-in-rate reported on the broker is the post-compression rate. Thanks, Jun On Tue, May 27, 2014 at 9:16 PM, Maung Than wrote: > Any idea when and which release that could be included in? > > We would like to have it sooner and can we do something about it? > > Thanks, > Maung > > On May 27, 2014, at 5:10 PM, Guozhang Wang wrote: > > > Maung, > > > > Yes, the throughput computed here is based on pre-compression bytes. In > the > > old producer we do not have a metric exposing the compression ratio, in > the > > new producer available in current trunk we can easily add that metric. I > > have just created a ticket for this a moment ago. > > > > https://issues.apache.org/jira/browse/KAFKA-1472 > > > > Guozhang > > > > > > > > On Tue, May 27, 2014 at 2:41 PM, Maung Than > wrote: > > > >> > >> Hi All, > >> > >> We are seeing total data sent below is the same with or without > >> compression. > >> Is it always raw data? > >> If so is there a way we can get the compressed volume? > >> > >> start.time, end.time, compression, message.size, batch.size, > >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >> SLF4J: Defaulting to no-operation (NOP) logger implementation > >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > further > >> details. > >> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, > >> 52.4285, 500, 130893.4789 > >> > >> > >> -bash-4.1$ > >> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh > >> --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic > imessage > >> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 > >> start.time, end.time, compression, message.size, batch.size, > >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >> SLF4J: Defaulting to no-operation (NOP) logger implementation > >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > further > >> details. > >> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, > >> 49.6102, 500, 123857.4153 > >> > >> Thanks, > >> Maung > >> > > > > > > > > -- > > -- Guozhang > >
Re: Question on output of kafka-producer-perf-test.sh
It would be probably included in the 0.8.2 release. I have looked into its implementation and it should be in fact a quite simple jira, so we would label it as "newbie" and you are more welcomed to try it on. Guozhang On Tue, May 27, 2014 at 9:16 PM, Maung Than wrote: > Any idea when and which release that could be included in? > > We would like to have it sooner and can we do something about it? > > Thanks, > Maung > > On May 27, 2014, at 5:10 PM, Guozhang Wang wrote: > > > Maung, > > > > Yes, the throughput computed here is based on pre-compression bytes. In > the > > old producer we do not have a metric exposing the compression ratio, in > the > > new producer available in current trunk we can easily add that metric. I > > have just created a ticket for this a moment ago. > > > > https://issues.apache.org/jira/browse/KAFKA-1472 > > > > Guozhang > > > > > > > > On Tue, May 27, 2014 at 2:41 PM, Maung Than > wrote: > > > >> > >> Hi All, > >> > >> We are seeing total data sent below is the same with or without > >> compression. > >> Is it always raw data? > >> If so is there a way we can get the compressed volume? > >> > >> start.time, end.time, compression, message.size, batch.size, > >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >> SLF4J: Defaulting to no-operation (NOP) logger implementation > >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > further > >> details. > >> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, > >> 52.4285, 500, 130893.4789 > >> > >> > >> -bash-4.1$ > >> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh > >> --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic > imessage > >> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 > >> start.time, end.time, compression, message.size, batch.size, > >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > >> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > >> SLF4J: Defaulting to no-operation (NOP) logger implementation > >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for > further > >> details. > >> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid > >> (kafka.utils.VerifiableProperties) > >> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, > >> 49.6102, 500, 123857.4153 > >> > >> Thanks, > >> Maung > >> > > > > > > > > -- > > -- Guozhang > > -- -- Guozhang
Re: Question on output of kafka-producer-perf-test.sh
Any idea when and which release that could be included in? We would like to have it sooner and can we do something about it? Thanks, Maung On May 27, 2014, at 5:10 PM, Guozhang Wang wrote: > Maung, > > Yes, the throughput computed here is based on pre-compression bytes. In the > old producer we do not have a metric exposing the compression ratio, in the > new producer available in current trunk we can easily add that metric. I > have just created a ticket for this a moment ago. > > https://issues.apache.org/jira/browse/KAFKA-1472 > > Guozhang > > > > On Tue, May 27, 2014 at 2:41 PM, Maung Than wrote: > >> >> Hi All, >> >> We are seeing total data sent below is the same with or without >> compression. >> Is it always raw data? >> If so is there a way we can get the compressed volume? >> >> start.time, end.time, compression, message.size, batch.size, >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec >> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >> SLF4J: Defaulting to no-operation (NOP) logger implementation >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further >> details. >> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, >> 52.4285, 500, 130893.4789 >> >> >> -bash-4.1$ >> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh >> --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic imessage >> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 >> start.time, end.time, compression, message.size, batch.size, >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec >> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >> SLF4J: Defaulting to no-operation (NOP) logger implementation >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further >> details. >> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid >> (kafka.utils.VerifiableProperties) >> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, >> 49.6102, 500, 123857.4153 >> >> Thanks, >> Maung >> > > > > -- > -- Guozhang
Re: Question on output of kafka-producer-perf-test.sh
Maung, Yes, the throughput computed here is based on pre-compression bytes. In the old producer we do not have a metric exposing the compression ratio, in the new producer available in current trunk we can easily add that metric. I have just created a ticket for this a moment ago. https://issues.apache.org/jira/browse/KAFKA-1472 Guozhang On Tue, May 27, 2014 at 2:41 PM, Maung Than wrote: > > Hi All, > > We are seeing total data sent below is the same with or without > compression. > Is it always raw data? > If so is there a way we can get the compressed volume? > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, > 52.4285, 500, 130893.4789 > > > -bash-4.1$ > /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh > --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic imessage > --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid > (kafka.utils.VerifiableProperties) > 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, > 49.6102, 500, 123857.4153 > > Thanks, > Maung > -- -- Guozhang
Question on output of kafka-producer-perf-test.sh
Hi All, We are seeing total data sent below is the same with or without compression. Is it always raw data? If so is there a way we can get the compressed volume? start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, 52.4285, 500, 130893.4789 -bash-4.1$ /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh --broker-list vp21q12ic-hpaj020921:9092 --messages 500 --topic imessage --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties) 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, 49.6102, 500, 123857.4153 Thanks, Maung
Re: kafka-producer-perf-test.sh help please
Hi Joel, Thanks for the prompt reply. gradlew jar and gradelw perf:jar seems to have gotten me what I needed: Again, thanks for your time. Chris On Fri, May 23, 2014 at 2:46 PM, Joel Koshy wrote: > > The producer performance class is not included in the binary release. > You can build it from the source release (as you attempted) - although > you need to use gradle. Just run "./gradlew jar" - you can see the > README.md file for more information. > > Joel > > On Fri, May 23, 2014 at 02:17:21PM -0500, Chris Neal wrote: > > Hi everyone. > > > > Looking for some help running the Kafka performance testing shell script. > > First got the NoClassDefFound error, and then built from src from these > > instructions: > > > > wget http://archive.apache.org/dist/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz > > tar -xvf kafka-0.8.1.1-src.tgz > > cd kafka-0.8.1.1-src > > sbt update > > sbt package > > sbt assembly-package-dependency > > > > The 'sbt assembly-package-dependency' gave this error: > > > > cneal@chriss-mbp[574]:~/workspace/kafka-0.8.1.1-src> sbt > > assembly-package-dependency > > [info] Set current project to kafka-0-8-1-1-src (in build > > file:/Users/cneal/workspace/kafka-0.8.1.1-src/) > > [error] Not a valid command: assembly-package-dependency > > [error] Not a valid project ID: assembly-package-dependency > > [error] Expected ':' (if selecting a configuration) > > [error] Not a valid key: assembly-package-dependency (similar: > > sbt-dependency) > > [error] assembly-package-dependency > > [error] > > > > I tried running the suggested 'sbt-dependency' and got this: > > > > cneal@chriss-mbp[579]:~/workspace/kafka-0.8.1.1-src> sbt sbt-dependency > > [info] Set current project to kafka-0-8-1-1-src (in build > > file:/Users/cneal/workspace/kafka-0.8.1.1-src/) > > [info] org.scala-sbt:sbt:0.13.2 > > > > ...which sure didn't seem like it did anything. I'm running sbt-0.13.2 on > > a Mac. > > > > But I still found this JAR file that seemed to have the classes required: > > > > /workspace/kafka-0.8.1.1-src/system_test/migration_tool_testsuite/0.7/lib> > > ls -l > > total 2800 > > -rw-r--r--@ 1 cneal staff 1306797 Apr 22 18:37 kafka-0.7.0.jar > > *-rw-r--r--@ 1 cneal staff55427 Apr 22 18:37 kafka-perf-0.7.0.jar* > > -rw-r--r--@ 1 cneal staff62913 Apr 22 18:37 zkclient-0.1.jar > > > > But then when I run the .sh scripts as follows, I get yet another exception: > > > > root@ip-10-0-0-42[397]:/opt/kafka/kafka> > >bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=$ZOOKEEPER > > --topic test_input --async --threads 2 --messages 10 > > > > start.time, end.time, compression, message.size, batch.size, > > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > > Exception in thread "main" java.lang.NoSuchMethodError: > > scala.Predef$.intWrapper(I)Lscala/runtime/RichInt; > > at kafka.perf.ProducerPerformance$.main(ProducerPerformance.scala:57) > > at kafka.perf.ProducerPerformance.main(ProducerPerformance.scala) > > > > Can someone please shed some light on how to get this to run? > > > > Most appreciated, > > > > Chris >
Re: kafka-producer-perf-test.sh help please
The producer performance class is not included in the binary release. You can build it from the source release (as you attempted) - although you need to use gradle. Just run "./gradlew jar" - you can see the README.md file for more information. Joel On Fri, May 23, 2014 at 02:17:21PM -0500, Chris Neal wrote: > Hi everyone. > > Looking for some help running the Kafka performance testing shell script. > First got the NoClassDefFound error, and then built from src from these > instructions: > > wget http://archive.apache.org/dist/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz > tar -xvf kafka-0.8.1.1-src.tgz > cd kafka-0.8.1.1-src > sbt update > sbt package > sbt assembly-package-dependency > > The 'sbt assembly-package-dependency' gave this error: > > cneal@chriss-mbp[574]:~/workspace/kafka-0.8.1.1-src> sbt > assembly-package-dependency > [info] Set current project to kafka-0-8-1-1-src (in build > file:/Users/cneal/workspace/kafka-0.8.1.1-src/) > [error] Not a valid command: assembly-package-dependency > [error] Not a valid project ID: assembly-package-dependency > [error] Expected ':' (if selecting a configuration) > [error] Not a valid key: assembly-package-dependency (similar: > sbt-dependency) > [error] assembly-package-dependency > [error] > > I tried running the suggested 'sbt-dependency' and got this: > > cneal@chriss-mbp[579]:~/workspace/kafka-0.8.1.1-src> sbt sbt-dependency > [info] Set current project to kafka-0-8-1-1-src (in build > file:/Users/cneal/workspace/kafka-0.8.1.1-src/) > [info] org.scala-sbt:sbt:0.13.2 > > ...which sure didn't seem like it did anything. I'm running sbt-0.13.2 on > a Mac. > > But I still found this JAR file that seemed to have the classes required: > > /workspace/kafka-0.8.1.1-src/system_test/migration_tool_testsuite/0.7/lib> > ls -l > total 2800 > -rw-r--r--@ 1 cneal staff 1306797 Apr 22 18:37 kafka-0.7.0.jar > *-rw-r--r--@ 1 cneal staff55427 Apr 22 18:37 kafka-perf-0.7.0.jar* > -rw-r--r--@ 1 cneal staff 62913 Apr 22 18:37 zkclient-0.1.jar > > But then when I run the .sh scripts as follows, I get yet another exception: > > root@ip-10-0-0-42[397]:/opt/kafka/kafka> >bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=$ZOOKEEPER > --topic test_input --async --threads 2 --messages 10 > > start.time, end.time, compression, message.size, batch.size, > total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec > Exception in thread "main" java.lang.NoSuchMethodError: > scala.Predef$.intWrapper(I)Lscala/runtime/RichInt; > at kafka.perf.ProducerPerformance$.main(ProducerPerformance.scala:57) > at kafka.perf.ProducerPerformance.main(ProducerPerformance.scala) > > Can someone please shed some light on how to get this to run? > > Most appreciated, > > Chris
kafka-producer-perf-test.sh help please
Hi everyone. Looking for some help running the Kafka performance testing shell script. First got the NoClassDefFound error, and then built from src from these instructions: wget http://archive.apache.org/dist/kafka/0.8.1.1/kafka-0.8.1.1-src.tgz tar -xvf kafka-0.8.1.1-src.tgz cd kafka-0.8.1.1-src sbt update sbt package sbt assembly-package-dependency The 'sbt assembly-package-dependency' gave this error: cneal@chriss-mbp[574]:~/workspace/kafka-0.8.1.1-src> sbt assembly-package-dependency [info] Set current project to kafka-0-8-1-1-src (in build file:/Users/cneal/workspace/kafka-0.8.1.1-src/) [error] Not a valid command: assembly-package-dependency [error] Not a valid project ID: assembly-package-dependency [error] Expected ':' (if selecting a configuration) [error] Not a valid key: assembly-package-dependency (similar: sbt-dependency) [error] assembly-package-dependency [error] I tried running the suggested 'sbt-dependency' and got this: cneal@chriss-mbp[579]:~/workspace/kafka-0.8.1.1-src> sbt sbt-dependency [info] Set current project to kafka-0-8-1-1-src (in build file:/Users/cneal/workspace/kafka-0.8.1.1-src/) [info] org.scala-sbt:sbt:0.13.2 ...which sure didn't seem like it did anything. I'm running sbt-0.13.2 on a Mac. But I still found this JAR file that seemed to have the classes required: /workspace/kafka-0.8.1.1-src/system_test/migration_tool_testsuite/0.7/lib> ls -l total 2800 -rw-r--r--@ 1 cneal staff 1306797 Apr 22 18:37 kafka-0.7.0.jar *-rw-r--r--@ 1 cneal staff55427 Apr 22 18:37 kafka-perf-0.7.0.jar* -rw-r--r--@ 1 cneal staff62913 Apr 22 18:37 zkclient-0.1.jar But then when I run the .sh scripts as follows, I get yet another exception: root@ip-10-0-0-42[397]:/opt/kafka/kafka> bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=$ZOOKEEPER --topic test_input --async --threads 2 --messages 10 start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.intWrapper(I)Lscala/runtime/RichInt; at kafka.perf.ProducerPerformance$.main(ProducerPerformance.scala:57) at kafka.perf.ProducerPerformance.main(ProducerPerformance.scala) Can someone please shed some light on how to get this to run? Most appreciated, Chris
