Huanli-Meng commented on a change in pull request #12336: URL: https://github.com/apache/pulsar/pull/12336#discussion_r727649275
########## File path: site2/docs/performance-pulsar-perf.md ########## @@ -8,149 +8,168 @@ The Pulsar Perf is a built-in performance test tool for Apache Pulsar. You can u ## Produce messages -This example shows how the Pulsar Perf produces messages with default options. For all configuration options available for the `pulsar-perf produce` command, see [configuration options](#configuration-options-for-pulsar-perf-produce). +> **Tip** +> +> For the latest and complete information about `pulsar-perf`, including commands, flags, descriptions, and more, see [`pulsar-perf`](https://pulsar.apache.org/tools/pulsar-perf/) or [here](reference-cli-tools.md#pulsar-perf). -``` -bin/pulsar-perf produce my-topic -``` +- This example shows how the Pulsar Perf produces messages with **default** options. + + **Input** + + ``` + bin/pulsar-perf produce my-topic + ``` -After the command is executed, the test data is continuously output on the Console. + After the command is executed, the test data is continuously output on the Console. -**Output** + **Output** -``` -19:53:31.459 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Created 1 producers -19:53:31.482 [pulsar-timer-5-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider -19:53:40.861 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 93.7 msg/s --- 0.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.575 ms - med: 3.460 - 95pct: 4.790 - 99pct: 5.308 - 99.9pct: 5.834 - 99.99pct: 6.609 - Max: 6.609 -19:53:50.909 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.437 ms - med: 3.328 - 95pct: 4.656 - 99pct: 5.071 - 99.9pct: 5.519 - 99.99pct: 5.588 - Max: 5.588 -19:54:00.926 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.376 ms - med: 3.276 - 95pct: 4.520 - 99pct: 4.939 - 99.9pct: 5.440 - 99.99pct: 5.490 - Max: 5.490 -19:54:10.940 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.298 ms - med: 3.220 - 95pct: 4.474 - 99pct: 4.926 - 99.9pct: 5.645 - 99.99pct: 5.654 - Max: 5.654 -19:54:20.956 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.1 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.308 ms - med: 3.199 - 95pct: 4.532 - 99pct: 4.871 - 99.9pct: 5.291 - 99.99pct: 5.323 - Max: 5.323 -19:54:30.972 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.249 ms - med: 3.144 - 95pct: 4.437 - 99pct: 4.970 - 99.9pct: 5.329 - 99.99pct: 5.414 - Max: 5.414 -19:54:40.987 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.435 ms - med: 3.361 - 95pct: 4.772 - 99pct: 5.150 - 99.9pct: 5.373 - 99.99pct: 5.837 - Max: 5.837 -^C19:54:44.325 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 7286 records sent --- 99.140 msg/s --- 0.775 Mbit/s -19:54:44.336 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 3.383 ms - med: 3.293 - 95pct: 4.610 - 99pct: 5.059 - 99.9pct: 5.588 - 99.99pct: 5.837 - 99.999pct: 6.609 - Max: 6.609 -``` + ``` + 19:53:31.459 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Created 1 producers + 19:53:31.482 [pulsar-timer-5-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider + 19:53:40.861 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 93.7 msg/s --- 0.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.575 ms - med: 3.460 - 95pct: 4.790 - 99pct: 5.308 - 99.9pct: 5.834 - 99.99pct: 6.609 - Max: 6.609 + 19:53:50.909 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.437 ms - med: 3.328 - 95pct: 4.656 - 99pct: 5.071 - 99.9pct: 5.519 - 99.99pct: 5.588 - Max: 5.588 + 19:54:00.926 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.376 ms - med: 3.276 - 95pct: 4.520 - 99pct: 4.939 - 99.9pct: 5.440 - 99.99pct: 5.490 - Max: 5.490 + 19:54:10.940 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.298 ms - med: 3.220 - 95pct: 4.474 - 99pct: 4.926 - 99.9pct: 5.645 - 99.99pct: 5.654 - Max: 5.654 + 19:54:20.956 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.1 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.308 ms - med: 3.199 - 95pct: 4.532 - 99pct: 4.871 - 99.9pct: 5.291 - 99.99pct: 5.323 - Max: 5.323 + 19:54:30.972 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.249 ms - med: 3.144 - 95pct: 4.437 - 99pct: 4.970 - 99.9pct: 5.329 - 99.99pct: 5.414 - Max: 5.414 + 19:54:40.987 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.435 ms - med: 3.361 - 95pct: 4.772 - 99pct: 5.150 - 99.9pct: 5.373 - 99.99pct: 5.837 - Max: 5.837 + ^C19:54:44.325 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 7286 records sent --- 99.140 msg/s --- 0.775 Mbit/s + 19:54:44.336 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 3.383 ms - med: 3.293 - 95pct: 4.610 - 99pct: 5.059 - 99.9pct: 5.588 - 99.99pct: 5.837 - 99.999pct: 6.609 - Max: 6.609 + ``` + + From the above test data, you can get the throughput statistics and the write latency statistics. The aggregated statistics is printed when the Pulsar Perf is stopped. You can press **Ctrl**+**C** to stop the Pulsar Perf. After the Pulsar Perf is stopped, the [HdrHistogram](http://hdrhistogram.github.io/HdrHistogram/) formatted test result appears under your directory. The document looks like `perf-producer-1589370810837.hgrm`. You can also check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html). For details about how to check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html), see [HdrHistogram Plotter](#hdrhistogram-plotter). + +- This example shows how the Pulsar Perf produces messages with `transaction` option. + + **Input** + + ```shell + bin/pulsar-perf produce my-topic -r 10 -m 100 -txn + ``` + + **Output** + + ```shell + 2021-10-11T13:36:15,595+0800 INFO [Thread-3] o.a.p.t.PerformanceProducer@499 - --- Transaction : 2 transaction end successfully ---0 transaction end failed --- 0.200 Txn/s -From the above test data, you can get the throughput statistics and the write latency statistics. The aggregated statistics is printed when the Pulsar Perf is stopped. You can press **Ctrl**+**C** to stop the Pulsar Perf. After the Pulsar Perf is stopped, the [HdrHistogram](http://hdrhistogram.github.io/HdrHistogram/) formatted test result appears under your directory. The document looks like `perf-producer-1589370810837.hgrm`. You can also check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html). For details about how to check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html), see [HdrHistogram Plotter](#hdrhistogram-plotter). - -### Configuration options for `pulsar-perf produce` - -You can get all options by executing the `bin/pulsar-perf produce -h` command. Therefore, you can modify these options as required. - -The following table lists configuration options available for the `pulsar-perf produce` command. - -| Option | Description | Default value| -|----|----|----| -| access-mode | Set the producer access mode. Valid values are `Shared`, `Exclusive` and `WaitForExclusive`. | Shared | -| admin-url | Set the Pulsar admin URL. | N/A | -| auth-params | Set the authentication parameters, whose format is determined by the implementation of the `configure` method in the authentication plugin class, such as "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}". | N/A | -| auth-plugin | Set the authentication plugin class name. | N/A | -| listener-name | Set the listener name for the broker. | N/A | -| batch-max-bytes | Set the maximum number of bytes for each batch. | 4194304 | -| batch-max-messages | Set the maximum number of messages for each batch. | 1000 | -| batch-time-window | Set a window for a batch of messages. | 1 ms | -| busy-wait | Enable or disable Busy-Wait on the Pulsar client. | false | -| chunking | Configure whether to split the message and publish in chunks if message size is larger than allowed max size. | false | -| compression | Compress the message payload. | N/A | -| conf-file | Set the configuration file. | N/A | -| delay | Mark messages with a given delay. | 0s | -| encryption-key-name | Set the name of the public key used to encrypt the payload. | N/A | -| encryption-key-value-file | Set the file which contains the public key used to encrypt the payload. | N/A | -| exit-on-failure | Configure whether to exit from the process on publish failure. | false | -| format-class | Set the custom formatter class name. | org.apache.pulsar.testclient.DefaultMessageFormatter | -| format-payload | Configure whether to format %i as a message index in the stream from producer and/or %t as the timestamp nanoseconds. | false | -| help | Configure the help message. | false | -| max-connections | Set the maximum number of TCP connections to a single broker. | 100 | -| max-outstanding | Set the maximum number of outstanding messages. | 1000 | -| max-outstanding-across-partitions | Set the maximum number of outstanding messages across partitions. | 50000 | -| message-key-generation-mode | Set the generation mode of message key. Valid options are `autoIncrement`, `random`. | N/A | -| num-io-threads | Set the number of threads to be used for handling connections to brokers. | 1 | -| num-messages | Set the number of messages to be published in total. If it is set to 0, it keeps publishing messages. | 0 | -| num-producers | Set the number of producers for each topic. | 1 | -| num-test-threads | Set the number of test threads. | 1 | -| num-topic | Set the number of topics. | 1 | -| partitions | Configure whether to create partitioned topics with the given number of partitions. | N/A | -| payload-delimiter | Set the delimiter used to split lines when using payload from a file. | \n | -| payload-file | Use the payload from an UTF-8 encoded text file and a payload is randomly selected when messages are published. | N/A | -| producer-name | Set the producer name. | N/A | -| rate | Set the publish rate of messages across topics. | 100 | -| send-timeout | Set the sendTimeout. | 0 | -| separator | Set the separator between the topic and topic number. | - | -| service-url | Set the Pulsar service URL. | | -| size | Set the message size. | 1024 bytes | -| stats-interval-seconds | Set the statistics interval. If it is set to 0, statistics is disabled. | 0 | -| test-duration | Set the test duration. If it is set to 0, it keeps publishing tests. | 0s | -| trust-cert-file | Set the path for the trusted TLS certificate file. | <empty string> | -| warmup-time | Set the warm-up time. | 1s | -| tls-allow-insecure | Set the allowed insecure TLS connection. | N/A | + 2021-10-11T13:36:15,614+0800 INFO [Thread-3] o.a.p.t.PerformanceProducer@503 - Throughput produced: 100 msg --- 0.0 msg/s --- 0.1 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.067 ms - med: 3.104 - 95pct: 3.747 - 99pct: 4.619 - 99.9pct: 6.760 - 99.99pct: 6.760 - Max: 6.760 + + 2021-10-11T13:36:15,710+0800 INFO [pulsar-perf-producer-exec-46-1] o.a.p.t.PerformanceProducer@834 - Aggregated latency stats --- Latency: mean: 3.067 ms - med: 3.104 - 95pct: 3.747 - 99pct: 4.619 - 99.9pct: 6.760 - 99.99pct: 6.760 - 99.999pct: 6.760 - Max: 6.760 + + 2021-10-11T13:36:29,976+0800 INFO [Thread-4] o.a.p.t.PerformanceProducer@815 - --- Transaction : 2 transaction end successfully --- 0 transaction end failed --- 2 transaction open successfully --- 0 transaction open failed --- 12.237 Txn/s + + 2021-10-11T13:36:29,976+0800 INFO [Thread-4] o.a.p.t.PerformanceProducer@824 - Aggregated throughput stats --- 102 records sent --- 4.168 msg/s --- 0.033 Mbit/s + ``` ## Consume messages -This example shows how the Pulsar Perf consumes messages with default options. +> **Tip** +> +> For the latest and complete information about `pulsar-perf`, including commands, flags, descriptions, and more, see [`pulsar-perf`](https://pulsar.apache.org/tools/pulsar-perf/) or [here](reference-cli-tools.md#pulsar-perf). -``` -bin/pulsar-perf consume my-topic -``` +- This example shows how the Pulsar Perf consumes messages with **default** options. + + **Input** + + If you have not created a topic (in this example, it is _my-topic_) before, the broker creates a new topic without partitions, then the consumer can not receive any messages. Consequently, before using `pulsar-perf consume`, make sure your topic has enough messages to consume. Review comment: 1. I think this sentence can be used as a Note or Tip instead of a sentence under "Input" 2. I am a little confused. A consumer cannot consume messages from a non-partitioned topic if the topic is automatically created by the broker? If so, we should create a partitioned topic in advance instead of ensure having enough messages. Please correct me if i misunderstand. ########## File path: site2/docs/performance-pulsar-perf.md ########## @@ -8,149 +8,168 @@ The Pulsar Perf is a built-in performance test tool for Apache Pulsar. You can u ## Produce messages -This example shows how the Pulsar Perf produces messages with default options. For all configuration options available for the `pulsar-perf produce` command, see [configuration options](#configuration-options-for-pulsar-perf-produce). +> **Tip** +> +> For the latest and complete information about `pulsar-perf`, including commands, flags, descriptions, and more, see [`pulsar-perf`](https://pulsar.apache.org/tools/pulsar-perf/) or [here](reference-cli-tools.md#pulsar-perf). -``` -bin/pulsar-perf produce my-topic -``` +- This example shows how the Pulsar Perf produces messages with **default** options. + + **Input** + + ``` + bin/pulsar-perf produce my-topic + ``` -After the command is executed, the test data is continuously output on the Console. + After the command is executed, the test data is continuously output on the Console. -**Output** + **Output** -``` -19:53:31.459 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Created 1 producers -19:53:31.482 [pulsar-timer-5-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider -19:53:40.861 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 93.7 msg/s --- 0.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.575 ms - med: 3.460 - 95pct: 4.790 - 99pct: 5.308 - 99.9pct: 5.834 - 99.99pct: 6.609 - Max: 6.609 -19:53:50.909 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.437 ms - med: 3.328 - 95pct: 4.656 - 99pct: 5.071 - 99.9pct: 5.519 - 99.99pct: 5.588 - Max: 5.588 -19:54:00.926 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.376 ms - med: 3.276 - 95pct: 4.520 - 99pct: 4.939 - 99.9pct: 5.440 - 99.99pct: 5.490 - Max: 5.490 -19:54:10.940 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.298 ms - med: 3.220 - 95pct: 4.474 - 99pct: 4.926 - 99.9pct: 5.645 - 99.99pct: 5.654 - Max: 5.654 -19:54:20.956 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.1 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.308 ms - med: 3.199 - 95pct: 4.532 - 99pct: 4.871 - 99.9pct: 5.291 - 99.99pct: 5.323 - Max: 5.323 -19:54:30.972 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.249 ms - med: 3.144 - 95pct: 4.437 - 99pct: 4.970 - 99.9pct: 5.329 - 99.99pct: 5.414 - Max: 5.414 -19:54:40.987 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.435 ms - med: 3.361 - 95pct: 4.772 - 99pct: 5.150 - 99.9pct: 5.373 - 99.99pct: 5.837 - Max: 5.837 -^C19:54:44.325 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 7286 records sent --- 99.140 msg/s --- 0.775 Mbit/s -19:54:44.336 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 3.383 ms - med: 3.293 - 95pct: 4.610 - 99pct: 5.059 - 99.9pct: 5.588 - 99.99pct: 5.837 - 99.999pct: 6.609 - Max: 6.609 -``` + ``` + 19:53:31.459 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Created 1 producers + 19:53:31.482 [pulsar-timer-5-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider + 19:53:40.861 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 93.7 msg/s --- 0.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.575 ms - med: 3.460 - 95pct: 4.790 - 99pct: 5.308 - 99.9pct: 5.834 - 99.99pct: 6.609 - Max: 6.609 + 19:53:50.909 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.437 ms - med: 3.328 - 95pct: 4.656 - 99pct: 5.071 - 99.9pct: 5.519 - 99.99pct: 5.588 - Max: 5.588 + 19:54:00.926 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.376 ms - med: 3.276 - 95pct: 4.520 - 99pct: 4.939 - 99.9pct: 5.440 - 99.99pct: 5.490 - Max: 5.490 + 19:54:10.940 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.298 ms - med: 3.220 - 95pct: 4.474 - 99pct: 4.926 - 99.9pct: 5.645 - 99.99pct: 5.654 - Max: 5.654 + 19:54:20.956 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.1 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.308 ms - med: 3.199 - 95pct: 4.532 - 99pct: 4.871 - 99.9pct: 5.291 - 99.99pct: 5.323 - Max: 5.323 + 19:54:30.972 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.249 ms - med: 3.144 - 95pct: 4.437 - 99pct: 4.970 - 99.9pct: 5.329 - 99.99pct: 5.414 - Max: 5.414 + 19:54:40.987 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.435 ms - med: 3.361 - 95pct: 4.772 - 99pct: 5.150 - 99.9pct: 5.373 - 99.99pct: 5.837 - Max: 5.837 + ^C19:54:44.325 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 7286 records sent --- 99.140 msg/s --- 0.775 Mbit/s + 19:54:44.336 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 3.383 ms - med: 3.293 - 95pct: 4.610 - 99pct: 5.059 - 99.9pct: 5.588 - 99.99pct: 5.837 - 99.999pct: 6.609 - Max: 6.609 + ``` + + From the above test data, you can get the throughput statistics and the write latency statistics. The aggregated statistics is printed when the Pulsar Perf is stopped. You can press **Ctrl**+**C** to stop the Pulsar Perf. After the Pulsar Perf is stopped, the [HdrHistogram](http://hdrhistogram.github.io/HdrHistogram/) formatted test result appears under your directory. The document looks like `perf-producer-1589370810837.hgrm`. You can also check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html). For details about how to check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html), see [HdrHistogram Plotter](#hdrhistogram-plotter). + +- This example shows how the Pulsar Perf produces messages with `transaction` option. + + **Input** + + ```shell + bin/pulsar-perf produce my-topic -r 10 -m 100 -txn + ``` + + **Output** + + ```shell + 2021-10-11T13:36:15,595+0800 INFO [Thread-3] o.a.p.t.PerformanceProducer@499 - --- Transaction : 2 transaction end successfully ---0 transaction end failed --- 0.200 Txn/s -From the above test data, you can get the throughput statistics and the write latency statistics. The aggregated statistics is printed when the Pulsar Perf is stopped. You can press **Ctrl**+**C** to stop the Pulsar Perf. After the Pulsar Perf is stopped, the [HdrHistogram](http://hdrhistogram.github.io/HdrHistogram/) formatted test result appears under your directory. The document looks like `perf-producer-1589370810837.hgrm`. You can also check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html). For details about how to check the test result through [HdrHistogram Plotter](https://hdrhistogram.github.io/HdrHistogram/plotFiles.html), see [HdrHistogram Plotter](#hdrhistogram-plotter). - -### Configuration options for `pulsar-perf produce` - -You can get all options by executing the `bin/pulsar-perf produce -h` command. Therefore, you can modify these options as required. - -The following table lists configuration options available for the `pulsar-perf produce` command. - -| Option | Description | Default value| -|----|----|----| -| access-mode | Set the producer access mode. Valid values are `Shared`, `Exclusive` and `WaitForExclusive`. | Shared | -| admin-url | Set the Pulsar admin URL. | N/A | -| auth-params | Set the authentication parameters, whose format is determined by the implementation of the `configure` method in the authentication plugin class, such as "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}". | N/A | -| auth-plugin | Set the authentication plugin class name. | N/A | -| listener-name | Set the listener name for the broker. | N/A | -| batch-max-bytes | Set the maximum number of bytes for each batch. | 4194304 | -| batch-max-messages | Set the maximum number of messages for each batch. | 1000 | -| batch-time-window | Set a window for a batch of messages. | 1 ms | -| busy-wait | Enable or disable Busy-Wait on the Pulsar client. | false | -| chunking | Configure whether to split the message and publish in chunks if message size is larger than allowed max size. | false | -| compression | Compress the message payload. | N/A | -| conf-file | Set the configuration file. | N/A | -| delay | Mark messages with a given delay. | 0s | -| encryption-key-name | Set the name of the public key used to encrypt the payload. | N/A | -| encryption-key-value-file | Set the file which contains the public key used to encrypt the payload. | N/A | -| exit-on-failure | Configure whether to exit from the process on publish failure. | false | -| format-class | Set the custom formatter class name. | org.apache.pulsar.testclient.DefaultMessageFormatter | -| format-payload | Configure whether to format %i as a message index in the stream from producer and/or %t as the timestamp nanoseconds. | false | -| help | Configure the help message. | false | -| max-connections | Set the maximum number of TCP connections to a single broker. | 100 | -| max-outstanding | Set the maximum number of outstanding messages. | 1000 | -| max-outstanding-across-partitions | Set the maximum number of outstanding messages across partitions. | 50000 | -| message-key-generation-mode | Set the generation mode of message key. Valid options are `autoIncrement`, `random`. | N/A | -| num-io-threads | Set the number of threads to be used for handling connections to brokers. | 1 | -| num-messages | Set the number of messages to be published in total. If it is set to 0, it keeps publishing messages. | 0 | -| num-producers | Set the number of producers for each topic. | 1 | -| num-test-threads | Set the number of test threads. | 1 | -| num-topic | Set the number of topics. | 1 | -| partitions | Configure whether to create partitioned topics with the given number of partitions. | N/A | -| payload-delimiter | Set the delimiter used to split lines when using payload from a file. | \n | -| payload-file | Use the payload from an UTF-8 encoded text file and a payload is randomly selected when messages are published. | N/A | -| producer-name | Set the producer name. | N/A | -| rate | Set the publish rate of messages across topics. | 100 | -| send-timeout | Set the sendTimeout. | 0 | -| separator | Set the separator between the topic and topic number. | - | -| service-url | Set the Pulsar service URL. | | -| size | Set the message size. | 1024 bytes | -| stats-interval-seconds | Set the statistics interval. If it is set to 0, statistics is disabled. | 0 | -| test-duration | Set the test duration. If it is set to 0, it keeps publishing tests. | 0s | -| trust-cert-file | Set the path for the trusted TLS certificate file. | <empty string> | -| warmup-time | Set the warm-up time. | 1s | -| tls-allow-insecure | Set the allowed insecure TLS connection. | N/A | + 2021-10-11T13:36:15,614+0800 INFO [Thread-3] o.a.p.t.PerformanceProducer@503 - Throughput produced: 100 msg --- 0.0 msg/s --- 0.1 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.067 ms - med: 3.104 - 95pct: 3.747 - 99pct: 4.619 - 99.9pct: 6.760 - 99.99pct: 6.760 - Max: 6.760 + + 2021-10-11T13:36:15,710+0800 INFO [pulsar-perf-producer-exec-46-1] o.a.p.t.PerformanceProducer@834 - Aggregated latency stats --- Latency: mean: 3.067 ms - med: 3.104 - 95pct: 3.747 - 99pct: 4.619 - 99.9pct: 6.760 - 99.99pct: 6.760 - 99.999pct: 6.760 - Max: 6.760 + + 2021-10-11T13:36:29,976+0800 INFO [Thread-4] o.a.p.t.PerformanceProducer@815 - --- Transaction : 2 transaction end successfully --- 0 transaction end failed --- 2 transaction open successfully --- 0 transaction open failed --- 12.237 Txn/s + + 2021-10-11T13:36:29,976+0800 INFO [Thread-4] o.a.p.t.PerformanceProducer@824 - Aggregated throughput stats --- 102 records sent --- 4.168 msg/s --- 0.033 Mbit/s + ``` ## Consume messages -This example shows how the Pulsar Perf consumes messages with default options. +> **Tip** +> +> For the latest and complete information about `pulsar-perf`, including commands, flags, descriptions, and more, see [`pulsar-perf`](https://pulsar.apache.org/tools/pulsar-perf/) or [here](reference-cli-tools.md#pulsar-perf). -``` -bin/pulsar-perf consume my-topic -``` +- This example shows how the Pulsar Perf consumes messages with **default** options. + + **Input** + + If you have not created a topic (in this example, it is _my-topic_) before, the broker creates a new topic without partitions, then the consumer can not receive any messages. Consequently, before using `pulsar-perf consume`, make sure your topic has enough messages to consume. + + ``` + bin/pulsar-perf consume my-topic + ``` + + After the command is executed, the test data is continuously output on the Console. + + **Output** + + ``` + 20:35:37.071 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Start receiving from 1 consumers on 1 topics + 20:35:41.150 [pulsar-client-io-1-9] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider + 20:35:47.092 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 59.572 msg/s -- 0.465 Mbit/s --- Latency: mean: 11.298 ms - med: 10 - 95pct: 15 - 99pct: 98 - 99.9pct: 137 - 99.99pct: 152 - Max: 152 + 20:35:57.104 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 99.958 msg/s -- 0.781 Mbit/s --- Latency: mean: 9.176 ms - med: 9 - 95pct: 15 - 99pct: 16 - 99.9pct: 17 - 99.99pct: 18 - Max: 18 + 20:36:07.115 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 100.006 msg/s -- 0.781 Mbit/s --- Latency: mean: 9.316 ms - med: 9 - 95pct: 15 - 99pct: 16 - 99.9pct: 17 - 99.99pct: 17 - Max: 17 + 20:36:17.125 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 100.085 msg/s -- 0.782 Mbit/s --- Latency: mean: 9.327 ms - med: 9 - 95pct: 15 - 99pct: 16 - 99.9pct: 17 - 99.99pct: 17 - Max: 17 + 20:36:27.136 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 99.900 msg/s -- 0.780 Mbit/s --- Latency: mean: 9.404 ms - med: 9 - 95pct: 15 - 99pct: 16 - 99.9pct: 17 - 99.99pct: 17 - Max: 17 + 20:36:37.147 [main] INFO org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 99.985 msg/s -- 0.781 Mbit/s --- Latency: mean: 8.998 ms - med: 9 - 95pct: 15 - 99pct: 16 - 99.9pct: 17 - 99.99pct: 17 - Max: 17 + ^C20:36:42.755 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceConsumer - Aggregated throughput stats --- 6051 records received --- 92.125 msg/s --- 0.720 Mbit/s + 20:36:42.759 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceConsumer - Aggregated latency stats --- Latency: mean: 9.422 ms - med: 9 - 95pct: 15 - 99pct: 16 - 99.9pct: 98 - 99.99pct: 137 - 99.999pct: 152 - Max: 152 + ``` + + From the output test data, you can get the throughput statistics and the end-to-end latency statistics. The aggregated statistics is printed after the Pulsar Perf is stopped. You can press **Ctrl**+**C** to stop the Pulsar Perf. -After the command is executed, the test data is continuously output on the Console. +- This example shows how the Pulsar Perf consumes messages with `transaction` option. + + **Input** + + If you have not created a topic (in this example, it is _my-topic_) before, the broker creates a new topic without partitions, then the consumer can not receive any messages. Consequently, before using `pulsar-perf consume`, make sure your topic has enough messages to consume. Review comment: Same comment as the previous one. ########## File path: site2/docs/reference-cli-tools.md ########## @@ -653,6 +675,43 @@ Options |`--cluster`|The cluster to test on|| |`-h`, `--help`|Help message|false| +### `transaction` + +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). + +**Usage** + +```bash +$ pulsar-perf transaction options +``` + +**Options** + +|Flag|Description|Default| +|---|---|---| +`-au`, `--admin-url`|Pulsar admin URL.|N/A +`--conf-file`|Configuration file.|N/A +`-h`, `--help`|Help messages.|N/A +`-c`, `--max-connections`|Maximum number of TCP connections to a single broker.|100 +`-ioThreads`, `--num-io-threads`|Set the number of threads to be used for handling connections to brokers. |1 +`-ns`, `--num-subscriptions`|Number of subscriptions per topic.|1 +`-threads`, `--num-test-threads`|Number of test threads. <br /><br />This thread is for a new transaction to ack messages from consumer topics, produce messages to producer topics, and commit or abort this transaction. <br /><br /> Increasing the number of threads increases the parallelism of the performance test, consequently, it increases the intensity of the stress test.|1 +`-nmc`, `--numMessage-perTransaction-consume`|Set the number of messages consumed in a transaction. <br /><br /> If transaction is disabled, it means the number of messages consumed in a task instead of in a transaction.|1 +`-`nmp`, `--numMessage-perTransaction-produce`|Set the number of messages produced in a transaction. <br /><br />If transaction is disabled, it means the number of messages produced in a task instead of in a transaction.|1 Review comment: ```suggestion `-nmp`, `--numMessage-perTransaction-produce`|Set the number of messages produced in a transaction. <br /><br />If transaction is disabled, it means the number of messages produced in a task instead of in a transaction.|1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
