[ https://issues.apache.org/jira/browse/KAFKA-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695730#comment-16695730 ]
ASF GitHub Bot commented on KAFKA-7667: --------------------------------------- kaushiksrinivas opened a new pull request #5939: KAFKA-7667: Synchronous send support for kafka performance producer java application URL: https://github.com/apache/kafka/pull/5939 ProducerPerformance java application now supports synchronous blocking sends to produce messages to kafka brokers. Till now, performance producer application supported only asynchronous sends with a callback function registered. this would be enough when trying to load the kafka brokers without bothering much on the replication & producer responses. But when kafka brokers replication capacity has to be loaded, asynchronous sends would not suffice. So synchronous blocking send calls are now supported with all the existing metrics calculation of producer performance. A boolean command line flag --synchronous-send has been added to enable/disable this feature. Blocking .get() calls are now done upon each send, and as of this commit ResultMetadata is not being used or logged. A new function has been introduced in the Stats class to update and get the iteration counter. Tests done: Comparison tests were done with and without synchronous sends. below were the configurations of kafka cluster: No of brokers: 3 CPU per broker: 10 cores Memory : 32GB Topic partitions : 1 replication factor : 3 min.insync.replicas: 1 (default). producer acks level : 1(default), response after only leader write. Similar configurations were given for performance producer with & without synchronous sends enabled. It was observed that, synchronous producer was producing records approximately 5 times slower than asynchronous producer but with guarantee of replication before subsequent sends. So this clearly indicates the impact of replication in brokers. And these tests would help optimise resources in kafka broker from replication performance point of view. Attached results snapshot of performance producer run with/without synchronous send flag enabled. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Need synchronous records send support for kafka performance producer java > application. > -------------------------------------------------------------------------------------- > > Key: KAFKA-7667 > URL: https://issues.apache.org/jira/browse/KAFKA-7667 > Project: Kafka > Issue Type: Improvement > Components: tools > Affects Versions: 2.0.0 > Reporter: kaushik srinivas > Assignee: kaushik srinivas > Priority: Major > > Why synchronous send support for performance producer ? > ProducerPerformance java application is used for load testing kafka brokers. > Load testing involves replicating very high throughput records flowing in to > kafka brokers and > many producers in field would use synchronous way of sending data i.e > blocking until the message has been > written completely on all the min.insyc.replicas no of brokers. > Asynchronous sends would satisfy the first requirement of loading kafka > brokers. > This requirement would help in performance tuning the kafka brokers when > producers are deployed with "acks": all, "min.insync.replicas" : > equal to replication factor and synchronous way of sending. > Throughput degradation happens with synchronous producers and this would help > in > tuning resources for replication in kafka brokers. > Also benchmarks could be made from kafka producer perspective with > synchronous way of sending records and tune kafka producer's > resources appropriately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)