Ben Stopford created KAFKA-5178:
-----------------------------------
Summary: Potential Performance Degradation in Kafka Producer when
using Multiple Threads
Key: KAFKA-5178
URL: https://issues.apache.org/jira/browse/KAFKA-5178
Project: Kafka
Issue Type: Bug
Reporter: Ben Stopford
Attachments: TestDetails.pdf
There is evidence that the Kafka Producer drops performance as we increase the
number of threads using it.
This is based on some benchmarking done in the community. I have not
independently validated these results. Details of the test setup attached.
...
*Effect of Shared KafkaProducer across threads*
Kafka documentation recommend using the KafkaProducer across multiple worker
threads.
||#Producers||#Consumers||#Topics||#Partitions per topic||RoundTrip Throughput
(events/sec)||Approx Broker Events (Millions/sec)||
|1|1|1|1|268,312|0.5|
|4|4|4|4|759,186|1.5|
|8|8|8|8|640,738|1.2|
|8|8|8|16|847,314|1.7|
|8|8|8|48|17,791|0.035|
|16|16|16|64|5,997|0.01|
Something appears to be wrong here, with 48 and 64 partitions the shared
KafkaProducer struggled to the point that performance became quite bad.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
