[ https://issues.apache.org/jira/browse/KAFKA-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antony Stubbs updated KAFKA-5178: --------------------------------- Comment: was deleted (was: My raw notes on researching the issue and slack conversations: Consumer Performance Notes https://confluent.slack.com/archives/C0KRA68SZ/p1492451673007148 Chris Matta [6:54 PM] what happens if a consumer is consuming from multiple partitions with differing cardinality, and they’ve set `max.poll.records` to 500, the first partition always has 500 records to pull, does that mean the other partitions will never be read? Ewen Cheslack-Postava [6:59 PM] @chris it will return the others. the completed fetches are tracked per-partition in a queue [7:00] and that queue is processed in order so that when a FetchRequest returns with data for multiple partitions, all that data gets enqueued then a subsequent FetchRequest is sent. any subsequent data from the same partitions will get queued up after the data from the first request/response Chris Matta [7:03 PM] ok, thanks @ewen, is this covered in documentation anywhere? Ewen Cheslack-Postava [9:03 PM] not that i'm aware of. not even sure how we'd do that very concisely since it requires a bit of understanding of the underlying protocol to not overstate the guarantees, i.e. you need to know a bit about fetch requests Ismael Juma [10:44 PM] @chris KIP-41 describes the behaviour of `max.poll.records`. KIP-74 is also somewhat relevant if you’re thinking about the behaviour of fetches as well. Not user docs though. KIP-41 describes the behaviour of `max.poll.records` https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records KIP-74 is also somewhat relevant if you’re thinking about the behaviour of fetches as well https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes https://confluent.slack.com/archives/C07FCMZ39/p1491263732459808 https://confluent.slack.com/archives/C07FCMZ39/p1491597142583282 https://issues.apache.org/jira/browse/KAFKA-4753 https://confluent.slack.com/archives/C07FCMZ39/p1492199660178716 Replicator issues with funding circle - hot fixes were shipped to them http://testing.confluent.io/confluent-kafka-system-test-results/?prefix=2017-04-28--001.1493388356--apache--trunk--bc10f5f/Benchmark/ https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ https://issues.apache.org/jira/browse/KAFKA-4753 KafkaConsumer susceptible to FetchResponse starvation) > Potential Performance Degradation in Kafka Producer when using Multiple > Threads > ------------------------------------------------------------------------------- > > Key: KAFKA-5178 > URL: https://issues.apache.org/jira/browse/KAFKA-5178 > Project: Kafka > Issue Type: Bug > Reporter: Ben Stopford > Attachments: TestDetails.pdf > > > There is evidence that the Kafka Producer drops performance as we increase > the number of threads using it. > This is based on some benchmarking done in the community. I have not > independently validated these results. Details of the test setup attached. > ... > *Effect of Shared KafkaProducer across threads* > > Kafka documentation recommend using the KafkaProducer across multiple worker > threads. > > ||#Producers||#Consumers||#Topics||#Partitions per topic||RoundTrip > Throughput (events/sec)||Approx Broker Events (Millions/sec)|| > |1|1|1|1|268,312|0.5| > |4|4|4|4|759,186|1.5| > |8|8|8|8|640,738|1.2| > |8|8|8|16|847,314|1.7| > |8|8|8|48|17,791|0.035| > |16|16|16|64|5,997|0.01| > > Something appears to be wrong here, with 48 and 64 partitions the shared > KafkaProducer struggled to the point that performance became quite bad. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)