[
https://issues.apache.org/jira/browse/KAFKA-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998280#comment-15998280
]
Antony Stubbs commented on KAFKA-5178:
--
My raw notes on researching the issue and slack conversations:
Consumer Performance Notes
https://confluent.slack.com/archives/C0KRA68SZ/p1492451673007148
Chris Matta
[6:54 PM]
what happens if a consumer is consuming from multiple partitions with differing
cardinality, and they’ve set `max.poll.records` to 500, the first partition
always has 500 records to pull, does that mean the other partitions will never
be read?
Ewen Cheslack-Postava
[6:59 PM]
@chris it will return the others. the completed fetches are tracked
per-partition in a queue
[7:00]
and that queue is processed in order so that when a FetchRequest returns with
data for multiple partitions, all that data gets enqueued then a subsequent
FetchRequest is sent. any subsequent data from the same partitions will get
queued up after the data from the first request/response
Chris Matta
[7:03 PM]
ok, thanks @ewen, is this covered in documentation anywhere?
Ewen Cheslack-Postava
[9:03 PM]
not that i'm aware of. not even sure how we'd do that very concisely since it
requires a bit of understanding of the underlying protocol to not overstate the
guarantees, i.e. you need to know a bit about fetch requests
Ismael Juma
[10:44 PM]
@chris KIP-41 describes the behaviour of `max.poll.records`. KIP-74 is also
somewhat relevant if you’re thinking about the behaviour of fetches as well.
Not user docs though.
KIP-41 describes the behaviour of `max.poll.records`
https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records
KIP-74 is also somewhat relevant if you’re thinking about the behaviour of
fetches as well
https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes
https://confluent.slack.com/archives/C07FCMZ39/p1491263732459808
https://confluent.slack.com/archives/C07FCMZ39/p1491597142583282
https://issues.apache.org/jira/browse/KAFKA-4753
https://confluent.slack.com/archives/C07FCMZ39/p1492199660178716
Replicator issues with funding circle - hot fixes were shipped to them
http://testing.confluent.io/confluent-kafka-system-test-results/?prefix=2017-04-28--001.1493388356--apache--trunk--bc10f5f/Benchmark/
https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
https://issues.apache.org/jira/browse/KAFKA-4753 KafkaConsumer susceptible to
FetchResponse starvation
> Potential Performance Degradation in Kafka Producer when using Multiple
> Threads
> ---
>
> Key: KAFKA-5178
> URL: https://issues.apache.org/jira/browse/KAFKA-5178
> Project: Kafka
> Issue Type: Bug
>Reporter: Ben Stopford
> Attachments: TestDetails.pdf
>
>
> There is evidence that the Kafka Producer drops performance as we increase
> the number of threads using it.
> This is based on some benchmarking done in the community. I have not
> independently validated these results. Details of the test setup attached.
> ...
> *Effect of Shared KafkaProducer across threads*
>
> Kafka documentation recommend using the KafkaProducer across multiple worker
> threads.
>
> ||#Producers||#Consumers||#Topics||#Partitions per topic||RoundTrip
> Throughput (events/sec)||Approx Broker Events (Millions/sec)||
> |1|1|1|1|268,312|0.5|
> |4|4|4|4|759,186|1.5|
> |8|8|8|8|640,738|1.2|
> |8|8|8|16|847,314|1.7|
> |8|8|8|48|17,791|0.035|
> |16|16|16|64|5,997|0.01|
>
> Something appears to be wrong here, with 48 and 64 partitions the shared
> KafkaProducer struggled to the point that performance became quite bad.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
