[
https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701241#comment-15701241
]
ysysberserk commented on KAFKA-4405:
------------------------------------
It seems to me that it is not only a performance problem but will cause big
problem .
For example , assume we set max.poll.records to 1 and each prefetch request can
fetch 1000 records and there are plenty unprocessed records to be read.
Then to consume all 1000 records in the first prefetch request, we need call
poll() 1000 times and 1000 prefetch request will be sent and 1000 * 1000
records are returned and stored in memory.
And when it goes on , more and more records will be fetched and need to wait
for a long time to be consumed.
When there are too many records, it may run out of memory.
So the count of prefetch must be strictly limited and be sent only when
necessary.
> Kafka consumer improperly send prefetch request
> -----------------------------------------------
>
> Key: KAFKA-4405
> URL: https://issues.apache.org/jira/browse/KAFKA-4405
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.10.0.1
> Reporter: ysysberserk
>
> Now kafka consumer has added max.poll.records to limit the count of messages
> return by poll().
> According to KIP-41, to implement max.poll.records, the prefetch request
> should only be sent when the total number of retained records is less than
> max.poll.records.
> But in the code of 0.10.0.1 , the consumer will send a prefetch request if it
> retained any records and never check if total number of retained records is
> less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message
> fetched , the poll() loop will send a lot of requests than expected and will
> have more and more records fetched and stored in memory before they can be
> consumed.
> So before sending a prefetch request , the consumer must check if total
> number of retained records is less than max.poll.records.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
