[
https://issues.apache.org/jira/browse/KAFKA-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484105#comment-15484105
]
Eno Thereska commented on KAFKA-4007:
-------------------------------------
[~hachikuji] isn't the point of max.poll.records to improve the fetch
pipelining. In the sense that if users set it to low they should expect
pipelining to degrade, while if they set it higher pipelining should improve.
Why do we want to mask that expected behaviour?
> Improve fetch pipelining for low values of max.poll.records
> -----------------------------------------------------------
>
> Key: KAFKA-4007
> URL: https://issues.apache.org/jira/browse/KAFKA-4007
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Jason Gustafson
> Assignee: Mickael Maison
>
> Currently the consumer will only send a prefetch for a partition after all
> the records from the previous fetch have been consumed. This can lead to
> suboptimal pipelining when max.poll.records is set very low since the
> processing latency for a small set of records may be small compared to the
> latency of a fetch. An improvement suggested by [~junrao] is to send the
> fetch anyway even if we have unprocessed data buffered, but delay reading it
> from the socket until that data has been consumed. Potentially the consumer
> can delay reading _any_ pending fetch until it is ready to be returned to the
> user, which may help control memory better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
