[ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700704#comment-15700704 ]
Jason Gustafson commented on KAFKA-4405: ---------------------------------------- I think a better approach is KAFKA-4007. The idea is to always send the next prefetch (regardless whether we still have data queued), but avoid reading it from the socket until we need to. This is also related to KIP-81: https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+Bound+Fetch+memory+usage+in+the+consumer. > Kafka consumer improperly send prefetch request > ----------------------------------------------- > > Key: KAFKA-4405 > URL: https://issues.apache.org/jira/browse/KAFKA-4405 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.10.0.1 > Reporter: ysysberserk > > Now kafka consumer has added max.poll.records to limit the count of messages > return by poll(). > According to KIP-41, to implement max.poll.records, the prefetch request > should only be sent when the total number of retained records is less than > max.poll.records. > But in the code of 0.10.0.1 , the consumer will send a prefetch request if it > retained any records and never check if total number of retained records is > less than max.poll.records.. > If max.poll.records is set to a count much less than the count of message > fetched , the poll() loop will send a lot of requests than expected and will > have more and more records fetched and stored in memory before they can be > consumed. > So before sending a prefetch request , the consumer must check if total > number of retained records is less than max.poll.records. -- This message was sent by Atlassian JIRA (v6.3.4#6332)