Dhruvil Shah created KAFKA-10518:
------------------------------------
Summary: Consumer fetches could be inefficient when lags are
unbalanced
Key: KAFKA-10518
URL: https://issues.apache.org/jira/browse/KAFKA-10518
Project: Kafka
Issue Type: Bug
Reporter: Dhruvil Shah
Consumer fetches are inefficient when lags are imbalanced across partitions,
due to head of the line blocking and the behavior of blocking for `max.wait.ms`
until data is available.
When the consumer receives a fetch response, it prepares the next fetch request
and sends it out. The caveat is that the subsequent fetch request would
explicitly exclude partitions for which the consumer received data in the
previous round. This is to allow the consumer application to drain the data for
those partitions, until the consumer fetches the other partitions it is
subscribed to.
This behavior does not play out too well if the consumer is consuming when the
lag is unbalanced, because it would receive data for the partitions it is
lagging on, and then it would send a fetch request for partitions that do not
have any data (or have little data). The latter will end up blocking for
fetch.max.wait.ms on the broker before an empty response is sent back. This
slows down the consumer’s overall consumption throughput since
fetch.max.wait.ms is 500ms by default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
