[
https://issues.apache.org/jira/browse/KAFKA-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield updated KAFKA-20460:
-------------------------------------
Description:
When draining a backlog of records using share consumers with record-limit
acquisition mode for a multi-partition topic with uneven record distribution,
it has been observed that sometimes there is a slow down in the consumption
rate of records. This is because a share consumer consumes from one broker at a
time in record-limit mode, and if one broker has records to fetch and another
has no records to fetch, the share consumer waits for record arrival on the
empty broker before fetching records from the non-empty broker. This is
expected, but could probably be optimised.
Thanks to [~vanlightly] for discovering this issue,
was:
When draining a backlog of records using share consumers with record-limit
acquisition mode, it has been observed that sometimes there is a slow down in
the consumption rate of records. This does not always occur, it does not occur
for batch-optimized acquisition mode, and it does not occur when there is
active production of records to the topic being consumed.
Thanks to [~vanlightly] for discovering this issue,
> Potential delivery slow-down for record-limit share consumers when draining
> record backlog
> ------------------------------------------------------------------------------------------
>
> Key: KAFKA-20460
> URL: https://issues.apache.org/jira/browse/KAFKA-20460
> Project: Kafka
> Issue Type: Improvement
> Components: clients, core
> Affects Versions: 4.2.0
> Reporter: Andrew Schofield
> Assignee: Andrew Schofield
> Priority: Major
>
> When draining a backlog of records using share consumers with record-limit
> acquisition mode for a multi-partition topic with uneven record distribution,
> it has been observed that sometimes there is a slow down in the consumption
> rate of records. This is because a share consumer consumes from one broker at
> a time in record-limit mode, and if one broker has records to fetch and
> another has no records to fetch, the share consumer waits for record arrival
> on the empty broker before fetching records from the non-empty broker. This
> is expected, but could probably be optimised.
> Thanks to [~vanlightly] for discovering this issue,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
