Anna Povzner created KAFKA-9677:
-----------------------------------
Summary: Low consume bandwidth quota may cause consumer not being
able to fetch data
Key: KAFKA-9677
URL: https://issues.apache.org/jira/browse/KAFKA-9677
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.3.1, 2.4.0, 2.2.2, 2.1.1, 2.0.1
Reporter: Anna Povzner
Assignee: Anna Povzner
When we changed quota communication with KIP-219, fetch requests get throttled
by returning empty response with the delay in `throttle_time_ms` and Kafka
consumer retrying again after the delay.
With default configs, the maximum fetch size could be as big as 50MB (or 10MB
per partition). The default broker config (1-second window, 10 full windows of
tracked bandwidth/thread utilization usage) means that < 5MB/s consumer quota
(per broker) may stop fetch request from ever being successful.
Or the other way around: 1 MB/s consumer quota (per broker) means that any
fetch request that gets >= 10MB of data (10 seconds * 1MB/second) in the
response will never get through.
h3. Proposed fix
Return less data in fetch response in this case: Cap `fetchMaxBytes` passed to
replicaManager.fetchMessages() from KafkaApis.handleFetchRequest() to <tracking
window> * <consume bandwidth quota>. In the example of default configs and
1MB/s consumer bandwidth quota, fetchMaxBytes will be 10MB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
