[
https://issues.apache.org/jira/browse/KAFKA-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianet Magrans updated KAFKA-20780:
-----------------------------------
Fix Version/s: 4.4.0
> syncKafkaConsumer does not keep a fetch request pending on the broker,
> causing latency and throughput regressions vs the classic consumer
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20780
> URL: https://issues.apache.org/jira/browse/KAFKA-20780
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Reporter: Blaž Šnuderl
> Assignee: Lianet Magrans
> Priority: Major
> Fix For: 4.4.0
>
>
> When testing out kip-848 we see worse end to end latencies on realtime topics.
> The AsyncKafkaConsumer (KIP-848, group.protocol=consumer) leaves gaps where
> no fetch request is pending on the broker. The classic KafkaConsumer keeps
> one in flight at all times. On the same low-throughput topic (~1 msg/s), the
> classic consumer sees ~3.2 fetches/s and ~28ms latency, while the async
> consumer sees ~0.47 fetches/s and ~2300ms latency — an ~80x latency
> regression from switching the group protocol
> alone.\{{ConsumerNetworkThread.runOnce()}} sleeps for
> \{{maximumTimeToWait()}} between iterations with no fetch pending, so
> messages arriving in the gap wait until the next wakeup.
> {\{AsyncKafkaConsumer.checkInflightPoll()}} does the \{{inflightPoll ==
> null}} check before \{{maybeClearCurrentInflightPoll()}}. So on even
> iterations \{{inflightPoll}} is still non-null, no new fetch event is
> submitted, and \{{pollForFetches}} blocks the full timeout. Net: one fetch
> per ~2.1s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
